Title: RESTRICTING THE FLOW: INFORMATION BOTTLENECKS FOR ATTRIBUTION

Abstract: Attribution methods provide insights into the decision-making of machine learning models like artificial neural networks. For a given input sample, they assign a relevance score to each individual input variable, such as the pixels of an image. In this work, we adopt the information bottleneck concept for attribution. By adding noise to intermediate feature maps, we restrict the flow of information and can quantify (in bits) how much information image regions provide. We compare our method against ten baselines using three different metrics on VGG-16 and ResNet-50, and find that our methods outperform all baselines in five out of six settings. The method's information-theoretic foundation provides an absolute frame of reference for attribution values (bits) and a guarantee that regions scored close to zero are not necessary for the network's decision.

1. INTRODUCTION:

Deep neural networks have become state of the art in many realworld applications. However, their increasing complexity makes it difficult to explain the model's output. For some applications such as in medical decision making or autonomous driving, model interpretability is an important requirement with legal implications. Attribution methods (Selvaraju et al., 2017; Zeiler & Fergus, 2014; Smilkov et al., 2017) aim to explain the model behavior by assigning a relevance score to each input variable. When applied to images, the relevance scores can be visualized as heatmaps over the input pixel space, thus highlighting salient areas relevant for the network's decision.

For attribution, no ground truth exists. If an attribution heatmap highlights subjectively irrelevant areas, this might correctly reflect the network's unexpected way of processing the data, or the heatmap might be inaccurate (Nie et al., 2018; Viering et al., 2019; Sixt et al., 2019) . Given an image of a railway locomotive, the attribution map might highlight the train tracks instead of the train itself. Current attribution methods cannot guarantee that the network is ignroing the low-scored locomotive for the prediction.

We propose a novel attribution method that estimates the amount of information an image region provides to the network's prediction. We use a variational approximation to upper-bound this estimate and therefore, can guarantee that areas with zero bits of information are not necessary for the prediction. Figure 1 shows an exemplary heatmap of our method. Up to 3 bits per pixel are available for regions corresponding to the monkeys' faces, whereas the tree is scored with close to zero bits per pixel. We can thus guarantee that the tree is not necessary for predicting the correct class.

To estimate the amount of information, we adapt the information bottleneck concept (Tishby et al., 2000; Alemi et al., 2017) . The bottleneck is inserted into an existing neural network and restricts the information flow by adding noise to the activation maps. Unimportant activations are replaced almost entirely by noise, removing all information for subsequent network layers. We developed two approaches to learn the parameters of the bottleneck -either using a single sample (Per-Sample Bottleneck), or the entire dataset (Readout Bottleneck).

We evaluate against ten different baselines. First, we calculated the Sensitivity-n metric proposed by Ancona et al. (2018) . Secondly, we quantified how well the object of interest was localized using bounding boxes and extend the degradation task proposed by Ancona et al. (2017) . In all these metrics, our method outperforms the baselines consistently. Additionally, we test the impact of cascading layer-wise weight randomizations on the attribution heatmaps (Adebayo et al., 2018) . For reproducibility, we share our source code and provide an easy-to-use * implementation. We name our method IBA which stands for Information Bottleneck Attribution. It provides a theoretic upper-bound on the used information while demonstrating strong empirical performance. Our work improves model interpretablility and increases trust in attribution results.

To summarize our contributions:

• We adapt the information bottleneck concept for attribution to estimate the information used by the network. Information theory provides a guarantee that areas scored irrelevant are indeed not necessary for the network's prediction. • We propose two ways -Per-Sample and Readout Bottleneck -to learn the parameters of the information bottleneck. • We contribute a novel evaluation method for attribution based on bounding boxes and we also extend the metric proposed by Ancona et al. (2017) to provide a single scalar value and improve the metric's comparability between different network architectures.

2. RELATED WORK:

Attribution is an active research topic. Gradient Maps (Baehrens et al., 2010) & Fergus, 2014) measures the importance as the drop in classification accuracy after replacing individual image patches with zeros. Similarly, blurring image patches has been used to quantify the importance of high-frequency features (Greydanus et al., 2018) . -Cam (Selvaraju et al., 2017) take the activations of the final convolutional layer to compute relevance scores. They also combine their method with GuidedBP: GuidedGrad-CAM. Ribeiro et al. (2016) uses image superpixels to explain deep neural networks. High-level concepts rather input pixels are scored by TCAV (Kim et al., 2018) . Khakzar et al. (2019) prune irrelevant neurons. Similar to our work, Macdonald et al. (2019) uses a rate-distortion perspective, but minimize the norm of the mask instead of the shared information. To the best of our knowledge, we are the first to estimate the amount of used information for attribution purposes.

Grad:

Although many attribution methods exist, no standard evaluation benchmark is established. Thus, determining the state of the art is difficult. The performance of attribution methods is highly dependent on the used model and dataset. Often only a purely visual comparison is performed (Smilkov et al., 2017; Springenberg et al., 2014; Montavon et al., 2017; Sundararajan et al., 2017; Bach et al., 2015) . The most commonly used benchmark is to degrade input images according to the attribution heatmap and measure the impact on the model output. (Kindermans et al., 2018; Samek et al., 2016; Ancona et al., 2017) . The Sensitivity-n score (Ancona et al., 2018) is obtained by randomly masking the input image and then measuring the correlation between removed attribution mass and model performance.

For the ROAR score (Hooker et al., 2018) , the network is trained from scratch on the degraded images. It is computationally expensive but avoids out-of-domain inputs -an inherent problem of masking. Adebayo et al. (2018) proposes a sanity check: if the network's parameters are randomized, the attribution output should change too.

Adding noise to a signal reduces the amount of information (Shannon, 1948) . It is a popular way to regularize neural networks (Srivastava et al., 2014; Kingma et al., 2015; Gal et al., 2017) . For regularization, the noise is applied independently from the input and no attribution maps can be obtained. In Variational Autoencoders (VAEs) (Kingma & Welling, 2013) , noise restricts the information capacity of the latent representation. In our work, we construct a similar information bottleneck that can be inserted into an existing network. Deep convolutional neural networks have been augmented with information bottlenecks before to improve the generalization and robustness against adversarial examples (Achille & Soatto, 2018; Alemi et al., 2017) . Zhmoginov et al. (2019) and Taghanaki et al. ( 2019) extract salient regions using information bottlenecks. In contrast to our work, they add the information bottleneck already when training the network and do not focus on post-hoc explanations.

3. INFORMATION BOTTLENECK FOR ATTRIBUTION:

Instead of a backpropagation approach, we quantify the flow of information through the network in the forward pass. Given a pre-trained model, we inject noise into a feature map, which restrains the flow of information through it. We optimize the intensity of the noise to minimize the information flow, while simultaneously maximizing the original model objective, the classification score. The parameters of the original model are not changed.

Generally, the information bottleneck concept (Tishby et al., 2000) describes a limitation of available information. Usually, the labels Y are predicted using all information from the input X. The information bottleneck limits the information to predict Y by introducing a new random variable Z.

The information the new variable Z shares with the labels Y is maximized while minimizing the information Z and X share :

max I[Y ; Z] -βI[X, Z] ,

(1) where I[X, Z] denotes the mutual information and β controls the trade-off between predicting the labels well and using little information of X. A common way to reduce the amount of information is to add noise (Alemi et al., 2017; Kingma & Welling, 2013) .

For attribution, we inject an information bottleneck into a pretrained network. The bottleneck is inserted into an layer which still contains local information, e.g. for the ResNet the bottleneck is added after conv3 * (after the last conv3 block). Let R denote the intermediate representations at this specific depth of the network, i.e. R = f l (X) where f l is the l-th layer output. We want to reduce information in R by adding noise. As the neural network is trained already, adding noise should preserve the variance of the input to the following layers. Therefore, we also damp the signal R when increasing the noise, effectively replacing the signal partly with noise. In the extreme case, when no signal is transmitted, we replace R completely with noise of the same mean and variance as R.

For this purpose, we estimate the mean µ R and variance σ 2 R of each feature of R empirically. As information bottleneck, we then apply a linear interpolation between signal and noise:

EQUATION

where ∼ N (µ R , σ 2 R ) and λ(X) controls the damping of the signal and the addition of the noise. The value of λ is a tensor with the same dimensions as R and λ i ∈ [0, 1]. Given λ i (X) = 1 at the feature map location i, the bottleneck transmits all information as Z i = R i . Whereas if λ i = 0, then Z i = and all information of R i is lost and replaced with noise. It could be tempting to think that Z from equation 2 has the same mean and variance as R. This is not the case in general as λ(X) and R both depend on X (for more detail see Appendix E).

In our method, we consider an area relevant if it contains useful information for classification. We need to estimate how much information Z still contains about R. This quantity is the mutual information I[R, Z] that can be written as:

EQUATION

where P (Z|R) and P (Z) denote the respective probability distributions. We have no analytic expression for P (Z) since it would be necessary to integrate over the feature map p(z) = R p(z|r)p(r)dr

λR+(1-λ) R input net- bottleneck -work logits α sigmoid λ blur ∼ N (µ R , σ 2 R )

Figure 2 : Per-Sample Bottleneck: The mask (blue) contains an α i for each r i in the intermediate feature maps R (green). The parameter α controls how much information is passed to the next layer. The mask α is optimized for each sample individually according to equation 6.

-an intractable integral. It is a common problem that the mutual information can not be computed exactly but is instead approximated (Poole et al., 2019; Suzuki et al., 2008) . We resort to a variational approximation

Q(Z) = N (µ R , σ R )

which assumes that all dimensions of Z are distributed normally and independent -a reasonable assumption, as activations after linear or convolutional layers tend to have a Gaussian distribution (Klambauer et al., 2017) . The independence assumption will generally not hold. However, this only overestimates the mutual information as shown below. Substituting Q(Z) into the previous equation 3, we obtain:

EQUATION

)

The derivation is shown in appendix D and follows Alemi et al. (2017) . The first term contains the KL-divergence between two normal distributions, which is easy to evaluated and we use it to approximate the mutual information. The information loss function L I is therefore:

EQUATION

We know that L I overestimates the mutual information, i.e. L I ≥ I[R, Z] as the second KLdivergence term D KL [P (Z)||Q(Z)] has to be positive. If L I is zero for an area, we can guarantee that information from this area is not necessary for the network's prediction. Information from this area might still be used when no noise is added.

We aim to keep only the information necessary for correct classification. Thus, the mutual information should be minimal while the classification score should remain high. Let L CE be the cross-entropy of the classification. Then, we obtain the following optimization problem:

EQUATION

where the parameter β controls the relative importance of both objectives. For a small β, more bits of information are flowing and less for a higher β. We propose two ways of finding the parameters λthe Per-Sample Bottleneck and the Readout Bottleneck. For the Per-Sample Bottleneck, we optimize λ for each image individually, whereas in the readout bottleneck, we train a distinct neural network to predict λ.

3.2. PER-SAMPLE BOTTLENECK:

For the Per-Sample Bottleneck, we use the bottleneck formulation described above and optimize L for individual samples -not for the complete dataset at once. Given a sample x, λ is fitted to the sample to reflect important and unimportant regions in the feature space. A diagram of the Per-Sample Bottleneck is shown in Figure 2 .

Parameterization The bottleneck parameters λ have to be in [0, 1]. To simplify optimization, we parametrize λ = sigmoid(α). This parametrization allows α ∈ R d and avoids any clipping of λ to [0, 1] during optimization.

Initialization As when training neural networks, the initialization of parameters matters. In the beginning, we want to retain all the information. For all dimensions i, we initialize α i = 5 and thus λ i ≈ 0.993 ⇒ Z ≈ R. At first, the bottleneck has practically no impact on the model performance.

It then deviates from this starting point to suppress unimportant regions.

λR+(1-λ) 1 2 R input net- bottleneck -work logits sigmoid λ blur network resize conv 1x1 conv 1x1 conv 1x1 readout net. ∼ N (µ R , σ 2 R )

Figure 3 : Readout Bottleneck: In the first forward pass 1 , feature maps are collected at different depths. The readout network uses a resized version of the feature maps to predict the parameters for the bottleneck layer. In the second forward pass 2 , the bottleneck is inserted and noise added. All parameters of the analyzed network are kept fixed.

Optimization We do 10 iterations using the Adam optimizer (Kingma & Ba, 2014) with learning rate 1 to fit the mask α. To stabilize the training, we copy the single sample 10 times and apply different noise to each. In total, we execute the model on 100 samples to create a heatmap, comparable to other methods such as SmoothGrad. After the optimization, the model usually predicts the target with probability close to 1, indicating that only negative evidence was removed.

Measure of information:

To measure the importance of each feature in Z, we evaluate D KL (P (Z|R)||Q(Z)) per dimension.

It shows where the information flows. We obtain a two-dimensional heatmap m by summing over the channel axis:

m [h,w] = c i=0 D KL (P (Z [i,h,w] |R [i,h,w] )||Q(Z [i,h,w] )).

As convolutional neural networks preserve the locality in their channel maps, we use bilinear interpolation to resize the map to the image size. For the ResNet-50, we insert the bottleneck after layer conv3 *. Choosing a later layer with a lower spatial resolution would increase the blurriness of the attribution maps, due to the required interpolation. The effect of different depth choices for the bottleneck is visualized in figure 4 .

Enforcing local smoothness Pooling operations and convolutional layers with stride greater than 1 are ignoring parts of the input. The ignored parts cause the Per-Sample Bottleneck to overfit to a grid structure (shown in Appendix B). To obtain a robust and smooth attribution map, we convolve the sigmoid output with a fixed Gaussian kernel with standard deviation σ s . Smoothing the mask during training is not equivalent to smoothing the resulting attribution map, as during training also the gradient is averaged locally. Combining everything, the parametrization for the Per-Sample Bottleneck is: λ = blur(σ s , sigmoid(α)) .

(7)

3.3. READOUT BOTTLENECK:

For the Readout Bottleneck, we train a second neural network to predict the mask α. In contrast to the Per-Sample Bottleneck, this model is trained on the entire training set. In Figure 3 , the Readout Bottleneck is depicted. Kümmerer et al. (2014) introduced the readout concept for gaze prediction. The general idea is to collect feature maps from different depths and then combine them using 1x1 convolutions.

In a first forward pass, no noise is added and we collect the different feature maps. As the spatial resolution of the feature maps differs, we interpolate them bilinearly to match the spatial dimensions of the bottleneck layer. The readout network then predicts the information mask based on the collected feature maps. In a second forward pass, we insert the bottleneck layer into the network and restrict the flow of information.

Except for the learned importance values, the Readout Bottleneck is identical to the mechanism of the Per-Sample Bottleneck. The measure of information works in the same way as for the Per-Sample Bottleneck and we also use the same smoothing. Given a new sample, we can obtain a heatmap by merely collecting the feature maps and executing the readout network. The readout network consists of three 1x1 convolutional layers. ReLU activations are applied between convolutional layers and a final sigmoid activation yields λ ∈ [0, 1]. As the input consists of upscaled feature maps, the field-of-view is large, although the network itself only uses 1x1 conv. kernels.

4. EVALUATION:



4.1. EXPERIMENTAL SETUP:

As neural network architectures, we selected the ResNet-50 (He et al., 2016) and the VGG-16

Simonyan & Zisserman (2014), using pretrained weights from the torchvision package (Paszke et al., 2017) . These two models cover a range of concepts: dimensionality by stride or max-pooling, residual connections, batch normalization, dropout, low depth (16-weight-layer VGG), and high depth (50-weight-layer ResNet). This variety makes the evaluation less likely to overfit on a specific model type. They are commonly used in the literature concerning attribution methods. For PatternAttribution on the VGG-16, we obtained weights for the signal estimators from Kindermans et al. (2018) .

As naive baselines, we selected random attribution, Occlusion with patch sizes 8x8 and 14x14, and Gradient Maps. SmoothGrad and Integrated Gradients cover methods that accumulate gradients. We include three methods with modified backpropagation rules: PatternAttribution, GuidedBP, and LRP. As our implementation of PatternAttribution and LRP does not support skip connections, we report no results for them on the ResNet-50. We also include Grad-CAM and its combination with GuidedBP, GuidedGrad-CAM.

For the compared methods, we use the hyperparameters suggested by the original authors. For LRP (Bach et al., 2015) , different rules exist. We include the most commonly used α=1, β =0, =0 rule which is also displayed in the figures. We also include α=0, β =-1, =5 as it gives better results on the bounding-box task. For sensitivity-n and sanity checks, only the later LRP variante is evaluted. For our methods, the hyperparameters are obtained using grid search with the degradation metric as objective (Appendix F). The readout network is trained on the training set of the ILSVRC12 dataset (Russakovsky et al., 2015) for E = 20 epochs.

The optimization objective of the bottleneck is L CE + βL I as given in equation 6. Generally, the information loss L I is larger than the classifier loss by several orders of magnitude as it sums over height h, width w and channels c. We therefore use k = hwc as a reference point to select β. In figure 4 , we display the effect of different orders of β and the effect of layer depth for ResNet-50 (see Appendix C for VGG-16). A uniform uninformative heatmap is obtained for β ≥ 1000/k -all information gets discarded, resulting in an output probability of 0 for the correct class. The heatmap for β = 0.1/k is similar to β = 1/k but more information is passed, i.e. less noises is added. In appendix F, we also compare pre-to post-training accuracy and estimate the mutual information for 

4.2. QUALITATIVE ASSESSMENT:

In Figure 5 , the heatmaps of all evaluated samples are shown (for more samples, see Appendix A). Subjectively, both the Per-Sample and Readout Bottleneck identify areas relevant to the classification well. While Guided Backpropagation and PatternAttribution tend to highlight edges, the Per-Sample Bottleneck focuses on image regions. For the Readout Bottleneck, the attribution is concentrated a little more on the object edges. Compared to Grad-CAM, both our methods are more specific, i.e. fewer pixels are scored high.

4.3 SANITY CHECK: RANDOMIZATION OF MODEL PARAMETERS Adebayo et al. (2018) investigates the effect of parameter randomization on attribution masks. A sound attribution method should depend on the entire network's parameter set. Starting from the last layer, an increasing proportion of the network parameters is re-initialized until all parameters are random. We excluded PatternAttribution as the randomization would require to re-estimate the signal directions. The difference between the original heatmap and the heatmap obtained from the randomized model is quantified using SSIM (Wang et al., 2004) . We discuss implementation details in the appendix G.1.

In figure 6 , we display the results of the sanity check. For our methods, we observe that randomizing the final dense layer drops the mean SSIM by around 0.4. The values for the Readout Bottleneck are of limited expressiveness as we did not re-train it after randomization. For SmoothGrad and Int.

Gradients, the SSIM drops by more than 0.4. The heatmaps of GuidedBP and LRP remain similar even if large portion of the network's parameters are randomized -they do not explain the network prediction faithfully. Nie et al. (2018) provides theoretical insights about why GuidedBP fails. Sixt et al. (2019) analyzes why LRP and other modified BP methods fail. Ancona et al. (2018) proposed Sensitivity-n as an evaluation metric for attribution methods. Sensitivity-n masks the network's input randomly and then measures how strongly the amount of attribution in the mask correlates with the drop in classifier score.Given a set T n containing n randomly selected pixel indices, Sensitivity-n measures the Pearson correlation coefficient:

4.4. SENSITIVITY-N:

EQUATION

where S c (x) is the classifier logit output for class c, R i is the relevance at pixel i and x [x Tn =0] denotes the input with all pixels in T n set to zero. As in the original paper, we pick the number of masked pixels n in a log-scale between 1 and 80% of all pixels. For each n, we generate 100 different index sets T and test each on 1000 randomly selected images from the validation set. The correlation is calculated over the different index sets and then averaged over all images.

In Figure 6 , the Sensitivity-n scores are shown for ResNet-50 and VGG-16. When masking inputs pixel-wise as done in Ancona et al. (2018) , the Sensitivity-n score across methods are not well discriminable, all scores range within the lower 10% of the scale. We ran the metric with a tile size of 8x8 pixels. This resulted in an increase of the Sensitivity-n score to 30% of the scale. Although not shown in the figure due to the zooming, Occlusion-8x8 performs perfectly for small n as its relevance scores correspond directly to the drop in logits per 8x8 tile. We find that the Per-Sample Bottlenecks β = 10/k perform best for both models above n = 2 • 10 3 pixels, i.e. when more then 2% of all pixels are masked.

4.5. BOUNDING BOX:

To quantify how well attribution methods identify and localize the object of interest, we rely on bounding boxes available for the ImageNet dataset. Bounding boxes may contain irrelevant areas, in particular for non-rectangular objects. We restrict our evaluation to images with bounding boxes covering less than 33% of the input image. In total, we run the bounding box evaluation on 11,849 images from the ImageNet validation set.

If the bounding box contains n pixels, we measure how many of the n-th highest scored pixels are contained in the bounding box. By dividing by n, we obtain a ratio between 0 and 1. The results are shown in Table 1 . The Per-Sample Bottleneck outperforms all baselines on both VGG-16 and ResNet-50 each by a margin of 0.152.

An alternative metric would be to take the sum of attribution in the bounding box and compare it to the total attribution in the image. We found this metric is not robust against extreme values. For the ResNet-50, we found basic Gradient Maps to be the best method as a few pixels receiving extreme scores are enough to dominate the sum.

4.6. IMAGE DEGRADATION:

As a further quantitative evaluation, we rely on the degradation task as used by Ancona et al. (2017) tile of the attribution. At each iteration, the highest-ranked tile is replaced with a constant value, the modified input is fed through the network, and the resulting drop in target class score is measured. A steep descent of the accuracy curve indicates a meaningful attribution map. When implemented in the described way, the most relevant tiles are removed first (MoRF). However, Ancona et al. (2017) argues that using only the MoRF curve for evaluation is not sufficient. For the MoRF score, it is beneficial to find tiles disrupting the output of the neural network as quickly as possible.

Neural networks are sensitive to subtle changes in the input (Szegedy et al., 2013) . The tiles do not necessarily have to contain meaningful information to disrupt the network. Ancona et al. (2017) proposes to invert the degradation direction, removing tiles ranked as least relevant by the attribution method first (LeRF). The LeRF task favors methods that identify areas sufficient for classification. We scale the network's output probabilities to be in [0, 1]:

EQUATION

)

where t 1 is the model's average top-1 probability on the original samples and b is the mean model output on the fully degraded images. Both averages are taken over the validation set. A score of 1 corresponds to the original model performance.

Both LeRF and MoRF degradation yield curves as visualized in Figure 7 , measuring different qualities of the attribution method. To obtain a scalar measure of attribution performance and to combine both metrics, we propose to calculate the integral between the MoRF and LeRF curves.

The results for all implemented attribution methods on the degradation task are given in Table 1 . We evaluated both models on the full validation set using 8x8 and 14x14 tiles. In Appendix H, we show the mean LeRF and MoRF curves for 14x14 tiles. The Per-Sample Bottleneck outperforms all other methods in the degradation benchmark except for GuidedGrad-CAM on ResNet-50 where it scores comparably (score difference of 0.004). The Readout Bottleneck generally achieves a lower degradation scores but still perform competitively.

5. CONCLUSION:

We propose two novel attribution methods that return an upper bound on the amount of information each input region provides for the network's decision. Our models' core functionality is a bottleneck layer used to inject noise into a given feature layer and a mechanism to learn the parametrized amount of noise per feature. The Per-Sample Bottleneck is optimized per single data point, whereas the Readout Bottleneck is trained on the entire dataset.

Our method does not constrain the internal network structure. In contrast to several modified backpropagation methods, it supports any activation function and network architecture. To evaluate our method, we extended the degradation task to quantify model performance deterioration when removing both relevant and irrelevant image tiles first. We also show results on how well ground-truth bounding boxes are scored. Our Per-Sample and Readout Bottleneck both show competitive results on all metrics used, outperforming state of the art with a significant margin for some of the tasks.

Generally, we would advise using the Per-Sample Bottleneck over the Readout Bottleneck. It performs better and is more flexible as it only requires to estimate the mean and variance of the feature map. The Readout Bottleneck has the advantage of producing attribution maps with a single forward pass once trained. Images with multiple object instances provide the network with redundant class information. The Per-Sample Bottleneck may discard some of the class evidence. Even for single object instances, the heatmaps of the Per-Sample Bottleneck may vary slightly due to the randomness of the optimization process.

The method's information-theoretic foundation provides a guarantee that the network does not require regions of zero-valued attribution for correct classification. To our knowledge, our attribution method is the only one to provide scores with units (bits). This absolute frame of reference allows a quantitative comparison between models, inputs, and input regions. We hope this contributes to a deeper understanding of neural networks and creates trust to use modern models in sensitive areas of application.

B GRID ARTIFACTS WHEN NOT USING SMOOTHING 

E MEAN AND VARIANCE OF Z

The λ(X) linearly interpolate between the feature map R and the noise ∼ N (µ R , σ 2 R ), where µ R and σ R are the estimated mean and standard derivation of R. Both R = f l (X) and λ(X) depend on the input random variable X. Z = λ(X)R + (1 -λ(X))

(17) For the mean of Z, we have: A similar problem araises for the variance:

E[Z] = E[λ(X)R] + E[(1 -λ(X)) ]

Var[Z] = E[Z 2 ] -E[Z] 2 = E[(λ(X)R + (1 -λ(X) ) 2 ] -(cov(λ(X), R) + E[ ])

2

The multiplication of λ(X) and R causes in general the variance of Z and R to not match:

Var[Z] = Var[R].

F HYPERPARAMETERS (He et al., 2016) . The first index denotes the block and the second the layer within the block. For the VGG-16, conv n denotes the n-th convolutional layer.

:



Figures and Tables:

Figure 1: Exemplary heatmap of the Per-Sample Bottleneck for the VGG-16.

Figure 4: Effect of varying layer depth and β on the Per-Sample Bottleneck for the ResNet-50. The color bars measure the bits per input pixel. The resulting output probability of the correct class p = p(y|x, β) is decreasing for higher β.

Figure 5: Heatmaps of all implemented methods for the VGG-16 (see Appendix A for more).

Figure 6: First row drop in SSIM score when network layers are randomized. Best viewed in color. Second-row Sensitivity-n scores for ResNet-50 and VGG-16. For the ResNet-50, also tile size 1x1 is shown. We clipped the y-axis at 0.4 to improve discriminability.

Figure 7: Mean MoRF and LeRF for the Per-Sample Bottleneck. The area is the final degradation score.

Figure 9: Development of D KL (Q(Z|X)||Q(Z)) of the Per-Sample Bottleneck for layer conv1 3 of the ResNet-50. Red indicate areas with maximal information flow and semi-transparent green for zero information flow. Top row: without smoothing the mask exhibits a grid structure. Bottom row: smoothing with σ s = 2. The smoothing both prevents the artifacts and reduces overfitting to small areas.

substituting in definition of Z = E[λ(X)R] + E[1 -λ(X)]E[ ] independence of λ and = cov(λ(X), R) + E[λ(X)]E[R] + E[1 -λ(X)]E[ ] cov(A, B) = E[AB] -E[A]E[B] ≈ cov(λ(X), R) + E[ ] E[ ] = µ R ≈ E[R]As λ(X) and R are multiplied together, they form a complex product distribution. If they do not correlate,E[Z] ≈ E[ ] ≈ E[R].

;Kindermans et al. (2018);Hooker et al. (2018);Samek et al. (2016). Given an attribution heatmap, the input is split in tiles, which are ranked by the sum of attribution values within each corresponding Degradation (deg.): Integral between LeRF and MoRF in the degradation benchmark for different models and window sizes over the ImageNet test set. Bounding Box (bbox): the ratio of the highest scored pixels within the bounding box. For ResNet-50, we show no results for PatternAttribution and LRP as no PyTorch implementation supports skip-connections.

Table content: <table><tr><td>Model &amp; Evaluation</td><td colspan="2">ResNet-50 deg.</td><td colspan="2">VGG-16 deg.</td><td>ResNet VGG</td></tr><tr><td/><td>8x8</td><td colspan="2">14x14 8x8</td><td colspan="2">14x14 bbox</td><td>bbox</td></tr><tr><td>Random</td><td colspan="2">0.000 0.000</td><td colspan="3">0.000 0.000 0.167</td><td>0.167</td></tr><tr><td>Occlusion-8x8</td><td colspan="2">0.162 0.130</td><td colspan="3">0.267 0.258 0.296</td><td>0.312</td></tr><tr><td>Occlusion-14x14</td><td colspan="2">0.228 0.231</td><td colspan="3">0.402 0.404 0.341</td><td>0.358</td></tr><tr><td>Gradient</td><td colspan="2">0.002 0.005</td><td colspan="3">0.001 0.005 0.259</td><td>0.276</td></tr><tr><td>Saliency</td><td colspan="2">0.287 0.305</td><td colspan="3">0.326 0.362 0.363</td><td>0.393</td></tr><tr><td>GuidedBP</td><td colspan="2">0.491 0.515</td><td colspan="3">0.460 0.493 0.388</td><td>0.373</td></tr><tr><td>PatternAttribution</td><td>-</td><td>-</td><td colspan="3">0.440 0.457 -</td><td>0.404</td></tr><tr><td>LRP α = 1, β = 0</td><td>-</td><td>-</td><td colspan="3">0.471 0.486 -</td><td>0.397</td></tr><tr><td>LRP α = 0, β = 1, = 5</td><td>-</td><td>-</td><td colspan="3">0.462 0.467 -</td><td>0.441</td></tr><tr><td>Int. Grad.</td><td colspan="2">0.401 0.424</td><td colspan="3">0.420 0.453 0.372</td><td>0.396</td></tr><tr><td>SmoothGrad</td><td colspan="2">0.485 0.502</td><td colspan="3">0.438 0.455 0.439</td><td>0.399</td></tr><tr><td>Grad-CAM</td><td colspan="2">0.536 0.541</td><td colspan="3">0.510 0.517 0.465</td><td>0.399</td></tr><tr><td>GuidedGrad-CAM</td><td colspan="2">0.565 0.577</td><td colspan="3">0.555 0.576 0.468</td><td>0.419</td></tr><tr><td>IBA Per-Sample β = 1/k</td><td colspan="2">0.573 0.573</td><td colspan="3">0.581 0.583 0.606</td><td>0.566</td></tr><tr><td>IBA Per-Sample β = 10/k</td><td colspan="2">0.572 0.571</td><td colspan="3">0.582 0.585 0.620</td><td>0.593</td></tr><tr><td colspan="3">IBA Per-Sample β = 100/k 0.534 0.535</td><td colspan="3">0.542 0.545 0.574</td><td>0.568</td></tr><tr><td>IBA Readout β = 10/k</td><td colspan="2">0.536 0.536</td><td colspan="3">0.490 0.536 0.484</td><td>0.437</td></tr></table>

Hyperparameters for Per-Sample Bottleneck. The layer notations for the ResNet-50 are taken from the original publication

Table content: <table/>

Peer review (iclr2020):
Summary --- (motivation) Lots of methods produce attribution maps (heat maps, saliency maps, visual explantions) that aim to highlight input regions with respect to a given CNN. These methods produce scores that highlight regions that are in a vague sense *important.* While that*s useful (relative importance is interesting), the scores don*t mean anything by themselves. This paper introduces another new attribution method that measures the amount of information (in bits!) each input region contains, calibrating this score by providing a reference point at 0 bits. Non-highlighted regions contribute 0 bits of information to the task, so they are clearly irrelevant in the common sense that they have 0 mutual information with the correct output. (approach - attribution methods) An information bottleneck is introduced by replacing a layer*s (e.g., conv2) output X with a noisy version Z of that output. In particular, Z is a convex combination of the feature map (e.g., conv2) with Gaussian noise with the same mean and variance as that feature map. The weights of the combination are found so they minimize the information shared between the input and Z and maxmimize information shared between Z and the task output Y. These weights are either optimized on 1) a per-image basis (Per-Sample) or 2) predicted by a model trained on the entire dataset (Readout). (approach - evaluation) The paper uses 3 metrics with differing degrees of novelty: 1) The bbox metric rewards attribution methods that put a lot of mass in ground truth bounding boxes. 2) The original Sensitivity-n metric from (Ancona et al. 2017) is reported with a version that uses 8x8 occlusions. 3) Least relevant image degredation is compared to most relevant image degredation (e.g., from (Ancona et al. 2017)) to form a new occlusion style metric. (experiments) Experiments consider many of the most popular baselines, including Occlusion, Gradients, SmoothGrad, Integrated Gradients, GuidedBP, LRP, Grad-CAM, and Pattern Attribution. They show: 1) Qualitatively, the visualizations highlight only regions that seem relevant. 2) Both Per-Sample and Readout approaches put higher confidence into ground truth bounding boxes than all other baselines. 3) Both Per-Sample and Readout approaches outperform all baselines almost all the time according to the new image degredation metric. Strengths --- The idea makes a lot of sense. I think heat maps are often thought of in terms of the colloquial sense of information, so it makes sense to formalize that intuition. The related work section is very well done. The first paragraph is particularly good because it gives not just a fairly comprehensive view of attribution methods, but also because it efficiently describes how they all work. The results show that proposed approaches clearly outperform many strong baselines across different metrics most of the time. Weaknesses --- * I*m not sure why the new degredation metric is a useful addition. What does it add that MoRF and LeRF don*t capture on their own independently? * I think [1] would be a nice addition to the evaluation section as it tests for something qualitatively different than the various metrics from section 4. It would also be a good addition to the related work. Missing Details / Points of Confusion --- * I think there*s an extra p(x) in eq. 11 in appendix D. * I think the variable X is overloaded. In eq. 1 it refers to the input (e.g., the pixels of an image) while in eq. 2 it refers to an intermediate feature map (e.g., conv2) even though it later seems to refer to the input again (e.g., eq. 3). Different notation should be used for intermediate feature maps and inputs. Presentation Weaknesses --- * In section 3.1 is lambda meant to be constrained in the range [0, 1]? This is only mentioned later (section 3.2) and should probably be mentioned when lambda is introduced. * *indicating that all negative evidence was removed.* I think this should read *indicating that only negative evidence was removed.* Suggestions --- *The bottleneck is inserted into an early layer to ensure that the information in the network is still local* I*d like this to be explored a bit more. Though deeper feature maps are certainly more spatially coarse they still might be somewhat *local*. To what degree to they loose localization information? My equally vague alternative intuition goes a bit differently: The amount of relevant information flowing through any spatial location seems like it shouldn*t change that much, only the way its represented should change. If the proposed visualizations were the same for every choice of layer then it would confirm this intuition. That would also be an interesting result because most if not all of the cited baseline approaches (where applicable) produce qualitatively different attributions at different layers (e.g., see Grad-CAM). [1]: Adebayo, Julius et al. “Sanity Checks for Saliency Maps.” NeurIPS (2018). Preliminary Evaluation --- Clarity: The paper is clearly written. Originality: The idea of using the formal notion of information in attribution maps is novel, as is the bbox metric. Significance: This method could be quite significant. I can see it becoming an important method to compare to. Quality: The idea is sound and the evaluation is strong. This is a very nice paper in all the ways listed above and it should be accepted! Post-rebuttal comments --- The author responses and other reviews have only increased my confidence that this paper should be accepted.