Restricting the Flow: Information Bottlenecks for Attribution
Authors: Karl Schulz
Leon Sixt
Federico Tombari
Tim Landgraf
Venue:  (2020)
Keywords: Attribution
Informational Bottleneck
Interpretable Machine Learning
Explainable AI

Abstract: Attribution methods provide insights into the decision-making of machine learning models like artificial neural networks. For a given input sample, they assign a relevance score to each individual input variable, such as the pixels of an image. In this work, we adopt the information bottleneck concept for attribution. By adding noise to intermediate feature maps, we restrict the flow of information and can quantify (in bits) how much information image regions provide. We compare our method against ten baselines using three different metrics on VGG-16 and ResNet-50, and find that our methods outperform all baselines in five out of six settings. The method’s information-theoretic foundation provides an absolute frame of reference for attribution values (bits) and a guarantee that regions scored close to zero are not necessary for the network's decision. 

Published as a conference paper at ICLR 2020
RESTRICTING THE FLOW:
INFORMATION BOTTLENECKS FOR ATTRIBUTION
Karl Schulz"", Leon Sixt"", Federico Tombari', Tim Landgraf contributed equally ? work done at the Freie Universität Berlin
Technische Universität München' Freie Universität Berlin? Corresponding authors: karl. schulz tum.de, leon.sixt@ fu-berlin.de
ABSTRACT
Attribution methods provide insights into the decision-makin of machine learning models like artificial neural networks. For given input sample, they assign a relevance score to each individual input variable, such as the pixels of an image. this work, we adopt the information bottleneck concept for attribution. By adding noise to intermediate feature maps, we restrict the flow of information and can quantify (in bits) how much information image regions provide. We compare our method against ten baselines using three different metrics on VGG-16 16 and ResNet-50, and find that methods outperform all baselines in five out of six settings. The method's 's nnarmatiiontheoretie foundation provides an absolute frame of reference for attribution values (bits) and a guarantee that regions scored close to zero are not necessary for the network's 's decision.
INTRODUCTION
Deep neural networks have become state of the in many real- world applications. However, their increasing complexity makes difficult to explain the model's output. For some applications such as in medical decision making or autonomous driving, model interpretability an important requirement with legal implications. Attribution methods (Selvaraju al. 2017 Zeiler & Fergus 2014 Smilkov et al. 2017) aim to explain the model behavior by assigning relevance score to each input variable. When applied to images, the relevance scores can be visualized as heatmaps over the input pixel space, thus highlighting salient areas relevant for the network's decision.
Bits / Pixel
Figure I: Exemplary heatmap of the Per-Sample Bottleneck for the VGG-16.
For attribution, ground truth exists. If attribution heatmap highlights subjectively irrelevant areas, this might correctly reflect the network's unexpected way of processing the data, the heatmap might be inaccurate (Nie et al. 2018; Viering et al. 2019) Sixt et al. 2019). Given an image of a railway locomotive, the attribution map might highlight the train tracks instead of the train itself. Current attribution methods cannot guarantee that the network is ignroing the low-scored locomotive for the prediction.
We propose novel attribution method that estimates the amount of information an image region provides to the network's 's prediction. We use a variational approximation to upper-boune this estimate and therefore, can guarantee that areas with zero bits of information are not necessary for the prediction. (shows an exemplary heatmap of our method. Up to 3 bits per pixel are available for regions corresponding to the monkeys' faces, whereas the tree is scored with close to zero bits per pixel. We can thus guarantee that the tree is not necessary for predicting the correct class.
To estimate the amount of information, we adapt the information bottleneck concept (Tishby et al. 2000; Alemi et al.) 2017). The bottleneck is inserted into an existing neural network and restricts the information flow by adding noise to the activation maps. Unimportant activations are replaced almost entirely by noise, removing all information for subsequent network layers. We developed two approaches to learn the parameters of the bottleneck either using a single sample (Per- Sarr-e Bottleneck), the entire dataset (Readout Bottleneck).

---

Published as a conference paper at ICLR 2020
We evaluate against ten different baselines. First, we calculated the Sensitivity-n metric proposed by |ncona et al. (2018). Secondly, we quantified how well the object of interest was localized using bounding boxes and extend the degradation task proposed by |Ancona et al. (2017). In all these metrics, our method outperforms the baselines consistently. Additionally, we test the impact of cascading layer-wise weight randomizations
the attribution heatmaps (Adebayo et al. 2018). For reproducibility, we share our source code and provide an easy-to-usd implementation.
We name our method IBA which stands for Information Bottleneck Autribution. It provides a theoretic upper-bound on the used information while demonstrating strong empirical performance. Our work improves model interpretablility and increases trust in attribution results.
To summarize our contributions:
We adapt the information bottleneck concept for attribution to estimate the information used by the network. Information theory provides a guarantee that areas scored irrelevant are indeed not necessary for the network's s prediction.
We propose two ways Seayys-Per-Saeeeee Per-Sample and Readout Bottleneck - to learn the parameters of the information bottleneck.
We contribute a novel evaluation method for attribution based on bounding boxes and we also extend the metric proposed by [Ancoona et al. (2017) to provide a single scalar value and improve the metric's comparability between different network architectures.
2 RELATED WORK
Attribution is an active research topic. Gradient Maps (Bachrens et al. 2010) and Saliency Maps (Si- monyan Zisserman 2014) are based on calculating the gradient of the target output neuron w.r.t. to the input features. Integrated Gradient (Sundararajan et al. 2017) and SmoothGrad (Smilkov et al., 2017) improve over gradient-base based attribution maps by averaging the gradient of multiple inputs, either over brightness level interpolations in local neighborhood. Other methods, such Layer- wise Relevance Propagation (LRP) (Bach et al. 2015), Deep Taylor Decomposition (DTD) (Montavon) let all.|[2.. (2017), Guided Backpropagation (GuidedBP) (Springenberg et al. (014)) or DeepLIFT (Shriku mar et al. 2017) modify the propagation rule. PatternAttribution (Kindermans et al., 2018) builds upon DTD by estimating the signal's direction for the backward propagation.
certurbation-base methods are not based on backpropagation and treat the model as black-box. Occlusion (Zeiler Fergus 2014) measures the importance the drop in classification accuracy after replacing in- dividual image patches with zeros. Similarly, blurring image patches has been used to quantify the importance of nhighffrequence features (Greydanus al. 2018
Grad-Cam (Selvaraju et al., (017)) take the activations of the final convolutional layer to compute relevance scores. They also combine their method with GuidedBP: GuidedGrad- CAM. Ribeiro et al., (2016) uses image superpixels to explain deep neural networks. High-level concepts rather input pixels are scored by TCAV (Kim et al., 2018). KKaakaar et al. 2019) prune irrelevant neurons. Similar to our work, Macdonald et al. (2019) uses a rate- rate-distortion perspective, but minimize the norm of the mask instead of the shared information. To the best of our knowledge, we are the first to estimate the amount of used information for attribution purposes.
Although many attribution methods exist, no standard evaluation benchmark is established. Thus, determining the state of the art is difficult. The performance of attribution methods is highly dependent on the used model and dataset. Often only a purely visual comparison is performed (Smilkov et al.)
2017: Springeenerr al. 2014 Montavon al., 2017; Sundararajan et al. 2017 Bach et al. | 2015). The most commonly used benchmark is to degrade input images according to the attribution heatmap and measure the impact the model output. (Kindermans et al. [0088] (Samek al. [0166] Anconal cet al. (017).. The Sensitivity-n score (Ancona et al. 2018) is obtained by randomly masking the input image and then measuring the correlation between removed attribution mass and model performance. For the ROAR score (Hooker et al., 2018), the network is trained from scratch on the degraded images. computationally expensive but avoids oft-of-domain inputs an inherent problem of masking. Adebayo et al. (2018) proposes a sanity check: if the network's parameters are randomized, the attribution output should change too.
https::/// http://htt...c.ccooooooooooo///a
2

---

Published as a conference paper at ICLR 2020
Adding noise to signal reduces the amount of information (Shannon 1948) It is popular way to regularize neural networks (Srivastava et al. 2014, Kingma et al. For regularization, the noise is applied independently from the input and no attribution maps can be obtained. In Variational Autoencoders (VAEs) (Kingma & Welling. 2013), noise restricts the information capacity of the latent representation. In work, we construct similar information bottleneck that can be inserted into an existing network. Deep convolutional neural networks have been augmented with information bottlenecks before improve the generalization and robustness against adversarial examples (Achille & Soatto 2018 Alemi et al. (017). Zhmoginov et al. (2019) and Taghanaki et al. 2 extract salient regions using information bottlenecks. contrast to our work, they add the information bottleneck already when training the network and do not focus on post-hoc explanations.
2015; Gal et al. 2017).
INFORMATION BOTTLENECK FOR ATTRIBUTION
Instead of a backpropagation approach, we quantify the flow of information through the network in the forward pass. Given a pre-trained model, we inject noise into feature map, which restrains the flow of information through it. We optimize the intensity of the noise to minimize the information flow, while simultancously maximizing the original model objective, the classification score. The parameters of the original model are not changed.
3.1 INFORMATION BOTTLENECK FOR ATTRIBUTION
Generally, the information bottleneck concept (Tishby et al. 2000) describes a limitation of available information. Usually, the labels Y are predicted using all information from the input X. The information bottleneck limits the information to predict Y by introducing new random variable Z. The information the new variable Z shares with the labels Y is maximized while minimizing the information Z and X share
max YY; - 3r[x
(1)
where I [[x, Z] denotes the mutual information and / controls the trade-off between predicting the labels well and using little information of X. A common way reduce the amount of information is to add noise (Alemi et al. 2017 Kingma & Welling 2013).
For attribution, inject an information bottleneck into pretrained network. The bottleneck is inserted into an layer which still contains local information, e.g. for the ResNet the bottleneck is added after conv3.* * (after the last conv3 block). Let R denote the intermediate representations at this specific depth of the network, i.e. R = fi(X) where fi is the (-th layer output. We want to reduce information in R by adding noise. As the neural network trained already, adding noise should preserve the variance of the input to the following layers. Therefore, we also damp the signal R when increasing the noise, effectively replacing the signal partly with noise. In the extreme case, when no signal is transmitted, we replace R completely with noise of the same mean and variance as R. For this purpose, we estimate the mean lR and variance of of each feature of R empirically. As information bottleneck, we then apply a linear interpolation between signal and noise:
Z = ( (1 x(X))
(2)
where e (N(n, F and A(X) controls the damping of the signal and the addition of the noise.
The value of X is a tensor with the same dimensions as R and X; E [0, 1]. Given X₂(X) at the feature map location the bottleneck transmits all information Z4 R, Whereas if X; O. then Zi = and all information of R is lost and replaced with noise. It could be tempting to think that Z from equation 2 has the same mean and variance as R. This is not the case in general as X(X) and R both depend on X (for more detail see Appendix E).
In our method, we consider an area relevant if it contains useful information for classification. We need to estimate how much information Z still contains about R. This quantity is the mutual information [R, 7] that can be written as:
IR, 2] = ER ER[Dx,|P(//R)jIPES
where R)Z and P(Z) denote the respective probability distributions. We have no analytic expres- sion for P(Z) since would be necessary to integrate over the feature map p(z) fn (==))))(T)dr
3

---

Published as a conference paper at ICLR 2020
Sigmoid
e
CNNuRR
blur
imput
net-
R
AR+ + (1 (1-X)e A)e
-work
logits
bottleneck
Figure 2: Per-Sample Bottleneck: The mask (blue) contains a for cach r; in the intermediate feature maps R (green). The parameter a controls how much information is passed to the next layer. The mask is optimized for each sample individually according equation 6
intractable integral. It is common problem that the mutual information can not be computed ex- actly but instead approximated (Poole et al. 2019) Suzuki et al. (008). We resort to a variational approximation Q(Z) (NRR σR) which assumes that all dimensions of Z are distributed normally and independent a reasonable assumption, as activations after linear or convolutional layers tend have Gaussian distribution (Klambauer al., 2017). The independence assumption will gener- ally not hold. However, this only overestimates the mutual information as shown below. Substituting (Z) into the previous equation 3
we obtain:
IR, 2 = ER[DKuIP(ZIRIIIOCC - - http://DDDD///AARRRe/0----//P/P-
The derivation is shown in appendix D and follows Alemi et al. (2017). The first term contains the KL-divergence between two normal distributions, which is easy to evaluated and we use it to approximate the mutual information. The information loss function C, is therefore:
C [n( ((()))
We know that L, overestimates the mutual information, i.e. L1 IRR Z] the second KL- divergence term [KL[P(Z) [Q(()) has to be positive. If Cy is zero for an area, we can guarantee that information from this area is not necessary for the network's prediction. Information from this area might still be used when no noise added.
We aim to keep only the information necessary for correct classification. Thus, the mutual information should be minimal while the classification score should remain high. Let Leg be the cross-entropy of the classification. Then, obtain the following optimization problem:
LCE BLI,
where the parameter 3 controls the relative importance of both objectives. For small B, more bits of information flowing and less for higher 3. We propose two ways of finding the parameters À the Per-Sample Bottleneck and the Readout Bottleneck. For the Per-Sample Bottleneck, we optimize A for each image individually, whereas in the readout bottleneck, we train distinct neural network to predict A.
3.2 PER-S SER-LE BOTTLENECK
For the Per-Sample Bottleneck, we use the bottleneck formulation described above and optimize L for individual samples- - not for the complete dataset at once. Given a sample x, is fitted to the sample to reflect important and unimportant regions in the feature space. A diagram of the Per-Sample Bottleneck is shown 2
The bottleneck parameters À have to be in [0, 1]. To simplify optimization, parametrize = (a).. This parametrization allows a E igd and avoids any clipping of À to [0, 1] during optimization.
Parameterization
Initialization As when training neural networks, the initialization of parameters matters. In the beginning, want to retain all the information. For all dimensions we initialize a; = 5 and thus X; Pe 0.993 + Z R. At first, the bottleneck has practically no impact on the model performance. It then deviates from this starting point suppress unimportant regions.
4

---

Published as a conference paper at ICLR 2020
readout net.
c~ NNn, x
sigmoid
blur
imput
net-
R
XR+ (1- A)e
-work
logits
network
bottleneck
Figure 3: Readout Bottleneck: the first forward pass ( feature maps collected at different depths. The readout network uses a resized version of the feature maps to predict the parameters for the bottleneck layer. In the second forward pass (2) the bottleneck inserted and noise added. All parameters of the analyzed network are kept fixed.
Optimization We do 10 iterations using the Adam optimizer (Kingma & Ba 2014) with learning rate to fit the mask To stabilize the training, we copy the single sample 10 times and apply different noise each. In total, we execute the model 100 samples create a heatmap, comparable to other methods such as SmoothGrad. After the optimization, the model usually predicts the target with probability close 1, indicating that only negative evidence was removed.
Measure of information To measure the importance of each feature in Z, we evalu- (
per dimension. It shows where the information flows. We obtain -wo-dimensional heatmap by summing over the channel axis: mfh.u] As convolutional neural networks preserve the local-
Ei DKL (P( (1Q. ity in their channel maps, we use bilinear interpolation to resize the map to the image size. For the ResNet-50, insert the bottleneck after layer conv3 # Choosing later layer with lower spatial resolution would increase the blurriness of the attribution maps, due to the required interpolation. The effect of different depth choices for the bottleneck visualized in figure 4
Enforcing local smoothness Pooling operations and convolutional layers with stride greater than I are ignoring parts of the input. The ignored parts cause the Per-Sample Bottleneck overfit grid structure (shown in Appendix[1 [B. To obtain robust and smooth attribution map. we convolve the sigmoid output with fixed Gaussian kernel with standard deviation o,, Smoothing the mask during training is not equivalent to smoothing the resulting attribution map, as during training also the gradient averaged locally. Combining everything, the parametrization for the Per-Sample Bottleneck is:
A = blur(a,, sigmoid(a))
3.3 READOUT BOTTLENECK
For the Readout Bottleneck, we train second neural network to predict the mask a. contrast to the Per-Sample Bottleneck, this model is trained on the entire training set. In Figure the Readout Bottleneck depicted. Kümmerer al. (2014) introduced the readout concept for gaze prediction. The general idea is to collect feature maps from different depths and then combine them using 1x1 convolutions.
In first forward pass, noise is added and we collect the different feature maps. As the spatial resolution of the feature maps differs, we interpolate them bilinearly to match the spatial dimensions of the bottleneck layer. The readout network then predicts the information mask based on the collected feature maps. In a second forward pass, we insert the bottleneck layer into the network and restrict the flow of information.
Except for the learned importance values, the Readout Bottleneck is identical to the mechanism the Per-Sample Bottleneck. The measure of information works in the same way for the Per-Sample Bottleneck and we also use the same smoothing. Given a new sample, we can obtain a heatmap by merely collecting the feature maps and executing the readout network.

---

Published as a conference paper at ICLR 2020
10
0.10
04
conno .4
10
0.06
0.0
0.00
B = 0.1/k
8 == 1/k
8 10/k
8 = 100/k
8 = 1000
p= 0.99
0.97
p 0.95
0.65
p 0.00
0 = 10
20
convl
conv2.3
conv3.4
conv4 6
conv5.3
Figure 4: Effect of varying layer depth and 3 on the Per-Sample Bottleneck for the ResNet-50. The color bars measure the bits per input pixel. The resulting output probability the correct class p = ply 2, 3) is decreasing for higher 3.
The readout network consists of three convolutional layers. ReLU activations are applied between convolutional layers and a final sigmoid activation yields À E [0, 1]. As the input consists of upscaled feature maps, the field-of-view is large, although the network itself only uses 1xl conv. kernels.
EVALUATION
4.1 EXPERIMENTAL. SETUP
As neural network architectures, we selected the ResNet-50 (He et al., 2016) and the VGG-16 Simonyan & Zisserman 2014), using pretrained weights from the torchvision package (Paszke et al. (017). These two models cover range of concepts: dimensionality by stride max- moolino.ing. residual connections, batch normalization, dropout, low depth (1--weight-layer VGG), and high depth (50-weight-laye ResNet). This variety makes the evaluation less likely to overfit specific model type. They are commonly used in the literature concerning attribution methods. For PatternAttributt on the VGG-16, we obtained weights for the signal estimators from Kindermans et al. (0018).
As naive baselines, we selected random attribution, Occlusion with patch sizes 8x8 and 14x 14, and Gradient Maps. SmoothGrad and Integrated Gradients cover methods that accumulate gradients. We include three methods with modified backpropagation rules: PatternAttribution GuidedBP, and LRP. As implementation of PatternAttribul and LRP does not support skip connections, report no results for them on the ResNet-50. We also include Grad-CAM and its combination with GuidedBP, GuidedGrad-CAM.
For the compared methods, we use the hyperparameters suggested by the original authors. For LRP (Bach et al., 20155), different rules exist. We include the most commonly used a= 8=0, e=0 rule which is also displayed in the figures. We also include a=0, 3=-1, e=5 as it gives better results on the bounding-box task. For sensitivity-n and sanity checks, only the later LRP variante is evaluted. For our methods, the hyperparameters are obtained using grid search with the degradation metric as objective (Appendix F.. The readout network trained on the training of the ILSVRC12 dataset (Russakovsky et al. 2015) for E 20 epochs.
The optimization objective of the bottleneck LCE + BL, as given in equation 6 Generally, the information loss L, is larger than the classifier loss by several orders of magnitude it sums over height h, width w and channels c. We therefore use = hwee as a reference point to select 3. In figure 4 we display the effect of different orders of 3 and the effect of layer depth for ResNet-50 (see Appendix for VGG-16) A uniform uninformative heatmap is obtained for 8 2 1000/k all information gets discarded, resulting in output probability of 0 for the correct class. The heatmap for 3 = 0.1/K 4 is similar to 3 = 1/k but more information is passed, ie. less noises is added. In appendix F also compare pre- to post-training accuracy and estimate the mutual information for

---

Published as a conference paper at ICLR 2020
(a) Gradient
(b) Saliency
(c) SmoothGrad
(d) Int. Grad.
(c) GuidedBP
(f) Occlusion 14
(g) Grad-CAM (h) G.Grad-CAM (i) PatternAttr.
(i) LRP
(k) Per-Sample
(l) Readout
Figure 5: Heatmaps of all implemented methods for the VGG-16 (see Appendix for more).
both Bottleneck types. For the Per-Sample Bottleneck, we investigate 3= 1/k, 10/k, 100/k. The Readout network is trained with the best performing value /3 = 10/k.
4.2 QUALITATIVE ASSESSMENT
the heatmaps of all evaluated samples are shown (for more samples, see Appendix AA. Subjectively, both the Per-Samppll Sample and Readout Bottleneck identify areas relevant to the classification well. While Guided Backpropagation and PatternAttribution tend highlight edges, the Per- Per-Sample Bottleneck focuses on image regions. For the Readout Bottleneck, the attribution is concentrated little more on the object edges. Compared to Grad-CAM both our methods are more specific, i.e.
In Figure 5
fewer pixels are scored high.
4.3 SANITY CHECK: RANDOMIZATION OF MODEL PARAMETERS
Adebayo et al. 2018) investigates the effect of parameter randomization on attribution masks. A sound attribution method should depend the entire network's 's parameter set. Starting from the last layer, an increasing proportion the network parameters is re-initialized until all parameters random. We excluded Pattern Attribntntti the randomization would require to re-estimate the signal directions. The difference between the original heatmap and the heatmap obtained from the randomized model is quantified using SSIM (Wang et 2004). We discuss implementation details in the appendix G.I
In figure 6 we display the results of the sanity check. For our methods, we observe that randomizing the final dense layer drops the mean SSIM by around 0.4. The values for the Readout Bottleneck are of limited expressiveness as we did not re-train after randomization. For SmoothGrad and Int. Gradients, the SSIM drops by more than 0.4. The heatmaps of GuidedBP and LRP remain similar even if large portion of the network's 's parameters are randomized they do not explain the network prediction faithfully, Nie et al. (2018) provides theoretical insights about why GuidedBP fails. Sixt ct al. 2019) analyzes why LRP and other modified BP methods fail.
4.4 SENSITIVITY-N
Ancona al. 2018) proposed Sensitivity-n evaluation metric for attribution methods. Sensitivity-n masks the network's input randomly and then measures how strongly the amount of attribution in the mask correlates with the drop in classifier Given set T. containing randomly selected pixel indices, Sensitivity-n measures the Pearson correlation coefficient:
COIT Σ Rx(z), Se(x) - Se nj)
VET.
where S,(x) is the classifier logit output for class c, R₁ is the relevance at pixel and Dir,, =0) denotes the input with all pixels in Tn set zero. As in the original paper, we pick the number of masked
7

---

Published as a conference paper at ICLR 2020
ResNet-50
VGG-16
Sanity Check SSIM
1.0
Per-Sam.
Guided
0.8
Per-Sam. p=10
Int. Grad.
Per-Sam. B=100KK
Readout
PattermAttr.
Uniform Noise
0cc1-14xx1
Smoothgrad
OceL-8x88
0.2
Grad-CAN
Saliency
GuidedGrad-Cne
Gradient
0.0
last 86
cODVS
conv4
conv3
conV2
conv
last fc
classifor
convs
conv4
conv2
convt
ResNet-50 (1x1)
ResNet-50 (8x8)
VGG-16 (8x8)
0.4
Sensitivity-n
0.3
0.1
10
10P
10a
10
10a
10
10a
replaced pixels
replaced pixels
replaced pixels
Figure 6: First row drop in SSIM score when network layers are randomized. Best viewed color. Second-row Sensitivity-n scores for ResNet-50 and VGG-16. For the ResNet-50, also tile size 1x1 is shown. We clipped the y-axis at 0.4 to improve discriminability.
pixels in log-scale between 1 and 80% of all pixels. For each generate 100 different index sets T and test each on 1000 randomly selected images from the validation set. The correlation is calculated over the different index and then averaged over all images.
In Figure 6 the Sensitivity-n scores are shown for ResNet-50 and VGG-16. When masking inputs pixel-wise done in Ancona al. (018)), the Sensitivity-n score across methods not well discriminable, all scores range within the lower 10% of the scale. We ran the metric with a tile size of 8x8 pixels. This resulted in an increase of the Sensitivity-n score to 30% of the scale. Although shown in the figure due to the zooming, Occlusion-8xx 8x8 performs perfectly for small as its relevance scores correspond directly to the drop logits per 8x8 tile. We find that the Per-Sample Bottlenecks 3 = 10/k k perform best for both models above n = 2 10 pixels, i.c. when more then 2% of all pixels are masked.
4.5 BOUNDING BOX
To quantify how well attribution methods identify and localize the object of interest, rely bounding boxes available for the ImageNet dataset. Bounding boxes may contain irrelevant areas, in particular for non-rectangular objects. We restrict evaluation images with bounding boxes covering less than 33% of the input image. In total, we run the bounding box evaluation on 11,849 images from the ImageNet validation
the bounding box contains n pixels, we measure how many of the n-th highest scored pixels are contained in the bounding box. By dividing by n. we obtain ratio between 0 and The results are shown in Table | The Per-Sample Bottleneck outperforms all baselines on both VGG-16 and ResNet-50 each by margin of 0.152.
An alternative metric would be to take the sum of attribution in the bounding box and compare it to the total attribution in the image. We found this metric robust against extreme values. For the ResNet-50, we found basic Gradient Maps to be the best method as few pixels receiving extreme scores are enough dominate the sum.
4.6 IMAGE DEGRADATION
As further quantitative evaluation, rely the degradation task used by Ancona al. (2017): Kindermans et al. (2018;; Hooker et al. (2018); (Samek et al. (2016). Given an attribution heatmap, the input split in tiles, which are ranked by the of attribution values within each corresponding
8

---

Published as a conference paper at ICLR 2020
Model & Evaluation
ResNet-50 deg.
VGG-16 deg.
ResNet
VGG
8x8
14x14
8x8
14x14
bbox
bbox
Random
0.000
0.000
0.000
0.000
0.167
0.167
Occlusion-87e 8x8
0.162
0.130
0.267
0.258
0.296
0.312
Occlusion-1444 14
0.228
0.231
0.402
0.404
0.341
0.358
Gradient
0.002
0.005
0.001
0.005
0.259
0.276
Saliency
0.287
0.305
0.326
0.362
0.363
0.393
GuidedBB
0.491
0.515
0.460
0.493
0.388
0.373
PattemAAtribution
0.440
0,457
0,404
LRP = -1,,,-00
0.471
0.486
0.397
LRP a - 00 00,
0.462
0.467
0,441
Int. Grad.
0.401
0.424
0.420
0.453
0.372
0.396
SmoothGrad
0.485
0.502
0.438
0.455
0.439
0.399
Grad-CAM
0.536
0.541
0.510
0.517
0.465
0.399
GuddedGrad-CAM
0.565
0.577
0.555
0.576
0.468
0.419
IBA Per-Sample 3 1
0.573
0.573
0.581
0.583
0.606
0.566
IBA Per-Sample 8 10/k
0.572
0.571
0.582
0.585
0.620
0.593
IBA Per-Sample 3 3= 100/k
0.534
0.535
0,542
0.545
0.574
0,568
IBA Readout 3 10/k
0.536
0.536
0.490
0.536
0.484
0.437
Table I: Degradation (deg.): Integral between LeRF and MoRF in the degradation benchmark for different models and window sizes over the ImageNet test set. Bounding Box (bbox): the ratio of the highest scored pixels within the bounding box. For ResNet-50, we show no results for PatternAtributti and LRP as no Py PoTccc implementation supports skip-connections.
tile of the attribution. At each iteration, the highest-ranked tile is replaced with a constant value, the modified input is fed through the network, and the resulting drop in target class score measured. steep descent of the accuracy curve indicates meaningful attribution map.
When implemented in the described way, the most relevant tiles are removed first (MoRF). However, Ancona et al. (2017)
argues that using only the MoRF curve for evaluation is not sufficient. For the MoRF score, it is beneficial to find tiles dis- rupting the output the neural network as quickly as possible. Neural networks
s(x) normalizze score
LeRF
score
0.6
sensitive to subtle changes in the input (Szegedy et al. 2013). The tiles do not necessarily have to con- tain meaningful information to disrupt the network. Ancona et al. (0117) proposes to invert the degradation direction, re- moving tiles ranked as least relevant by the attribution method first (LeRF). The LeRF task favors methods that identify ar- eas sufficient for classification. We scale the network's output probabilities to be in [0, 1]:
0.4
40
00
100
X deegragation of [%)
Figure 7: Mean MoRF and LeRF for the Per-Sample Bottleneck. The area is the final degradation score.
sxx) p(y b
t
where & the model's average top-1 probability on the original samples and b the mean model output on the fully degraded images. Both averages are taken over the validation set. A score of corresponds to the original model performance.
Both LeRF and MoRF degradation yield curves as visualized in Figure 7 measuring different qualities of the attribution method. To obtain a scalar measure of attribution performance and to combine both metrics, propose calculate the integral between the MoRF and LeRF curves.
The results for all implemented attribution methods on the degradation task are given in Table [ We evaluated both models on the full validation set using 8x8 and 14x14 tiles. In Appendix H we show the mean LeRF and MoRF curves for 14x14 tiles. The Per-Sample Bottleneck outperforms all other methods in the degradation benchmark except for GuidedGrad-CAr CAM ResNet-50 where scores comparably (score difference 0.004). The Readout Bottleneck generally achieves a lower degradation scores but still perform competitively.

---

Published as a conference paper at ICLR 2020
5 CONCLUSION
We propose two novel attribution methods that return an upper bound on the amount of information each input region provides for the network's decision. Our models' core functionality is bottleneck layer used to inject noise into a given feature layer and mechanism to learn the parametrized amount of noise per feature. The Per-Sample Bottleneck is optimized per single data point, whereas the Readout Bottleneck is trained on the entire dataset.
Our method does not constrain the internal network structure. In contrast to several modified backpropagation methods, it supports any activation function and network architecture. To evaluate our method, we extended the degradation task to quantify model performance deterioration when removing both relevant and irrelevant image tiles first. We also show results on how well ground-trun bounding boxes are scored. Our Per-Sample and Readout Bottleneck both show competitive results on all metrics used, outperforming state of the art with significant margin for some of the tasks.
Generally, we would advise using the Per-Sample Bottleneck over the Readout Bottleneck. performs better and is more flexible it only requires estimate the mean and variance of the feature map. The Readout Bottleneck has the advantage of producing attribution maps with a single forward pass once trained. Images with multiple object instances provide the network with redundant class information. The Per-Sample Bottleneck may discard some of the class evidence. Even for single object instances, the heatmaps of the Per-Sample Bottleneck may vary slightly due to the randomness of the optimization process.
The method's information-theorels foundation provides guarantee that the network does require regions of zero-valued attribution for correct classification. To our knowledge, our attribution method is the only one to provide scores with units (bits). This absolute frame of reference allows quantitative comparison between models, inputs, and input regions. We hope this contributes to deeper understanding of neural networks and creates trust to use modern models sensitive areas of application.
ACKNOWLEDGEMEN
We want to thank our anonymous reviewers for their valuable suggestions. In particular, we thank reviewer 1 for encouraging to include the sanity checks. We indebted to B. Wild, A. Elimelech, M. Granz, and D. Dormagen for helpful discussions and feedback. For LRP, we used the open source implementation by M. Böhle and F. Eitel (Böhle al. (019) and thank A-K. Dombrowski for sharing an implementation and patterns for PatternAttribution with us. We thank Chanakya Ajit Ekbote for pointing out mistake in equation 4 LS was supported by the Elsa-Neumann Soumann-Scholarshig by the state of Berlin. We are also grateful to Nvidia for providing us with a Titan Xp and to ZEDAT for granting us access their HPC system.
10

---

Published as a conference paper at ICLR 2020
REFERENCES
Alessandro Achille and Stefano Soatto. Information dropout: Learning optimal representations through noisy computation. IEEE transactions on pattern analysis and machine intelligence, 40 (12):2897-2905, 2018.
Julius Adcbayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. In Advances in Neural Information Processing Systems, pp. 9505-9515, 2018.
Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information bottleneck. In 5th International Conference Learning Representations (ICLR 2017), 2017.
Marco Ancona, Enea Ceolini, Cengiz Öztireli, and Markus Gross. unified view gradient-based attribution methods for deep neural networks. In NIPS 2017- 2017-Workshop Interpreting, Explaining and Visualizing Deep Learning. ETH Zurich, 2017.
Marco Ancona, Enea Ceolini, Cengiz. Oztireli, and Markus Gross. Towards better understanding of gradient-based attribution methods for deep neural networks. 6th International Conference on Learning Representations (ICLR 2018), 2018.
Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, -oburttobert Müller, and Wojeiech Samek. On tixel-wise explanations for non-linear classifier decisions by layer-wist relevance propagation. PLoS ONE, 10(7), 2015.
David Baehrens, Timon Schroeter, Stefan Harmeling, Motoaki Kawanabe, Katja Hansen, and Klaus- Robert Müller. How explain individual classification decisions. Journal of Machine Learning Research, 110un):1833-1833 2010.
Moritz Böhle, Fabian Eitel, Martin Weygandt, and Kerstin Ritter. Layer-wise relevance propagation for explaining deep neural network decisions in mri-based alzheimer's disease classification. Frontiers in Aging Neuroscience, 11:194, 2019, ISSN 1663-4365. doi: 10.3389ffnggi.20 0tt99//fnaii211.00111111 9,00194.
Yarin Gal, Jiri Hron, and Alex Kendall. Concrete dropout. In Advances in Neural Information Processing Systems, pp. 3581-53180 3581-3 2017.
Samuel Greydanus, Anurag Koul, Jonathan Dodge, and Alan Fern. Visualizing and understanding atari agents. In International Conference on Machine Learning, pp. 1787-1796, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770-778, 2016.
Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, and Been Kim. Evaluating Feature Importance Estimates. arXiv e-prints, 2018.
Ashkan Khakzar, Soroosh Baselizadeh, Saurabh Khanduja, Scong Tae Kim, and Nassir Navab. Explaining neural networks via perturbing important learned features. arXiv preprint /9t1///9//11000/ 2019.
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International Conference Machine Learning, pp. 2668-2677. 2018.
Pieter-Jan Kindermans, Kristof T. Schütt, Maximilian Alber, Klaus-Robert Müller, Dumitru Er- han, Been Kim, and Sven Dähne. Learning how to explain neural networks: Patternnet and patternatrrbutton. In International Conference on Learning Representations, 2018.
Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. arXiv e-prints, 2014.
Diederik Kingma and Max Welling. arXiv: 2013.
Auto-encoding variational bayes.
arXiv preprint
11

---

Published as a conference paper at ICLR 2020
Durk P Kingma, Tim Salimans, and Max Welling. Variational dropout and the local reparameteriization trick. In Advances in Neural Information Processing Systems, pp. 2575-2582 2015.
Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. self-normalizing neural networks. Advances in neural information processing systems, pp. 971-980, 2017.
Matthias Kümmerer, Lucas Theis, and Matthias Bethge. Deep gaze Boosting saliency prediction with feature maps trained on imagenet. arXiv:1411.106 [cs, q-bio, stat], 2014.
Jan Macdonald, Stephan Wäldchen, Sascha Hauch, and Gitta Kutymiok. A dite-distontion framework for explaining neural network decisions. arXiv preprint arXiv: 19X.v::905..1099 2019.
Grégoire Montavon, Sebastian Lapuschkin, Alexander Binder, Wojciech Samek, and Klaus-Robers Müller. Explaining nonlinear classification decisions with deep taylor decomposition. Pattern Recognition, 65:21 2017.
Weili Nie, Yang Zhang, and Ankit Patel. A theoretical explanation for perplexing behaviors of backpropagation-basel visualizations. In International Conference on Machine Learning, pp. 3809-3818, 2018.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in PyTorch. In NIPS Autodiff Workshop, 2017.
Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceed- ings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 5171-5180. PMLR, 2019.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Why should trust you?: Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1135-11444 ACM, 2016.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115 (3):211-252, 2015.
Wojeiech Samek, Alexander Binder, Grégoire Montavon, Sebastian Lapuschkin, and Müller. Evaluating the visualization of what a deep neural network has learned. IEEE transactions on neural networks and learning systems, 28( (1 (8(11))66602675 2016.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In Proceedings of the IEEE International Conference on Computer Vision, pp. 618-68 06-26, 2017.
Claude Elwood Shannon. A mathematical theory of communication. Bell system technical journal, 27(3):379443 423, 1948.
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In Proceedings of the 34th International Conference on Machine Learning-Volas Volume 70, pp. 3145-3153. JMLR. 2017.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv=1409 1556, 2014.
Lcon Sixt, Maximilian Granz, and Tim Landgraf. When Explanations Lie: Why Many Modified BP Attributions Fail. arXiv preprint arXiv: /tt. 2019.
Daniel Smilkov, Nikhil Thorat, Been Kim, Fermanda Viégas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise. arXiv: /706.00067722 [cs, stat], 2017.
12

---

Published as a conference paper at ICLR 2020
Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for Simplicity: The All Convolutional Net. arXiv e-prints, 2014.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, llya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-1958 2014.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 3319-3328 JMLR. org, 2017.
Taiji Suzuki, Masashi Sugiyama, Jun Sese, and Takafumi Kanamori. Approximating mutual informa- tion by maximum likelihood density ratio estimation. New challenges for feature selection in data mining and knowledge discovery, pp. 5-20, 2008
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint 0rXiv:1312.61999 2013.
Saeid Asgari Taghanaki, Mohammad Havaei, Tess Berthier, Francis Dutil, Lisa Di Jorio, Ghassan Hamarneh, and Yoshua Bengio. Infomask: Masked variational latent representation to localize chest discase. In Dinggang Shen, Tianming Liu, Terry M. Peters, Lawrence H. Staib, Caroline Essert, Sean Zhou, Pew-Thian Yap, and Ali Khan (eds.), Medical Image Computing and Computer Assisted Intervention MICCAI 2019, pp. 739-747, Cham, 2019. Springer International Publishing. ISBN 978-3-030-32226-7.
Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. arXiv preprint physics/0004057, 2000.
Tom Viering, Ziqi Wang, Marco Loog, and Elmar Fisemann. How Manipulate CNNs to Make Them Lie: the GradCAM Case.
[cs), August 2019. URL http://axxiv
hxg/abb/1907..c 10901
Zhou Wang, Alan C Bovik, Hamid R Sheikh, Eero P Simoncelli, et al. Image quality assessment: From error visibility structural similarity. IEEE transactions image processing, 13(4): 600-612, 2004.
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference on computer vision, pp. 818 833.. Springer, 2014.
Andrey Zhmoginov, lan Fischer, and Mark Sandler. Information-bottlenedd approach self-attention. In ICML 2019 Workshop on Self-Supervised learning, 2019.
13

---

Published as a conference paper at ICLR 2020
A
VISUAL COMPARISON OF ATTRIBUTION METHODS
Inpux
Grad-CA ResNet-50
Ocel-14 ResNect50
Sumple ResNer-50
Grad-CAM VGG-16
Per-Sample VOG-16
RlsNer
Reshet-50
VGG.1
VGG-16
VGG-16 16
Weimarater
Barow
Capachin
Mtspod
Soup dispe.
Jackk o -ant.
Zocchini
Broum
Bagel
Fridh
Sloth bear
Pascho
Siglissh..
Perfume
Redshank
Figure 8: Blue indicates negative relevance and red positive. The authors promise that the samples were picked truly randomly, no sherry-picking no hww.ppllaagaaddds-ooooo-o-----ee
14

---

Published as a conference paper at ICLR 2020
B GRID ARTIFACTS WHEN NOT USING SMOOTHING
=1=
(b)
(c) T 10
(d) T' 100
Figure 9: Development of DKL (Q()) the Per-Sample Bottleneck for layer convl.3 of the ResNet-50. Red indicate areas with maximal information flow and semi-treae transparnpp green for zero information flow. Top row: without smoothing the mask exhibits a grid structure. Bottom row: smoothing with 0s = 2. The smoothing both prevents the artifacts and reduces overfitting small areas.
C EFFECTS OF DIFFERENT 3 AND LAYER DEPTH FOR THE VGG-16
0.2
conv4
255
8 0.1/K
8 1/k
10/k
100/k
8 1000 /k:
== 0.99
= 0.95
p == 0.80
= 0.55
== 0.00
10
0.4
0.
0.2
0.0
0,0
conv
conv2_1
conv4_1
conv5_1
conv5_3
Figure 10: Effect of varying layer depth and on the Per-Sample Bottleneck for the VGG-16. The resulting ouput probability for the correct class is given p.
15

---

Published as a conference paper at ICLR 2020
D DERIVATION OF THE UPPER-BOUND OF MUTUAL INFORMATION
For the mutual information IR, 21, we have:
RR, /|
((]])
(10)
1 p(r)
(f8 p(2]r) log p(z|r) dz dr
(11)
p(z)
J J2
p(2|r) q(=) dzdr
p(r, 2) log
(12)
p22)
q(2)
S It / p(r, log
p(zlr) dzdr + 1 p(r, (1r:))oog log g(z)
dzdr
(13)
(())
p(z)
p(2|r) dedr + { p(=)) ( p(rl))ar) log 9(2)
p(r, 2) log
de
(14)
Q(2)
p(2)
ER [DRL [ ((Z -
(15)
(16)
E MEAN AND VARIANCE OF Z
The X(X) linearly interpolate between the feature map R and the noise ~ NNun, where un and n are the estimated mean and standard derivation of R. Both = fi(X) and A(X) depend on the input random variable X.
- X(X)R X(X))e
(17)
For the mean Z, we have:
E|Z] ([((X) R| [[(L X(X))
substituting in definition of Z independence of X and e
E[A(X)R) R + Ei1 [Ed
cov()(X), R) + (|X(X)]E(R) + E[1 [(
cov (A, B) E[AB] [[
cov(A(X), R) + E(c]
> E|c] μm = [[R]
As X(X) and R are multiplied together, they form complex product distribution. they do not correlate, E 2 E(e] E[A].
A similar problem araises for the variance:
Var Z] = E[ - 2
([(((X)R (1 -
+ B(e))²
The multiplication of X(X) and R causes in general the variance of Z and R to not match: Var [2 + Var [R].
F HYPERPARAMETERS
Parameter
ResNet-50 VGG-16
Search space
Target layer
conv3_4
conv4_1
Optimizer Learning Rate
Adam KKngmma & Ba (2014)
1)
(0.03, 0.1, 0.3, 1,3, 10) (0.001,0.01,,. 10, 100, 300) [1,3, 5, 10, 100)
Balance Factor
B = 10/k
Iterations
10
Batch Size
B = 10
[1,5, 10, 30)
Smoothing
0, 1
[0.5, 1,
Table 2: Hyperparameters for Per-Sample Bottleneck. The layer notations for the ResNet-50 are taken from the original publication (He ct al. 2016). The first index denotes the block and the second the layer within the block. For the VGG-16, conv_n denotes the n-th convolutional layer.
16

---

Published as a conference paper at ICLR 2020
Orig.
Initial
8 0.01/
0.1/k
1/k
10/k
100
Li/k Top-5 Acc.
2.500
3.174
0.525
0.248
0.098
0.019
Per-Sample
0.928
0.928
1.000
0.971
1.000
0.990
0.522
Top-1 Acc.
0.760
0.760
1.000
0.963
1.000
0.984
0.427
Li/k Top-5 Acc. 0.928
2.500
1.822
0.628
0.222
0.079
0.023
Readout
0.928
0.930
0.928
0.917
0.870
0.505
Top-1 Acc.
0.760
0.760
0.761
0.756
0.735
0.660
0.302
Table 4: Influence of 3 on the information loss L₁ and the test accuracy on ResNet-50, h is the size of the feature map, i.e. k hwc. Initial: Configuration of the untrained bottleneck with 5. Original: Values for the original model without the bottleneck.
Parameter
ResNet-50
VGG-16
Search space
Target layer Reading out
conv2
conv3.1
conv2 3
conv3.1
conv3.4
conv4_1
conv4 6
conv5_1
conv5_3
conv5.3
fc
fc
Optimizer
Adam (Kingma & Ba (2014)
Learning Rate
!) 10
(e-4, e-6}
Balance Factor
3 10/k
(0.1/k, 1/k, 10/k 100/k}
Epochs
E 10
Batch Size
B 16
Smoothing
dg
Table 3: Hyperparameters for the Readout Bottleneck.
In Table 2 we provide hyperparameters for the Per-Sample Bottleneck. In Table provide hyperparameters for the Readout Bottleneck.
In T comparison of pre- post-training accuracy and the estimated mutual information is shown for both Bottleneck types. The Per-Sample Bottleneck can learn to suppress negative evidence for each sample and the accuracy is close for 10/k: For 3 100/k, also positive evidence is discarded and the accuracy decreases to 0.43. The Readout Bottleneck learns to suppress negative evidence for small 3 < 0.1/k / and slightly improves the final accuracy.
G EVALUATION METRICS
G.1 SANITY CHECK: WEIGHT RANDOMIZATION
We use the cascading parameter randomization sanity check from/Adebaye et al. (2018). Following the original paper, we used the skimage SSIM implementation with window of size 5. For LRP. we found that the weight randomization flips the values of the saliceny heatmap, e.g. h, hh where h, the heatmap with random weights and ho the heatmap on the original model. Therefore for LRP, used: maxxssimm(f
normalize(h., (h,)) ssim (h,, Normalize(-h,-)) by first clamping the 1-th and 99-th percentile and then rescaling the heatmap it to [0, 1]. We run the sanity check on 200 randomly selected samples from the ImageNet valdiation set.
We normalized the heatmaps
17

---

Published as a conference paper at ICLR 2020
H MORF AND LERF DEGRADATION PATHS
A0 MOsIlted SOUCS
400 RECrad
Gegiapottie of [6]
Z-dogtogicoo of DS)
Orgragatier ofx [4]
2- Z-dggggicioo ofx [4]
degiagatoo of DS3
Random
Occlinionn88 8x8
Orclosion-
(d) Gradient
(c) Saliency
Copearation [S]
Z-degnipeso [5g
deggegpttionn
--geregatton (5g
Guided (g)
Smoochgrad
① CaAMeGGrad-CMM
Inceaated serure
(k) Par-Snngie 3
1/k
(1) Per-Sample
@
10/k
(m)) Pan-Senn
3 100/k
(n) Readom 3 10/k
Figure 11: MoRF and LeRF for the ResNet-50 network using 14x14 tiles.
500
Ceggraaatte <-dagnipesm [5g
Dagragatior
d-degrigation
deggnggeteen
Occlasion-8st
Occlusion- 14x14
(d) Gradient
(c) Saliency
(l) Guided HP
PitternAtt.
(b) LRP
( Int. Grad.
Smoothgrad
of (b9
[8)
d-degrapetion
(k) Grad-CAM
(1) Guldee3Grad-CAM
(m) Par-Sauple 1/k
(n) Par-Sumple = 10/k
(0) Per-Sam. 100/k
CONpTICAtto of 24)
(p) Readout 10/k
Figure 12: MoRF and LeRF paths for the VGG-16 network using 14x14 tiles.
18


Reviews:
[{"review_id": "S1xbOKTJ9S", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nSummary\n---\n\n(motivation)\nLots of methods produce attribution maps (heat maps, saliency maps, visual explantions) that aim to highlight input regions with respect to a given CNN.\nThese methods produce scores that highlight regions that are in a vague sense \"important.\"\nWhile that's useful (relative importance is interesting), the scores don't mean anything by themselves.\nThis paper introduces another new attribution method that measures the amount of information (in bits!) each input region contains, calibrating this score by providing a reference point at 0 bits.\nNon-highlighted regions contribute 0 bits of information to the task, so they are clearly irrelevant in the common sense that they have 0 mutual information with the correct output.\n\n(approach - attribution methods)\nAn information bottleneck is introduced by replacing a layer's (e.g., conv2) output X with a noisy version Z of that output.\nIn particular, Z is a convex combination of the feature map (e.g., conv2) with Gaussian noise with the same mean and variance as that feature map.\nThe weights of the combination are found so they minimize the information shared between the input and Z and maxmimize information shared between Z and the task output Y.\nThese weights are either optimized on\n1) a per-image basis (Per-Sample) or\n2) predicted by a model trained on the entire dataset (Readout).\n\n(approach - evaluation)\nThe paper uses 3 metrics with differing degrees of novelty:\n1) The bbox metric rewards attribution methods that put a lot of mass in ground truth bounding boxes.\n2) The original Sensitivity-n metric from (Ancona et al. 2017) is reported with a version that uses 8x8 occlusions.\n3) Least relevant image degredation is compared to most relevant image degredation (e.g., from (Ancona et al. 2017)) to form a new occlusion style metric.\n\n(experiments)\nExperiments consider many of the most popular baselines, including Occlusion, Gradients, SmoothGrad, Integrated Gradients, GuidedBP, LRP, Grad-CAM, and Pattern Attribution. They show:\n1) Qualitatively, the visualizations highlight only regions that seem relevant.\n2) Both Per-Sample and Readout approaches put higher confidence into ground truth bounding boxes than all other baselines.\n3) Both Per-Sample and Readout approaches outperform all baselines almost all the time according to the new image degredation metric.\n\n\nStrengths\n---\n\nThe idea makes a lot of sense. I think heat maps are often thought of in terms of the colloquial sense of information, so it makes sense to formalize that intuition.\n\nThe related work section is very well done. The first paragraph is particularly good because it gives not just a fairly comprehensive view of attribution methods, but also because it efficiently describes how they all work.\n\nThe results show that proposed approaches clearly outperform many strong baselines across different metrics most of the time.\n\n\nWeaknesses\n---\n\n\n* I'm not sure why the new degredation metric is a useful addition. What does it add that MoRF and LeRF don't capture on their own independently?\n\n* I think [1] would be a nice addition to the evaluation section as it tests for something qualitatively different than the various metrics from section 4. It would also be a good addition to the related work.\n\n\nMissing Details / Points of Confusion\n---\n\n* I think there's an extra p(x) in eq. 11 in appendix D.\n\n* I think the variable X is overloaded. In eq. 1 it refers to the input (e.g., the pixels of an image) while in eq. 2 it refers to an intermediate feature map (e.g., conv2) even though it later seems to refer to the input again (e.g., eq. 3). Different notation should be used for intermediate feature maps and inputs.\n\n\nPresentation Weaknesses\n---\n\n* In section 3.1 is lambda meant to be constrained in the range [0, 1]? This is only mentioned later (section 3.2) and should probably be mentioned when lambda is introduced.\n\n* \"indicating that all negative evidence was removed.\" I think this should read \"indicating that only negative evidence was removed.\"\n\n\nSuggestions\n---\n\n\"The bottleneck is inserted into an early layer to ensure that the information in the network is still local\"\nI'd like this to be explored a bit more. Though deeper feature maps are certainly more spatially coarse they still might be somewhat \"local\". To what degree to they loose localization information? My equally vague alternative intuition goes a bit differently: The amount of relevant information flowing through any spatial location seems like it shouldn't change that much, only the way its represented should change. If the proposed visualizations were the same for every choice of layer then it would confirm this intuition. That would also be an interesting result because most if not all of the cited baseline approaches (where applicable) produce qualitatively different attributions at different layers (e.g., see Grad-CAM).\n\n\n[1]: Adebayo, Julius et al. “Sanity Checks for Saliency Maps.” NeurIPS (2018).\n\n\nPreliminary Evaluation\n---\n\nClarity: The paper is clearly written.\nOriginality: The idea of using the formal notion of information in attribution maps is novel, as is the bbox metric.\nSignificance: This method could be quite significant. I can see it becoming an important method to compare to.\nQuality: The idea is sound and the evaluation is strong.\n\nThis is a very nice paper in all the ways listed above and it should be accepted!\n\nPost-rebuttal comments\n---\n\nThe author responses and other reviews have only increased my confidence that this paper should be accepted.\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "BJxHG52nFB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\nThe paper proposes a novel perturbation-based method for computing attribution/saliency maps for deep neural network based image classifiers. In contrast to most previous work on perturbation-based attribution, the paper proposes to inject carefully crafted noise into an early layer of the network. Importantly, the noise is chosen such that it optimizes an information-theoretically motivated objective (rate-distortion/info bottleneck) that ensures that decision-relevant signal is flowing while constraining the overall channel-capacity, such that decision-irrelevant signal is blocked from flowing. The flow of signal is controlled by the amount of noise injected, which translates into a certain amount of mutual information between input image regions and noisy activations/features. This mutual information can be visualized in the input image, but it also has a clear, quantitative meaning that is readily interpretable. The paper introduces two ways to construct the injected noise, based on the information bottleneck. Resulting attribution maps are computed and evaluated on VGG-16 and ResNet-50 (on ImageNet), and are compared against an impressive number of previously proposed attribution methods. Importantly, the paper uses three different quantitative measures to compare the quality of attribution maps. The proposed method performs well on all three measures.\n\nContributions\ni) Derivation of a novel method for constructing attribution maps. Importantly, the method is grounded on solid theoretical footing for extracting minimal relevant information (rate-distortion theory / information bottleneck method).\n\nii) Proposal of a novel quantitative measure to compare quality of pixel-level attribution maps in image classification, and extension of a previously reported method.\n\niii) Evaluation and comparison against a large body of state-of-the-art attribution methods.\n\nQuality, Clarity, Novelty, Impact\nThe paper is clear and well written, with a nice introduction to the information bottleneck method. Experiments are well described and hyper-parameter settings are given in the appendix. To the best of my knowledge, the proposed method is sufficiently novel and the application of the information bottleneck framework to pixel-level attribution has not been reported before. Some of the design- and implementation-choices needed to render the intractable info bottleneck objective tractable could perhaps be discussed and potentially even improved in light of recent results in other fields (Bayesian DL, deep latent-variable generative models, and variational methods for deep neural network compression), but I currently don’t consider this a major issue. To me personally the work in convincing and mature enough to vote for acceptance - perhaps most importantly it lays important groundwork for important connections to the theory of relevant information and puts a lot of much needed emphasis on objective evaluation of attribution methods (i.e. without subjective visual judgement of saliency maps). My suggestions below are aimed at helping improve the paper even further.\n\n\nImprovements\nI) A short section of current shortcomings/limitations could be added to the discussion.\n\nII) Perturbation-based approaches that inject noise (into the input image directly) have been proposed previously. Most notably: Visualizing and Understanding Atari Agents, Greydanus et al. 2018 and potentially follow-up citations. It would be interesting to compare both works empirically, but perhaps also theoretically/conceptually. Could the Greydanus work be related to applying the noise directly to the input image along with some additional constraints?\n\n\nMinor Comments\na) Is there a particular reason for this choice of colormap? While it seems to be roughly perceptually uniform (which is of course good), why not choose a simple sequential colormap (instead of a rainbow-like one)? At least the use of red and green at the same time should rather be avoided to maximize colormap readability under the most common forms of color vision deficiencies.\n\nb) Just a pointer - no need to act on this for the current paper. Large parts of the field of neural network compression are concerned with a similar kind of attribution - the question is which weights/neurons/filters are relevant and which ones are not and can thus be removed from the network without loss in accuracy. Information-bottleneck style objectives (or the closely related ELBO / variational free energy) in conjunction with sparsity inducing priors have been proven to be quite fruitful. See e.g. Variational Dropout Sparsifies Deep Neural Networks, Molchanov et al. 2017 for interesting work, that aims at learning the variance of Gaussian noise that is injected into neural network weights using a similar construction and variational objective as shown in this paper. Perhaps some ideas can be borrowed/translated for future, improved versions of the method from that body of literature (Molchanov 2017, but also more sophisticated follow-up work).", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJeXnpJAKB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents an information-bottleneck-based approach to infer the regions/pixels that are most relevant to the output. For all the metrics listed in the paper, the proposed approaches all achieve very good performance. It turns out, the proposed two architectures are better (at least alternative) choices to the other existing attribution methods.\n\nI do agree that the proposed two models (Per-Sample and Readout) can be used to roughly infer regions of interest, which has been strongly supported by the comprehensive experiments. To minimize equation (6), we need to make beta*L_I small. Minimizing L_{CE} in (6) tries to maximize the mutual information between Z and output (labels); while minimizing L_I with respect to weight beta would try to inject noise to each dimension of Z. However, L_{CE} needs to ensure it can get enough information for prediction, and thus would prevent the noise injection process for “the key regions”. By choosing reasonable beta (similar to variational information bottleneck), the proposed approaches are capable to highlight key regions used for prediction.\n\nOverall, I think the method is elegant for approximately estimating the relevance score map.\nBelow are some of my (minor) questions/concerns:\n\n1. What we learned = What we want?\nThe proposed approach seeks a sort of “sparse heatmap”. \nThe larger the beta, the more regions/pixels would be suppressed while smaller beta might fail to suppress non-important regions in the image.\nIn the paper, the beta used for calculating the per-sample bottleneck is among [100/k , 10/k, 1/k].\nThe beta for ReadOut bottleneck is 10/k.\nHowever, according to Table 1, only when beta is smaller than 1/k, the accuracy of the model does not degrade too much. \nWhen using beta=10/k to get the \"heat map\" (where 10/k is the best choice of per-smaple bottleneck for degradation task), how close is the \"heat map in beta=10/k\" to the \"ground-truth heatmap\"?\nTo better understand the proposed methods, I have a small suggestion:\n------ Try betas in a broader range including very small betas, e.g. [0.0001/k, 0.001/k,....,1/k,10/k], for both Table one and visualization. \nFix a few images and visualize the heatmap given different betas.\nWe might better see how the growth of beta changes the heatmap.\n\n2. About zero-valued attributions.\nI agree with you that equation (5) is an upper bound of MI (eq (4)).\nHowever, I am not sure if I totally agree with the claim \"If L_1 is zero for an area, we can guarantee that no information from this area is used for prediction.\"\n----- Given L_1=0 really implies that no information of the corresponding region is used for the certain beta, but is this true for the original model (beta=0)? Table one shows that different beta would lead to very different downstream task accuracy.\n\n3. Specific to the two approaches you proposed, can you explain/motivate in what situations per-sample bottle would be better and in what cases we should prefer ReadOut bottleneck?\n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: All three reviewers strongly recommend accepting this paper. It is clear, novel, and a significant contribution to the field. Please take their suggestions into account in a camera ready version. Thanks!
Decision: Accept (Talk)