Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: 
Summary
---

(motivation)
Lots of methods produce attribution maps (heat maps, saliency maps, visual explantions) that aim to highlight input regions with respect to a given CNN.
These methods produce scores that highlight regions that are in a vague sense "important."
While that's useful (relative importance is interesting), the scores don't mean anything by themselves.
This paper introduces another new attribution method that measures the amount of information (in bits!) each input region contains, calibrating this score by providing a reference point at 0 bits.
Non-highlighted regions contribute 0 bits of information to the task, so they are clearly irrelevant in the common sense that they have 0 mutual information with the correct output.

(approach - attribution methods)
An information bottleneck is introduced by replacing a layer's (e.g., conv2) output X with a noisy version Z of that output.
In particular, Z is a convex combination of the feature map (e.g., conv2) with Gaussian noise with the same mean and variance as that feature map.
The weights of the combination are found so they minimize the information shared between the input and Z and maxmimize information shared between Z and the task output Y.
These weights are either optimized on
1) a per-image basis (Per-Sample) or
2) predicted by a model trained on the entire dataset (Readout).

(approach - evaluation)
The paper uses 3 metrics with differing degrees of novelty:
1) The bbox metric rewards attribution methods that put a lot of mass in ground truth bounding boxes.
2) The original Sensitivity-n metric from (Ancona et al. 2017) is reported with a version that uses 8x8 occlusions.
3) Least relevant image degredation is compared to most relevant image degredation (e.g., from (Ancona et al. 2017)) to form a new occlusion style metric.

(experiments)
Experiments consider many of the most popular baselines, including Occlusion, Gradients, SmoothGrad, Integrated Gradients, GuidedBP, LRP, Grad-CAM, and Pattern Attribution. They show:
1) Qualitatively, the visualizations highlight only regions that seem relevant.
2) Both Per-Sample and Readout approaches put higher confidence into ground truth bounding boxes than all other baselines.
3) Both Per-Sample and Readout approaches outperform all baselines almost all the time according to the new image degredation metric.


Strengths
---

The idea makes a lot of sense. I think heat maps are often thought of in terms of the colloquial sense of information, so it makes sense to formalize that intuition.

The related work section is very well done. The first paragraph is particularly good because it gives not just a fairly comprehensive view of attribution methods, but also because it efficiently describes how they all work.

The results show that proposed approaches clearly outperform many strong baselines across different metrics most of the time.


Weaknesses
---


* I'm not sure why the new degredation metric is a useful addition. What does it add that MoRF and LeRF don't capture on their own independently?

* I think [1] would be a nice addition to the evaluation section as it tests for something qualitatively different than the various metrics from section 4. It would also be a good addition to the related work.


Missing Details / Points of Confusion
---

* I think there's an extra p(x) in eq. 11 in appendix D.

* I think the variable X is overloaded. In eq. 1 it refers to the input (e.g., the pixels of an image) while in eq. 2 it refers to an intermediate feature map (e.g., conv2) even though it later seems to refer to the input again (e.g., eq. 3). Different notation should be used for intermediate feature maps and inputs.


Presentation Weaknesses
---

* In section 3.1 is lambda meant to be constrained in the range [0, 1]? This is only mentioned later (section 3.2) and should probably be mentioned when lambda is introduced.

* "indicating that all negative evidence was removed." I think this should read "indicating that only negative evidence was removed."


Suggestions
---

"The bottleneck is inserted into an early layer to ensure that the information in the network is still local"
I'd like this to be explored a bit more. Though deeper feature maps are certainly more spatially coarse they still might be somewhat "local". To what degree to they loose localization information? My equally vague alternative intuition goes a bit differently: The amount of relevant information flowing through any spatial location seems like it shouldn't change that much, only the way its represented should change. If the proposed visualizations were the same for every choice of layer then it would confirm this intuition. That would also be an interesting result because most if not all of the cited baseline approaches (where applicable) produce qualitatively different attributions at different layers (e.g., see Grad-CAM).


[1]: Adebayo, Julius et al. “Sanity Checks for Saliency Maps.” NeurIPS (2018).


Preliminary Evaluation
---

Clarity: The paper is clearly written.
Originality: The idea of using the formal notion of information in attribution maps is novel, as is the bbox metric.
Significance: This method could be quite significant. I can see it becoming an important method to compare to.
Quality: The idea is sound and the evaluation is strong.

This is a very nice paper in all the ways listed above and it should be accepted!

Post-rebuttal comments
---

The author responses and other reviews have only increased my confidence that this paper should be accepted.


review_title: Official Blind Review #1

final_score: rating: 8: Accept
review_content: This paper presents an information-bottleneck-based approach to infer the regions/pixels that are most relevant to the output. For all the metrics listed in the paper, the proposed approaches all achieve very good performance. It turns out, the proposed two architectures are better (at least alternative) choices to the other existing attribution methods.

I do agree that the proposed two models (Per-Sample and Readout) can be used to roughly infer regions of interest, which has been strongly supported by the comprehensive experiments. To minimize equation (6), we need to make beta*L_I small. Minimizing L_{CE} in (6) tries to maximize the mutual information between Z and output (labels); while minimizing L_I with respect to weight beta would try to inject noise to each dimension of Z. However, L_{CE} needs to ensure it can get enough information for prediction, and thus would prevent the noise injection process for “the key regions”. By choosing reasonable beta (similar to variational information bottleneck), the proposed approaches are capable to highlight key regions used for prediction.

Overall, I think the method is elegant for approximately estimating the relevance score map.
Below are some of my (minor) questions/concerns:

1. What we learned = What we want?
The proposed approach seeks a sort of “sparse heatmap”. 
The larger the beta, the more regions/pixels would be suppressed while smaller beta might fail to suppress non-important regions in the image.
In the paper, the beta used for calculating the per-sample bottleneck is among [100/k , 10/k, 1/k].
The beta for ReadOut bottleneck is 10/k.
However, according to Table 1, only when beta is smaller than 1/k, the accuracy of the model does not degrade too much. 
When using beta=10/k to get the "heat map" (where 10/k is the best choice of per-smaple bottleneck for degradation task), how close is the "heat map in beta=10/k" to the "ground-truth heatmap"?
To better understand the proposed methods, I have a small suggestion:
------ Try betas in a broader range including very small betas, e.g. [0.0001/k, 0.001/k,....,1/k,10/k], for both Table one and visualization. 
Fix a few images and visualize the heatmap given different betas.
We might better see how the growth of beta changes the heatmap.

2. About zero-valued attributions.
I agree with you that equation (5) is an upper bound of MI (eq (4)).
However, I am not sure if I totally agree with the claim "If L_1 is zero for an area, we can guarantee that no information from this area is used for prediction."
----- Given L_1=0 really implies that no information of the corresponding region is used for the certain beta, but is this true for the original model (beta=0)? Table one shows that different beta would lead to very different downstream task accuracy.

3. Specific to the two approaches you proposed, can you explain/motivate in what situations per-sample bottle would be better and in what cases we should prefer ReadOut bottleneck?


review_title: Official Blind Review #3

final_score: rating: 8: Accept
review_content: Summary
The paper proposes a novel perturbation-based method for computing attribution/saliency maps for deep neural network based image classifiers. In contrast to most previous work on perturbation-based attribution, the paper proposes to inject carefully crafted noise into an early layer of the network. Importantly, the noise is chosen such that it optimizes an information-theoretically motivated objective (rate-distortion/info bottleneck) that ensures that decision-relevant signal is flowing while constraining the overall channel-capacity, such that decision-irrelevant signal is blocked from flowing. The flow of signal is controlled by the amount of noise injected, which translates into a certain amount of mutual information between input image regions and noisy activations/features. This mutual information can be visualized in the input image, but it also has a clear, quantitative meaning that is readily interpretable. The paper introduces two ways to construct the injected noise, based on the information bottleneck. Resulting attribution maps are computed and evaluated on VGG-16 and ResNet-50 (on ImageNet), and are compared against an impressive number of previously proposed attribution methods. Importantly, the paper uses three different quantitative measures to compare the quality of attribution maps. The proposed method performs well on all three measures.

Contributions
i) Derivation of a novel method for constructing attribution maps. Importantly, the method is grounded on solid theoretical footing for extracting minimal relevant information (rate-distortion theory / information bottleneck method).

ii) Proposal of a novel quantitative measure to compare quality of pixel-level attribution maps in image classification, and extension of a previously reported method.

iii) Evaluation and comparison against a large body of state-of-the-art attribution methods.

Quality, Clarity, Novelty, Impact
The paper is clear and well written, with a nice introduction to the information bottleneck method. Experiments are well described and hyper-parameter settings are given in the appendix. To the best of my knowledge, the proposed method is sufficiently novel and the application of the information bottleneck framework to pixel-level attribution has not been reported before. Some of the design- and implementation-choices needed to render the intractable info bottleneck objective tractable could perhaps be discussed and potentially even improved in light of recent results in other fields (Bayesian DL, deep latent-variable generative models, and variational methods for deep neural network compression), but I currently don’t consider this a major issue. To me personally the work in convincing and mature enough to vote for acceptance - perhaps most importantly it lays important groundwork for important connections to the theory of relevant information and puts a lot of much needed emphasis on objective evaluation of attribution methods (i.e. without subjective visual judgement of saliency maps). My suggestions below are aimed at helping improve the paper even further.


Improvements
I) A short section of current shortcomings/limitations could be added to the discussion.

II) Perturbation-based approaches that inject noise (into the input image directly) have been proposed previously. Most notably: Visualizing and Understanding Atari Agents, Greydanus et al. 2018 and potentially follow-up citations. It would be interesting to compare both works empirically, but perhaps also theoretically/conceptually. Could the Greydanus work be related to applying the noise directly to the input image along with some additional constraints?


Minor Comments
a) Is there a particular reason for this choice of colormap? While it seems to be roughly perceptually uniform (which is of course good), why not choose a simple sequential colormap (instead of a rainbow-like one)? At least the use of red and green at the same time should rather be avoided to maximize colormap readability under the most common forms of color vision deficiencies.

b) Just a pointer - no need to act on this for the current paper. Large parts of the field of neural network compression are concerned with a similar kind of attribution - the question is which weights/neurons/filters are relevant and which ones are not and can thus be removed from the network without loss in accuracy. Information-bottleneck style objectives (or the closely related ELBO / variational free energy) in conjunction with sparsity inducing priors have been proven to be quite fruitful. See e.g. Variational Dropout Sparsifies Deep Neural Networks, Molchanov et al. 2017 for interesting work, that aims at learning the variance of Gaussian noise that is injected into neural network weights using a similar construction and variational objective as shown in this paper. Perhaps some ideas can be borrowed/translated for future, improved versions of the method from that body of literature (Molchanov 2017, but also more sophisticated follow-up work).
review_title: Official Blind Review #2

Meta-review: All three reviewers strongly recommend accepting this paper. It is clear, novel, and a significant contribution to the field. Please take their suggestions into account in a camera ready version. Thanks!
Decision: Accept (Talk)