Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: "an information theoretic principle, information bottleneck principle" in the abstract is quite redundant with the use of 'principle' twice

'"great, great" and "great, thought provoking". They have the same level of sparsity.' What kind of sparsity are you referring to with this example? Why can't sparsity reduce semantic redundancy? Please explain further.

"However, the first explanation has a large MI with the input document where "great" occurs a lot." What example input document are you referring to?

You should save the explanation of how your method in Equation 2 differs from the original information bottleneck of Equation 1 until after you have actually written out Equation 2. As it is now, you are referencing Equation 2 before it has been seen. 

I find Equation 2 confusing. Is it possible to make the dependence of the expression on z more explicit. It isn't clear from the equation itself how p(z|x) influences either quantity in Equation 2. Perhaps you should wait to introduce this equation until you have first explained how z relates to x and t. As it is, z is not clearly defined. I can gather information about it from the figure, from how you describe the difference in your method from the original information bottleneck, but the relationship is not clear enough by reading only the text of the paper before Equation 2 is presented.

Can you explain briefly how your "hierarchical LSTM" works in the main text of the paper? Its an unusual enough term that I would want to see a citation or brief explanation right away rather than having it deferred to the Appendix. Why not use a state-of-the-art model for IMDb? Are you not using the standard splits for IMDb? the

In Appendix B.1 "output vector is averaged and followed by log-softmax calculation. The final layer is formed to return a log-probability indicating which cognitive chunks should be taken as an input to the approximator" The single output vector of the biLSTM is averaged and followed by log-softmax? the final layer is formed? What does this mean?

I find the phrasing of "Negative Sentiment if any negative words" and the corresponding title for positive in Fig 2 confusing. What do you mean by "if any"? The phrasing makes it sound like the prediction of the model somehow depends on a logical step based on whether there are any negative/positive words found.

I find the lack of a comparison to some kind of attentional method somewhat glaring in the IMDb example, since I would expect that many classifiers with attention would simply attend to the same words. What does your method give us that attention would not?

The same can be said for the MNIST example regarding an attention map.

"by the human intelligences" sounds quite robotic

Can you provide some sense of inter annotator agreement for labeling the images and sentences?

It does seem that there is key information like the definition of approximator fidelity in the appendices which is crucial to actually understanding the paper.
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: The paper proposes a method to learn an explanation of black-box systems from its outputs. The method is based on the information bottleneck as the objective function is designed to measure mutual information between input x, system output y, and narrowed information of input t. t is constructed by filtering x with maintaining interpretability of y, so that it is finally assumed as the explanation of the system extracted by the proposed method.

The paper is well motivated and well written. Enough experiments were conducted to assess the advantage of the proposed method in the classification tasks. It looks a good paper.

Maybe the paper is focused on only tasks that the predictor does not generate much information, such as classification. It is still unclear how the proposed method work when it is applied to the output-rich models, i.e., the model should keep as much information as inputs.

The proposed method automatically selects some important chunks from inputs, but the chunks still rely on some task-specific hand-crafted chunking strategies. The paper also conducted some experiments by changing the strategy, but it is still unclear what is the important criteria.

It is also good to show how actually the thickness of the bottleneck (controlled by k) works in actual cases, e.g., showing results for the same example with moving k.

Trivial comments:
* The example in 3.2 "great, great" and "great, thought provoking" looks still ambiguous to explain what the section want to say.
* "x_i \times z_j" in p.iv looks ambiguous.
* z_j^* in 3rd eqn. of p.v should take l: z_j^{*(l)}
* the max operator over l in 3rd eqn. of p.v looks to hide other values than the highest one (specifically, the L1 norm of z_j^* does not become k by this eqn. as the k-hot vector does). Summation looks intuitively better than max. Could you explain how this eqn. was constructed?
* f(.) in the 4th eqn. of p.v may be undefined in the main text.
* \beta_1 of Adam looks to be set to not a standard value (0.9). Is there any reason?
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: [Due to the rebuttal, my score was raised from a weak reject to a weak accept]

Summary
The paper addresses the problem of interpreting predictions/decisions of a black-box classifier/regressor by masking the parts of the input that were most relevant. The proposed approach consists of, first, manually designing “cognitive chunks” of input data, e.g. individual words for sentiment classification or fixed-size image-patches for image-classification. Then, a variational IB framework is used to infer which of these chunks are relevant for the classifier’s decision. Additionally, there is a (hard) constraint, making sure that only a fixed (small) number of chunks is used. The bottleneck variable, in this case, is a sparse-chunk representation of the data. The latter is obviously a more compressed representation of the data, but importantly it is a more compressed representation that contains the largest possible amount of relevant information about the decision (because of properties of the information-bottleneck objective). Both factors together, according to the paper, constitute a “good” (i.e. brief but comprehensive) explanation which allows for interpretability and attribution of the black-box system’s decision. The method is evaluated on three tasks (sentiment prediction, image classification, TCR to epitope binding prediction) and performance is reported to be on-par or better than state-of-the-art methods.

Contributions
-) Application of the IB-method for generating summaries of decision-relevant input-data, which are good candidates for interpretability. The theoretical properties of the IB objective are appealing for producing interpretable data-summaries.
-) Adaptation of the variational IB framework, using bits and pieces reported in the literature such that the bottleneck variable is a sparse, binary vector over “cognitive chunks”.
-) Experimental evaluation, where human judges rate the “interpretability“ of various state-of-the-art attribution methods.

Quality, Clarity, Novelty, Impact
The paper addresses a timely and important problem, particularly the IB framework could add some solid theoretical footing (the “theory of relevant information”) to the field of interpetability methods. The paper is well written (though it needs another pass for typos, etc.), related methods and literature are discussed and compared against, and the specific variational IB objective is introduced nicely. Large parts of the method (deep variational IB, VI with categorical variables) have been published before, but these parts are combined in a novel and original way. My main issues with the current paper are (I) interpretability and comprehensiveness are not necessarily the same as maximum compression of maximally relevant information, (II) the method (in theory) depends strongly on the quality of the approximator, this is currently not mentioned and not explored, (III) the experimental section is currently not very strong, in particular the MNIST experiment. See more details for the main issues below. Overall, I personally think that the main idea of the paper is interesting, mature and fleshed out enough for a publication, however, the experimental section is somewhat lacking and (II) is missing from the current manuscript. While the method has theoretical advantages, empirically it seems to perform more or less equal to L2X (but the chunks produced seem qualitatively different which is interesting). I am therefore slightly leaning towards suggesting a major revision of the paper, but I am happy to be convinced otherwise by the other reviewers and the authors during discussion/rebuttal. (I would rate the paper as "borderline", but it seems that this year's review system only allows for "weak reject" or "weak accept", so I'll go for "weak reject" for now).

Improvements / major issues

(I) Good compression of highly relevant information is not (always) the same as good interpretability/comprehensiveness. In the limit, the bottleneck variable captures a minimal sufficient statistic, i.e. a maximally compressed version of all relevant information - for finite beta, the bottleneck approximates such a minimal sufficient statistic. From a theoretical point of view this is very appealing, since it is guaranteed to cover a maximal amount of information (given a certain level of compression). But the way this information is represented matters a lot for interpretability - any reversible mapping of the bottleneck variable does not change its information content but can have substantial effects on interpretability, e.g. consider encrypting or randomly perturbing elements of the explanation (i.e. the selected cognitive chunks). This is a major open problem, and some theoretical grounding in the IB framework helps by talking about this problem in very concrete terms. While I would not expect the paper to solve the problem in full generality, some discussion, and perhaps adding a “shortcomings” section would be nice.

(II) The relevance of information is measured via I(t;y), which ultimately boils down to the approximation q(y|t). The quality of this approximation is crucial, which can of course be seen by how it influences the tightness of the bound. While I appreciate that the paper investigates the quality of the approximation to some degree (by inspecting the approximator fidelity), I would highly appreciate a thorough discussion of this issue (because ultimately the method will produce cognitive chunks that are relevant for q(y|t), not p(y|x) - the interpretations can be trusted only if q matches fairly well). It would be very interesting to see how quickly interpretability degrades with lower-quality q(y|t) - the latter would of course require more experiments with human “interpreters” which I would not expect to be easily feasible within the rebuttal period.
Another interesting experiment to test the match between q(y|t) and p(y|x) would be to “minimally intervene” on the input-chunks suggested by the method and see whether that actually affects the predictions of the black-box models. E.g. do small random perturbations to the selected cognitive chunks in the MNIST digits change the prediction of the black-box classifier? Compare this against small random perturbations in arbitrary chunks of the input.

(III) Experimental section: I’m fairly happy with the IMDB experiment, and the TCR to Epitope binding is a nice non-standard application but I find the quality and significance of the results a bit hard to judge. My main concern though is the MNIST experiment: what I would have expected was the following: cognitive chunks are shown to participants and they need to guess the correct number (just like in the IMDB experiment). In the experiment reported in the paper, I’m afraid that there’s a certain bias for judges favoring explanations that lie on the digits rather than off digits. It remains unclear whether they simply prefer the chunks selected by VIBI over other methods, or whether they have actually gained more understanding of how the black-box makes decisions.

(IV) Table 2 can easily be misleading because entries with highest mean-accuracy are marked in bold, regardless of whether confidence intervals overlap with other entries or not. Please fix this by either only marking entries in bold where the error bars don’t overlap with an entry in the same row, or marking all entries in bold that lie within the error bars of the best-performing entry. Particularly for “Approximate Fidelity” VIBI often does not perform significantly better than L2X but performs roughly equally well. Of course it would also help to run more repetitions to potentially shrink confidence intervals.

(V) Please state the (parametric form) of the prior r(z*) used for the experiments. Also state the analytical expression for the KL-term in the final objective that this prior leads to.

Minor comments

a) Please add some discussion on how the method depends on hand-crafting cognitive chunks, and how hard/easy this might be for different domains.

b) Rather than fixing r(z), other papers have proposed to optimize the prior as well (typically in the context of VAEs / VIB) which is well justified from an IB perspective. It might be interesting to explore these possibilities for VIBI as well in the future.

[1] Fixing a broken ELBO. Alemi et al. 2017
[2] The beta-VAE’s Implicit Prior. Hoffman et al. 2017

c) Instead of fixing the number of cognitive chunks in advance, it could also be interesting to infer that number as well (as a future extension of the method). This could either be achieved via a sparsity-inducing prior r(z), or perhaps by borrowing some ideas from the Deterministic IB [3], and its variational version.

[3] The deterministic information bottleneck. Strouse and Schwab. 2016
review_title: Official Blind Review #2

Meta-review: The authors present a system-agnostic interpretable method based on the idea of that provides a brief (=compressed) but comprehensive (=informative) explanation. Their system is build upon the idea of VIB. The authors compare against 3 state-of-the-art interpretable machine learning methods and the evaluation is terms of interpretability (=human understandable) and fidelity (=accuracy of approximating black-box model). Overall, all reviewers agreed that the topic of model interpretability is an important one and the novel connection between IB and interpretable data-summaries is a very natural one.  

This manuscript has generated a lot of discussion among the reviewers during the rebuttal and there are a number of concerns that are currently preventing me from recommending this paper for acceptance. The first concern relates to the lack of comparison against attention methods (I agree with the authors that this is a model-specific solution whereas they propose a model-agnostic one), however attention is currently the elephant in room and the first thing someone thinks of when thinking of interpretability. As such, the authors should have presented such a comparison. The second concern relates to the human evaluation protocol which could be significantly improved  (Why 100 samples from all models but 200 for VIBI? Given the small set of results, are these model differences significant? Similarly, assuming that we have multiple annotations per sample, what is the variance in the annotations?).

This paper is currently borderline and given reviewers' concerns and the limited space in the conference program I cannot recommend acceptance of this paper. 
Decision: Reject