Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper proposes a hierarchical decomposition method to encode the natural language as mathematical formulation such that the properties of the words and phrases can encoded properly and their importance be preserved independent of the context. This formulation is intuitive and more efficient compared to blindly learning contextual information in the model. The proposed method is a modification of contextual decomposition algorithm by adding a sampling step. They also adapt the proposed sampling method into input occlusion algorithm as another variant of their method. The proposed method is tested on LSTM and BERT models over sentiment datasets of Stanford Sentiment Treebank-2 and Yelp Sentiment Polarity and TACRED relation extraction dataset and showed more interpretable generated hierarchical explanations compared to baselines.

review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: Summary:
The authors proposed a method for generating hierarchical importance attribution for any neural sequence models (LSTM, BERT, etc.) Towards this goal, the authors propose two desired properties: 1) non-additivity, which means the importance of a phrase should be a non-linear function over the importance of its component words; 2) context independence, which means that the attribution of any given phrase should be independent of its context. For example, in the sentence "the film is not interesting", the attribution of "interesting" should be positive while the attribution of "not interesting" should be negative.

Following these two properties, the authors designed three algorithms to post-hoc analysis the importance of a given phrase p.
1. [Sec 3.2] eq 4. expected differences in model predictions between the a sentence that contains p and the same sentence with p removed. The expectation is computed over the conditional probability Prob(sentence | p in sentence). In practice, the authors use eq 3 as a proxy to eq 4.
2. [Sec 3.3] eq 5.  expected differences in the activation values of each layer. The expectation is computed over the conditional probability Prob(context-dependent representations | phrase-dependent representations).
3. [Sec 3.4] eq 8. similar to 1 but we replace the phrase p with padded tokens.

The authors conducted experiments on SST and Yelp. Results show that their proposed context-independent attribution correlates better with a trained linear model's coefficient, achieves higher human trust.

Decision: reject.

While I found the idea of marginalizing out the local context interesting, I think the paper still needs more work on its formulation, experiments and writing.

Formulation:
1. In eq 3, the expectation is taken over the difference between the prediction on the sampled sentence and the one with the phrase removed. This may be problematic for longer inputs (a pargraph), where the overall prediction may not change a lot when you remove a single phrase (since the evidence is everywhere). For example, consider the input: "The movie is the best that I have ever seen. It is remarkable!". Removing the word "best" alone doesn't alter the prediction much. 
2. In eq 5, the expectation is computed over P(h | beta). It is NOT THE SAME as sampling words p(x_{\delta} | x_{-\delta}) and then consider their hidden states.
3. In Sec 3.1, you mentioned that CD is limited since the decomposition of activation sigma evolves context information gamma, and you resolved this by marginalization. But it seems to me that the computation of element wise multiplication also evolves context information. How do you deal with these?
3. What's the difference between eq3 and eq8? Are you just changing from remove the phrase completely to replace it by mask?

Experiments:
1. The performance of CD in Table 1 seems very different to the original CD paper (which is 0.758 for SST and 0.520 for Yelp). I am not sure what contributes to this big difference. Is it the trained model or data splits?
2. Table 1 shows that your methods achieves higher correlation to linear model's coefficients. But why shall we consider linear model's coefficients as the ground truth for the learned neural model? For example, the fine-tuned BERT achieves lower correlation corresponding to the LSTM. Does that mean the BERT model performs worse than LSTM?

Missing related references:
1. Explaining Image Classifiers by Counterfactual Generation
2. Rationalizing Neural Predictions
3. Learning to Explain: An Information-Theoretic Perspective on Model Interpretation
4. L-Shapley and C-Shapley: Efficient Model Interpretation for Structured Data
review_title: Official Blind Review #4

final_score: rating: 8: Accept
review_content: Summary - The paper addresses the problem of hierarchical explanations in deep models that handle compositional semantics of words and phrases. The paper first highlights desirable properties for importance attribution scores in hierarchical explanations, specifically, non-additivity and context independence, and shows how prior work on additive feature attribution and context decomposition doesn’t accurately capture these notions. After highlighting the said properties in context of related work, the authors propose an approach to calculate the context-independent importance of a phrase by computing the difference in scores with and without masking out the phrase marginalized over all possible surrounding word contexts (approximated by sampling surrounding context for a fixed radius under a language model). Furthermore, based on the above, the authors propose two more score attribution approaches -- based on integrating the above sampling step with (1) the contextual decomposition pipeline and (2) the input occlusion pipeline. Experimentally, the authors find that the attribution scores assigned by the proposed approach are more correlated with human annotations compared to prior approaches and additionally, the generated explanations turn out to be more trustworthy when humans evaluate their quality.

Strengths

- The paper is well-written and generally easy to follow. The authors do a good job of motivating and highlighting the desired properties of importance attribution scores and developing the proposed scoring mechanism. The proposed scoring mechanism ties in seamlessly with the existing contextual decomposition and occlusion pipelines and leads to improved performance when the generated explanations are evaluated.

- The proposed approach involving masking out the phrase and marginalizing over possible surrounding word-concepts is novel and offers an interesting perspective on how to approach context independent scoring of phrases -- (1) phrases don’t exist independent of the surrounding context and therefore marginalizing over all possible surrounding concepts makes sense and (2) replacing the intractable enumeration over all possible surrounding concepts with samples from a language model makes the score attribution process faster and more scalable modulo the learnt language model.

- Sec. 4.4 offers interesting insights. I like that the authors performed this ablation given that the expectation over surrounding contexts is computed approximately via samples under a language model. There’s a clear increase in terms of the attribution scores as the number of samples increases and the neighborhood size is increased. It is interesting to note that there is an approaching plateau region where increasing the neighborhood size won’t affect the assigned scores. This experiment provides a holistic picture of the behavior of the interpretability toolkit (manifesting in terms of attribution scores) given the approximations involved. I would encourage the authors to flesh this out even more. 

Weaknesses

Having said that, there are some minor comments that I’d like to point out / get the authors’ opinion on. Highlighting these below:

- While SOC and SCD don’t always end up outperforming other approaches (specifically Statistic) on the SST-2, Yelp and TACRED datasets (Table. 1), for the human evaluation experiments, the authors only compare with CD, Direct Feed, ACD and GradSHAP. Do the authors have any insights on how well does Statistic perform on the human-evaluation set of experiments?

- While inspiring trust in users is one aspect of evaluating explanations via humans, it’s slightly unclear what ‘trust’ in this sense inherently identifies. Although, it might implicitly capture some notion of reliability (and predictability of the explanations by humans), asking users to rank explanations across a spectrum of ‘best’ to ‘worst’ doesn’t explicitly capture that. Another possible aspect to look into could be -- ‘’Do the generated explanations help humans predict the output of the model?’’ This captures reliability in a very explicit sense. Do the authors have any thoughts on this and potential experiments that might address this? I don’t think not addressing this is necessarily detrimental to the paper but I’m curious to hear the thoughts of the authors on the same. 

Reasons for rating

Beyond the above points of discussion, I don’t have major weaknesses to point 	out. I generally like the paper. The authors do a good job of identifying the sliver in which they make their contribution and motivate the same appropriately. The proposed phrase attribution scoring mechanism is motivated from a novel perspective and has a reasonable approximation characterized appropriately by the ablations performed. The strengths and weaknesses highlighted above form the basis of my rating.
review_title: Official Blind Review #1

Meta-review: The authors present a hierarchical explanation model for understanding the underlying representations produced by LSTMs and Transformers.  Using human evaluation, they find that their explanations are better, which could lead to better trust of these opaque models.

The reviewers raised some issues with the derivations, but the author response addressed most of these.  
Decision: Accept (Spotlight)