Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: CONTRIBUTIONS:
I use (unqualified) “self-attention” to refer to attention of tokens in a sequence to other tokens in the same sequence, as described by [some corrected version of] Eq (1) and the paragraph following it (citing Bahdanau et al. 2015). This contrasts with “Transformer self-attention” and “cross-sequence attention”.
C1. Self-attention is gating. (Prop. 4.1, Sec. 4)
C2. Gating cannot provide explanations in the way that attention is alleged to do.
C3. Single input sequence models deploy only self-attention, so by C1-C2, attention cannot be used for explanation of these models; attention in two input sequence models is not limited to self-attention and is not equivalent to gating, so attention can provide explanations in these models.
C4. Outputs are sensitive to alteration of attention weights for two-sequence but not one-sequence models. (Tables 2-3 vs. 1; Figs. 2-3) 
C5. Human judgments (“manual evaluation”) of the intuitive importance, for the output, of items with highest attention weights show that such intuitive importance is found for both one- and two-sequence models. (Fig. 5)
RATING: Reject
REASONS FOR RATING (SUMMARY). Because the modeling experiments closely follow previous work, the primary contribution rests on the account provided of why explanation-by-attention works only sometimes --- on the basis of the Proposition identifying attention with gating. But the reasoning and the math as presented are problematic. There is a worthwhile contribution from the human judgment experiment, but this is not sufficient to overcome the weakness with the main argument of the paper. (I also have reasons to question whether one- vs two-sequence inputs is the right distinction that needs to be accounted for.) 
REVIEW
C1. If this point were made in plain English, ‘attention weights and gating units both multiply activation values’, I would say, “yes, obviously”. But the point is stated as a Proposition, with a Proof, so the math given should be correct. I don’t see a way to make it correct, and that shouldn’t be the job of the reader anyway. There are several errors in the dimensions of the matrices which make the equations incoherent. Eq. (1) contains W*h, a matrix product, where the dimensions stated are W in R^{d x d’} and h in R{T x m}; these cannot be multiplied. This unclarity about matrix dimensions propagates into Prop. 4.1. In the definition of g(X), we have WX + b, where b is presumably a vector. Addition then requires that WX also be a vector, but X is stated to be in R^{n x d}, so WX cannot be a vector. Whether WX + b is actually a vector or a matrix, g(X) = c^T tanh(WX + b) is not a matrix: it is either a scalar or a vector. But this can’t be. The definition of h uses the elementwise product, which requires that both arguments have the same dimensions, so g(X) must have the same dimensions as f(X). We’re told f is the identity function, so f(X), like X, must be a matrix. Furthermore, the statement of Prop. 4.1 says that self-attention can be interpreted as *a* gating unit. By the standard definition of ‘unit’, this should mean that self-attention is a scalar.
Throughout the paper, we are never told what kind of “RNN” is being assumed. If the RNN unit contains gates, as in an LSTM or a GRU, I can imagine that the intention is for Prop. 4.1 to say that (*) “the effect of self-attention can be reproduced without attention by adjusting the weights in the gating mechanism already present in the RNN”, so that attention doesn’t increase the capacity of the model. But what I see in the paper does not convince me that (*) is true. (Because of the kind of global normalization required by the softmax, I actually suspect it is not.)
C2. I don’t see why (formally or intuitively) gating is not a legitimate candidate for explaining the behavior of networks containing gates; I would assume just the opposite, actually. How can it *not* be part of a satisfactory explanation? And why should changing a name from “attention” to “gating” have any bearing on whether it (whatever it is called) can potentially serve for explanation of network behavior?
C3. Leaving the formalism aside, I don’t see intuitively why, whatever an analysis of self-attention might entail about explanation, the same implication shouldn’t apply to straightforward (not Rocktaeschel) attention when two sequences are present. Why can’t we just treat the concatenation of the input sequences as a single input sequence, as standardly done for example for the Transformer? If the formal content of Prop. 4.1 were clear, perhaps this could be justified, but it is simply asserted without justification in the proof that “the same reduction does not hold in the case of pair sequence”.
C4. Claims C1-C3 attempt to give an account for why various tests of the explanatory role of attention turn out positive for two-sequence but not one-sequence tasks, a pattern previously reported and verified with new results in the present paper. I fear however that one- vs two- is not the correct empirical generalization about attention that one should try to account for. Messing about with attention weights would not be expected to alter outputs if the output is determined by the input considered as a bag of words. And there is a troubling confound in the tasks compared: the one-sequence tasks are sentiment and topic classification, where a BOW model should do very well – and I suspect that is the real reason why these tasks don’t show strong sensitivity to attention weight distribution. But the two-sequence tasks are NLI and QA, where (ideally) BOW models should not do nearly so well: paying attention to the right tokens should be important. The same is true of translation. So the confound in the tasks examined leaves it undetermined whether the crucial factor to account for is one- vs two-sequence or BOW-friendly vs BOW-unfriendly tasks.
C5. Put together, the human judgments and the sensitivity-to-altering-attention-weights seem to indicate that attention tends always to be allocated to intuitively important tokens, and that matters for the output of the two-sequence models but not for the one-sequence models. This is what we’d expect if attention is always being allocated appropriately, but for BOW-friendly tasks that doesn’t make much difference.

review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: This paper investigates the degree to which we might view attention weights as explanatory across NLP tasks and architectures. Notably, the authors distinguish between single and "pair" sequence tasks, the latter including NLI, and generation tasks (e.g., translation). The argument here is that attention weights do not provide explanatory power for single sequence tasks like classification, but do for NLI and generation. Another notable distinction from most (although not all; see the references below) prior work on the explainability of attention mechanisms in NLP is the inclusion of transformer/self-attentive architectures. 

Overall, this is a nice contribution that unifies some parallel lines of inquiry concerning explainability, and attention mechanism for different NLP tasks. The contrast in findings between single sequence (classification) and other NLP tasks and the implications for explainability is interesting, and provides a piece missing from the analyses conducted so far. 

The main issue I have with the paper is that I'm not sure I follow the rationale in Section 4 arguing that because attention "works as a gating unit", it follows that we cannot view single sequence tasks "as attention". Can the authors elaborate on why the conclusion follows from the premise? In other words, why is attention inherently *not gating*? This seems like an interesting connection, but again I do not see why attention acting as a gate implies that we should not view it as attention at all. Perhaps the authors could elaborate on their reasoning. 

Some additional references the authors might consider. Note that the latter two papers, I think, would seem to broadly disagree regarding self-attention and tasks aside from single sequence (classification): 
- "Fine-grained Sentiment Analysis with Faithful Attention": https://arxiv.org/abs/1908.06870
- "Interrogating the Explanatory Power of Attention in Neural Machine Translation": https://arxiv.org/abs/1910.00139
- "On Identifiability in Transformers": https://arxiv.org/abs/1908.04211
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: Motivated by an existing paper, the paper analyzes the interpretability of attention mechanism over three NLP tasks. The previous paper claims that attention mechanism is not interpratable. The paper makes incremental contribution by showing that attentions are not interpratable when they mimic gating units (single sequence task) but are interpretable for two sequence and generation tasks. Experimental results are given to support the claims, which can also help readers to gain insights into the attention mechanism. The paper is well written and all claims are supported. I also have some questions below for clarification. If I get reasonable answers to these questions, I tend to accept the paper. 

1. Jain & Wallace (2019) show that for the SNLI problem, attentions weakly correlate with the output based on Kendall's correlation and JSD which contradicts to your observation. Could you explain how this happens? are there any model settings different?
2. in figure 5, for the single sequence (original), most of attentions leading to correct predictions are labeled meaningful. Does this mean that even the attention doesn't necessarily contribute too much to the prediction correctness but they are also interpretable if they are allowed to be trained. Also, in Table 1, with modified attentions, all scores go down a little. This means that attention still can contribute to the final prediction but not significant enough (some like yelp are significant). I am gussing that the sequence encoding already present useful features for the final prediction. Did you check the distance of different word encodings? Are these encodings all very similar?
review_title: Official Blind Review #1

Meta-review: This paper investigates the degree to which we might view attention weights as explanatory across NLP tasks and architectures. Notably, the authors distinguish between single and "pair" sequence tasks, the latter including NLI, and generation tasks (e.g., translation). The argument here is that attention weights do not provide explanatory power for single sequence tasks like classification, but do for NLI and generation. Another notable distinction from most (although not all; see the references below) prior work on the explainability of attention mechanisms in NLP is the inclusion of transformer/self-attentive architectures. 

Unfortunately, the paper needs work in presentation (in particular, in Section 3) before it is ready to be published.
Decision: Reject