Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: CONTRIBUTIONS:
Topic: Analysis of self-attention in the standard Transformer, applied to models of language. 
Notation for review (see also definitions in C1-C4): cell(t,L) is the cell in layer L at input-string-position t (“corresponding to input token t”); Z(t,L) is the output vector for cell(t,L). Also, in the review, points are sometimes labeled by “(label)”, which is intended to label the point that immediately *follows* it in the text.
C1. “Attention identifiability.” Uniquely identifying the attention-weight vector from the output of attention is not possible when D < T: For an individual head, the vector R returned by attention uniquely determines the attention-weight vector A only if the dimension D of the value vector equals or exceeds the length T of the input. 
C2. “Effective attention vector” A1: When D < T, there is a method for uniquely decomposing A as A = A1 + A2 where A2 has no effect on R. 
C3. “Token identifiability” method M. Identifying t from Z(t,L): A method M is proposed for this, and experimental results presented showing that, roughly, M succeeds at lower levels L but can fail at higher levels.
C4. “Hidden token attribution” method: A gradient-attribution-based method is proposed to quantify C(t,t’), the proportional contribution of input token t’ to Z(t,L). C(t,t) is the “[same-]token contribution”; the “contextual contributions” are the C(t,t’) for t’ =! t. Experimental results are presented showing that after L = 1, 6, 12 the median token values are about 30%, 15%, 10%.
MAIN ARGUMENT: Interpretation of the weight of cell(t,L)’s attention to cell(t’,L) as a measure of the influence of token t’ on the level-L representation of token t is (P0) frequent, but problematic, because (P1) the attention weight itself includes an irrelevant quantity (the A2 contribution, which should be omitted to get the effective attention [C2]) and because (P2) Z(t,L) is not properly viewed as representing token t, since (P2) it can be insufficient to even determine which token t it corresponds to [C3] as (P3) it can contain strong contributions from other tokens [C4]. 
RATING: Weak accept
REASONS FOR RATING (SUMMARY). The paper raises important challenges to a standard practice for interpreting the knowledge in Transformer models (especially BERT) by examining attention weights. It proposes methods for reaching more theoretically justified interpretations. It introduces important new general interpretive concepts in the process. The main argument is sufficiently well supported by theoretical and empirical results to justify making the ICLR community aware of it, although there is a missing step in the argument: justification that the proposed means of resolving the unidentifiability of attention yields an optimal result.
REVIEW
Weaknesses.
Minor complaint concerning exposition: The central result, C1 above, is actually rather obvious, unless I’m missing a subtlety (in which case it would be useful for that subtlety to be pointed out explicitly in the paper). (*) The dimension-D vector R is a linear combination of T vectors; if T > D, these T vectors cannot be linearly independent, so the weighting coefficients that yield R cannot be unique. The argument presented is rather overblown and may hide the obvious correctness of the result. However, I think over-deriving is clearly preferable to under-deriving, so this complaint truly is extremely minor. It might be worthwhile to add a sentence like (*) however to make sure that simple observation doesn’t get lost in the formalism. A further advantage of the more detailed derivation is that it sets up nicely the unique decomposition of A into the component A2 in the null space and the component A1 orthogonal to the null space (of the matrix containing the T vectors being linearly combined). 
A more substantial question here: What argument or evidence is there that this particular decomposition A = A1 + A2  is the one that yields a definition of “effective attention” that is best for interpretative purposes? Is there a metric of success that would validate this? Given that A is underdetermined, there is a whole subspace of attention-weight vectors that yield the same result R of attention. The proposed choice of A1 within that space is a natural one, but what makes it the *right* one? The incontrovertible point is that the A vector a model happens to use has a degree of arbitrariness in it that makes using it for interpretation problematic. But what is the right criterion for picking an alternative “effective” version of A, A1? Being orthogonal to the null space is one well-defined choice, but is it clearly the right choice? (I believe this choice is also the one with minimal L2 norm. Is that important?) For example, Sec. 3.3 shows that raw attention, but not effective attention, gives evidence that attention shifts from [CLS] to [SEP] to periods and commas as L increases. This shows they are different, but what is the argument that the raw-attention conclusion is less ‘correct’ than the effective-attention conclusion here? 
More generally, it would strengthen the paper even further to document more concrete cases where effective attention yields different interpretive conclusions than raw attention, especially to the extent that it can be argued that the former interpretations are somehow better than that latter.
Strengths.
The paper gives convincing evidence of the MAIN ARGUMENT’s propositions (P0) – (P3).
(P0) [Interpretation of the attention weight as a measure of the influence of one token on another] is frequent] Sec 4, 2nd paragraph, cites 16 papers to illustrate the point. While I have not examined all of them, even allowing for some possible disagreements of interpretation, I am confident there is ample relevant evidence among these papers and probably others as well.
(P1) [the attention weight itself includes an irrelevant quantity [C2]] The argument provided for this in Sec 3 seems correct to me, but as stated above as a ‘minor complaint concerning exposition’, it may not make as clear as possible the obviousness of the conclusion. 
(P2) [Z(t,L) can be insufficient to determine which token t it corresponds to [C3]] This point is operationalized in Sec 4.2 by picking a function-approximation architecture G, and attempting to use it to perform the map Z(t,L) -> t, for a given L and for all input strings and tokens t. A linear and a non-linear MLP G are used. An experiment applying the pre-trained BERT-base Transformer to the Microsoft Research Paraphrase Corpus gives success rates that fall to 73% and 82% by final layer 12. That 18% of tokens cannot be recovered from their corresponding layer-12 encoding is the basis of the claim that equating attention from cell(t,L) to cell(t’,L) with attention from token t to token t’ is not justified in general. 
That the non-linear approximator does 9% better than the linear one is the basis of the claim that ‘the non-linearities in BERT play an important role in maintaining token identifiability’ (Sec. 4.2). This claim should be spelled out explicitly. It reads to me to say that if the non-linearities in BERT were removed, token identifiability would be reduced, but the experiment does not show that. Rather, it shows that, in the presence of the non-linearities in BERT, a linear approximation to the token-identifying map G is significantly less accurate than the non-linear approximation: hardly a surprising conclusion. This bit of the argument needs to be clarified.
(P3) [Z(t,L) can contain strong contributions from tokens t’ =! t [C4]] As defined in Sec. 5, the contribution of token t’ to Z(t,L), for a given L, is proportional to the norm of the gradient of Z(t,L) with respect to the input from token t’; these are normalized to sum to 1. This is a transparent definition that resembles the previous gradient attribution proposal, but applied to internal rather than output representations. Using the same experimental set-up, it is shown that the average contribution of token t to Z(t,L) declines from 30% to 10% as L goes from 1 to 12. The proportion of values t such that the largest contribution to Z(t,L) is from a token t’ =! T rises from 0 for L = 1 through 3 to over 30% for L = 10 through 12. This is further evidence for the claim that equating attention between Z(t,L) and Z(t’,L) with attention between tokens t and t’ is problematic.
review_title: Official Blind Review #3

final_score: rating: 8: Accept
review_content: This work studies the identifiability of attention, i.e., to what extent does the attention weight uniquely determine the output. The paper (1) formally establishes, using a rank argument, that at least for the cases where the attention head dimensions are smaller than the sources, infinite different attention weights can yield the same attention output. Based on such an observation, (2) a principled tool to extract the `effective` attention is introduced, with which several previous observations are challenged. This tool can certainly inspire future research. The paper further (3) explores how much token and context information is mixed at different layers of a transformer model, using both a carefully designed probing task and a gradient-based method.

Overall I like the paper a lot. Its hypothesis and arguments are very clearly presented, which are then fleshed out by carefully designed experiments. I vote for an acceptance.

I don't have any major complaint. Below are some questions and comments.

- Since the paper only experimented with BERT, I think the authors mean "BERT" instead of the transformer models in general, whenever an empirical argument is drawn.

- Section 4.1, can the authors justify the use of MRPC for this analysis? Adding onto this, do the authors think the conclusion here holds for other input text domain? What about other transformer models, e.g., GPT, or a transformer trained using MT objective?

- Section 4.2, 2nd paragraph. I'm not sure why that g^{MLP} is better than g^{lin} can be interpreted as "non-linearities play an important role in maintaining token identifiability." Could it be the case that the non-linearities make the token information more opaque, such that it can only be extracted with a more expressive model (MLP)? A better way to support this claim, in my opinion, is to show a better token identifiability in a BERT model trained without non-linearities.

- In section 4.2, many conclusions are drawn that the last layer behaves differently than the rest. To eliminate the potential effect of depth, have the authors considered conducting a controlling experiment, where the probing task is to recover e_{i}^{l-1} (i.e., the representation from layer l-1, instead of the input token embedding) from e_{i}^{l}?

review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This very interesting paper proposes new perspectives for investigating the self-attention distribution and contextual embedding and challenges many existing findings in the literature. The conclusion presented in this paper partially aligned with some concurrent works (Danish Pruthi et al. 2019, Serrano et al. 2019 and etc.)
Many concrete findings in the paper make sense very much (not surprising), like 1) Input tokens retain their identity in the first a few hidden layers and then progressively become less identifiable, 2) Non-linear activations are crucial in preserving token identity and 3) Strong mixing of input information in the generation of contextual embedding.

The neat proof in the paper, though is based on some assumptions, formally shows that the multi-head self-attention is not always identifiable. The conclusion is that when the sequence length is larger than the size of attention head, there is redundant parameterization space for the attention.
The authors propose “effective attention” which is orthogonal to the null space as a diagnostic tool. In Figure one, the fact that the Pearson correlation between raw and effective attention weight decreases as the sequence length increases experimentally support the theorem. 

However, I am not sure what benefit identifiable attention would bring us. It seems that "redundancy" sometimes have some "benefits". As shown in the original Transformer paper, very large d_v or very small d_v both lead to worse performance. I am not sure how “effective attention” would ease the training, though I agree it is a good "diagnostic tool".

Another concern is about how the “non-identifiability” really hurts the model? A pre-trained transformer-based model can achieve very good performance for discriminative tasks after fine-tuning. This shows that a deep model, though there are many uncertain choices in the earlier and middle layers, they finally give us correct discriminative labels.
As the authors claim that contextual embeddings are strong mixing of input (which I agree), my question is "will the non-identifiable attention weights, dramatically affect the mixing component of the contextual embeddings"? Or, will the uncertain choices in earlier layers finally come to close contextual representations in layers closer to output?

Towards the claim that “non-linear activations in preserving token identity”, the authors provide evidence in Figure 2.
Can we also do experiments that using the embedding e^{l}_j (Layer l and position j), through non-linear activations to recover neighboring tokens of position j? I suspect that neighboring tokens can probably also being covered very well by MLP g^{MLP}_l.


review_title: Official Blind Review #2

Meta-review: This paper investigates the identifiability of attention distributions in the context of Transformer architectures. The main result is that, if the sentence length is long enough, difference choices of attention weights may result in the same contextual embeddings (i.e. the attention weights are not identifiable). A notion of "effective attention" is proposed that projects out the null space from attention weights. 

In the discussion period, there were some doubts about the technical correctness of the identifiability result that were clarified by the authors. The attention matrix A results from a softmax transformation, therefore each of its rows is constrained to be in the probability simplex -- i.e. we have A >= 0 (elementwise) and A1 = 1. In the present version of the paper, when analyzing the null space of T (Eqs. 4 and 5) this constraint on A is not taken into account. In particular, in Eq. 5 the existence of a \tilde{A} in the null space of T is not clear at all, since for (A + \tilde{A})T = AT to hold we would need to require, besides A >= 0 and A1 = 1, that A + \tilde{A} >= 0 and A1 + \tilde{A}1 = 1, i.e.

\tilde{A} <= A (elementwise)
\tilde{A}1 = 0

The present version of the paper does not make it clear that the intersection of the null space of T with these two constraints is non-empty in general -- which would be necessary for attention not to be identifiable, one of the main points of the paper. 

The authors acknowledged this concern and provided a proof. I suggest the following simplified version of their proof:

We're looking for a vector \tilde{A} satisfying

(1) \tilde{A}’*T = 0 (to be in the null space of T)

and 

(2) \tilde{A}’*1 = 0 
(3) \tilde{A} >= -A

(to make sure A + \tilde{A} are in the probability simplex).

Conditions (1) and (2) are equivalent to require \tilde{A} to be in the null space of [T; 1]. It is fine to assume this null space exists for a general T (it will be a linear subspace of dimension ds - dv - 1). 

To take into account condition (3) here’s a simpler proof: since A is a probability vector coming from a softmax transformation (hence it is strictly > 0 elementwise), there is some epsilon > 0 such that any point in the ball centered on 0 with radius epsilon is >= -A.
 
Since the null space of [T; 1] contains 0, any point \tilde{A} in the intersection of this null space with the epsilon-ball above satisfies (1), (2), and (3). This should work for any ds - dv > 1  and as long as A is not a one-hot distribution (otherwise it collapses to a single point \tilde{A} = 0).

I am less convinced about the justification to use an “effective attention” which is not in the probability simplex, though (not even in the null space of [T; 1] but only null(T)). That part deserves more clarification in the paper. 

I recommend acceptance of this paper provided these clarifications are provided and the proof is included in the final version. 


Decision: Accept (Poster)