On Identifiability in Transformers
Authors: Gino Brunner
Yang Liu
Damian Pascual
Oliver Richter
Massimiliano Ciaramita
Roger Wattenhofer
Venue:  (2020)
Keywords: Self-attention
interpretability
identifiability
BERT
Transformer
NLP
explanation
gradient attribution

Abstract: In this paper we delve deep in the Transformer architecture by investigating two of its core components: self-attention and contextual embeddings. In particular, we study the identifiability of attention weights and token embeddings, and the aggregation of context into hidden tokens. We show that, for sequences longer than the attention head dimension, attention weights are not identifiable. We propose effective attention as a complementary tool for improving explanatory interpretations based on attention. Furthermore, we show that input tokens retain to a large degree their identity across the model. We also find evidence suggesting that identity information is mainly encoded in the angle of the embeddings and gradually decreases with depth. Finally, we demonstrate strong mixing of input information in the generation of contextual embeddings by means of a novel quantification method based on gradient attribution. Overall, we show that self-attention distributions are not directly interpretable and present tools to better understand and further investigate Transformer models.   

Published as a conference paper at ICLR 2020
ON IDENTIFIABILITY IN TRANSFORMERS
Gino Brunner Yang Liu Damián Pascual Oliver Richter', Massimiliano Ciaramita', Roger Wattenhofer' Departments of Elletrrcca Engineering and Information Technology, *Computer Science ETH Zurich, Switzerland
GGoogle Research, Zurich, Switzerland
1 [[brunnegi, dpascual, richtero, wattenhofeee//eettt....l
2 Fiiu. yang@a ummni. ettz..chc. ch "massi tmassi@google..o com
ABSTRACT
In this paper we delve deep the Transformer architecture by investigating two of its core components: self-attention and contextual embeddings. In particular, we study the identifiability of attention weights and token embeddings, and the ag- gregation of context into hidden tokens. We show that, for sequences longer than the attention head dimension, attention weights are not identifiable. We propose effective attention tions based on attention. Furthermore, we show that input tokens retain to a large degree their identity across the model. We also find evidence suggesting that iden- tity information mainly encoded in the angle the embeddings and gradually decreases with depth. Finally, we demonstrate strong mixing of input information in the generation of contextual embeddings by means of a novel quantification method based gradient attribution. Overall, we show that self-attention distri- butions are not directly interpretable and present tools to better understand and further investigate Transformer models.
complementary tool for improving explanatory interpreta-
INTRODUCTION
In this paper we investigate neural models of language based on self-attention by concentrating on the concept of identifiability Intuitively, identifiability refers to the ability of a model to learn stable representations. This is arguably a desirable property, it affects the replicability and interpretabil- ity of the model's predictions. Concretely, we focus on two aspects of identifiability. The first is related to structural identifiability (Bellman & Aström 1970): the theoretical possibility (a priori) to learn a unique optimal parameterization of a statistical model. From this perspective, we analyze the identifiability of attention weights, what we call attention identifiabiliy, in the self-attention components of transformers (Vaswani et al., (017). one of the most popular neural architectures for language encoding and decoding. We also investigate token identifiability, as the fine-grained word-level mappings between input and output generated by the model. The role of attention as means of recovering input-output mappings, and various types of explanatory insights, is currently the focus of much research and depends to a significant extent on both types of identifiability.
We contribute the following findings to the ongoing work: With respect to attention indentifiability. in Section 3 we show that under mild conditions with respect to input sequence length and atten- tion head dimension the attention weights for given input are not identifiable. This implies that there can be infinitely many different attention weights that yield the same output. This finding chal- lenges the direct interpretability attention distributions. As a supplement, we propose the concept of effective attention, a diagnostic tool that examines attention weights for model explanations by removing the weight components that do not influence the model's predictions.
"Equal contribution with authors in alphabetical order. Yang Liu initiated the transformer models study, per- ceived and performed the study of attention identifiahility and effective attention, i.c., Section and Appendix A, and contributed to the token attribution discussions and calculations.

---

Published as a conference paper at ICLR 2020
With respect to token identifiability, in Section 4 we devise an experimental setting where probe the hypothesis that contextual word embeddings maintain their identity as they pass through succes- sive layers of transformer. This is assumption made in much current research, which has not received a clear validation yet. Our findings give substance to this assumption, although it does not always hold in later layers. Furthermore, we show that the identity information is largely encoded in the angle of the embeddings and that it can be recovered by a nearest neighbour lookup after a learned linear mapping from hidden to input token space.
In Section 5 we further investigate the contribution of all input tokens in the generation of the con- textual embeddings in order to quantify the mixing of token and context information. We introduce Hidden Token Attribution, a quantification method based on gradient attribution. We find that self- attention strongly mixes context and token contributions. Token contribution decreases monoton- ically with depth, but the corresponding token typically remains the largest individual contributor. We also find that, despite visible effects of long term dependencies, the context aggregated into the hidden embeddings is mostly local. We notice how, remarkably, this must be effect of learning.
2 BACKGROUND ON TRANSFORMERS
The Transformer (Vaswani et al., 2017) is currently the neural architecture of choice for natural language processing (NLP). its core if consists of several multi- self-attention layers. In these layers, every token of the input sequence attends to all other tokens by projecting its embedding to a query, key and value vector. Formally, let Q E Rd, xd, be the query matrix, K E Rd.xd, the key matrix and V E Rdd the value matrix, where d, is the sequence length and dy and d, the dimension of query and value vectors, respectively. The output of an attention head is given by:
QQT Vdl
Attention
(aatiion(Q,A QQ = A.V V
with A = softmax
(1)
The attention matrix A E Xd, calculates, for each token in the sequence, how much the compu- tation of the hidden embedding at this sequence position should be influenced by each of the other (hidden) embeddings. Belf-attention is a non-local operator, which means that at any layer token can attend all other tokens regardless of the distance in the input. Self-attention thus produces so- called contextual word embeddings, as successive layers gradually aggregate contextual information into the embedding of the input word.
We focus on Transformer model called BERT (Devlin et al., 2019), although our analysis can be easily extended to other models such as GPT, (Radford et al. [0018] 2019) RoBERTa (Liu et al.] 2019). XLNet (Yang et | (019b), or ALBERT (Lan et al.] 2020). BERT operates on input se- quences of length dg. We denote input tokens the sentence as a,, where i (I....,d, d.]. We use 2, Rd with embedding dimension d to refer to the sum of the token-, segment- and position embeddings corresponding to the input word at position i. We denote the contextual embedding at position i and layer 1 as e/. Lastly, refer to the inputs and embeddings of all sequence positions as matrices X and E, respectively, both in RRd xd For all experiments we use the pre-trained uncased BERT-Base model as provided by Devlin al. 2019
ATTENTION IDENTIFIABILITY
We begin with the identifiability analysis of self-attention weights. Drawing an analogy with struc- tural identifiability (Bellman & Aström 1970), we state that the attention weights of an attention head for a given input are identifiable if they can be uniquely determined from the head's output? We emphasize that attention weights are input dependent and not model parameters. However, their identifiability affects the interpretability of the output, i.e., whether attention weights can provide the basis for explanatory insights on the model's predictions (cf.. Jain & Wallace (2019) and Wiegreffe & Pinter 2019).. If attention is not identifiable, explanations based on attention may be unwarranted.
https:/ //tt:////tttbb/.///// com/ gowg/ e- resea /bert CC. Appendix A.I for more background on indentifiability.
2

---

Published as a conference paper at ICLR 2020
The output of multi-head attention layer is the summation each of the h single head outputs (cf. Eq. D multiplied by the matrix H E RR xd with reduced head dimension d, d/h,
Attention (Q, K, V AEW H AT
where W E RRdxd, projects the embedding E' into the value matrix V = EWV, and we define T EWV H. Here, the layer and head indices are omitted for simplicity, since the proof below is valid for each individual head and layer in Transformer models. Intuitively, the head output is linear combination of the T vectors using the attention as weighting coefficients. If the sequence length, i.e. the number of weighting coefficient, is larger than the rank of T. attention weights are other words,
not uniquely determined from the head output: i.e., they include free variables. some of the T rows are linear combinations of others. We now prove, by analyzing the null space dimension of T, that attention weights are not identifiable using the head or final model output.
3.1 UPPER BOUND FOR RANK(T)
We first derive the upper bound of the rank of matrix T EWV H. Note that rank (ABC) min (rank(A), rank [), rank(C')) therefore,
S
rank (T) I min (rank(E), rarraank (WV), raak(HH)
(3)
I min(ds, d, d, du dus d)
min (da, du).
The second step holds since rank(F) F) S min rank s min(d, d,) and rank s min(d,: d).
3.2
THE NULL SPACE OF T
The (left) null space LN(T) of T describes all vectors that are mapped to the zero vector by T:
LN(T) { E BRIxdd xxd 1t TT T 0}
(4)
Its special property is that, for À [a], 22 Fd, 1T where a! are vectors in this null space,
(A + À)T AT.
If the dimension of LN(T) is not zero, there exist infinitely many attention weights A + Á yield- ing the exact same attention layer output and final model outputs. By applying the Rank-Nullity theorem, the dimension of the null space is:
dd. - dv. d, > d, (0,
dim(LN(TT) de rank (T) d. min (d,, de)
(6)
otherwise
here we make use of the facts that dim (im(L((T)) dim(N(T") and rank(T)) = rank(TTT) where N(T) represents the null space of matrix T. Equality holds if E, WV and H are of full rank and their matrix product does not bring further rank reductions.
Hence, when the sequence length is larger than the attention head dimension (d, > d,), self- attention is not unique. Furthermore, the null space dimension increases with the sequence length. In the next section show that the attention weights are also non-identifiable; i.e., the non-trivial null space of T exists, even when the weights are constrained within the probability simplex.
3.3 THE NULL SPACE WITH PROBABILITY CONSTRAINTS
Since A is the result of softmax operation, its rows are constrained within the probability simplex: A > O (element-wise), and A1 1, where E Rd. is the vector of all ones. However, the derivation in Section (3.2) does not take these constraints into account. shows that A is not unique, but it does not prove that alternative attention weights exist within the probability simplex, and thus that A is not identifiable. Below, show that A exists in LN(T) also when constraining the weights of (A Á) the probability simplex.
3

---

Published as a conference paper at ICLR 2020
For the row vectors from an alternative attention matrix A A to be valid distributions, we require. in
addition to A > 0 (clement-wise and A1 1. that A+A2 2 0 (iementtwwsse) and A1 + A1 1.
Furthermore, A must be in the (left) null space of T. We formalize the three conditions below:
a) ÀT
b) A1
c) A A
Conditions (7 and (7) can be combined as A A 1 O, where (T, 1] is the augmented matrix re-
sulting from adding column vector of ones to T. Reusing the argument presented in Section (3.2). the dimension of its (left) null space is: dim(LN([T) ([T(()))) 11) 2 max(d, - d, 1, 0). Thus, for d2 d, 1, the null space of [T, 1] exists: it is a linear subspace of LT).).
We now prove that condition (7+) can also be satisfied. We begin by providing an intuitive justifica- tion. The condition restricts the space of from ( 1]) to be a bounded region which could be different for each row vector ing a surface passing through the origin. Since a is a probability vector, resulting from a softmax transformation, each of its components is strictly positive, i.e., A (element-wise). Hence, there exists > 0 such that any point à in the sphere centered at the origin with radius e will satisfy condition (7)
(a1, .2....) of A. The null space LN (T. 1) contains O, defin-
aa. Crucially, this sphere intersects the null space, as they share the origin. Any point in this intersection satisfies all three conditions in
Formally, the construction of the null space vector for the alternative attention weights à + a goes as follows. For a vector à = (ā1, ā.....) E LN([T, 1l), to ensure one of its negative components a, < 0 satisfying condition (7)) that a; 2 -a;, one could shrink its magnitude into λa with 0 S A v -a:/0s, so Ad; 2 -q. Considering all negative components i, the overall scaling factor is Amaz { a-aaaaa/_ so that the direction from the origin {Aâ]o S X S Amaz) satisfies condition (7). Here Amaz is strictly greater than 0 because a; 0. Only when there exists an index i that a; < 0 and a; R O, then this particular null space direction ā is highly confined. In the extreme case, where is not distribution, the solution should be an â with only one negative component. If such an à does not exist in LN([T, 1)), the solution collapses to the trivial single point ā = O. However, in general, LN((T, 1]) with probability constraints is non-trivial
3.4 EFFECTIVE ATTENTION
The non-identifiability of self-attention, due the existence of the non-trivial null space of T, challenges the interpretability of attention weights. However, one can decompose attention weights A into the component in the null space A and the component orthogonal to the null space A
AT + A-T
since All E LN(T) - A 0. Hence, we propose novel concept named effective attention,
A = A Projection; A,
which is the part of the attention weights that actually affects the model output. The null space pro- jection is calculated by projecting attention weights into the left null space basis, i.e., the associated left singular vectors.
Here the definition of effective attention uses LN(T) instead of the null space LN([T, 1]) with probability constraints. As a consequence, effecttve attention is not guaranteed to be probability distribution; e.g., some of the weights might be negative. One could define effective attention as the minimal norm alternative attention using LN((T, 1)), possibly constrain it within the probability simplex. However, in this case, the minimal norm alternative attention is not orthogonal to the null space LN(T)) anymore. It seems unclear how to interpret the minimal norm alternative attention. The reason being that the distinction between components that affect or do not affect the output computation are defined with respect to LN(T) and not with respect to LN([T, 1]). In fact, there may be useful information in the sign of the effective attention components. Although the combi- nation of value vectors the transformer architecture uses weights the probability simplex, these probability constraints on attention may not be necessary. Hence, we provide here the base version of an effective attention, and leave the investigation of other formulations for future research.
A key aspect of the concept of identifiability.
4

---

Published as a conference paper at ICLR 2020
0.8
Effective Attention
0.8
[CLS]
Raw Attention
Pearson '
0.8
0.6
0.6
0.4
0.4
0.6
0.2
0.2
0.4
0
50
100
5
10
5
10
Seq.
Length
Layer
Layer
(a)
(b)
(c)
Figure I: (a) Each point represents the Pearson correlation coefficient of effective attention and raw attention function of token length. (b) Raw attention vs. (c) effective attention, where each point represents the average (effective) attention of a given head to a token type.
3.5 EMPIRICAL EVIDENCE
We conclude by providing some initial empirical evidence in support of the notion that effective at- tention can serve as a complementary diagnostic tool for examining how attention weights influence model outputs.
First, we show that effective attention can be detected, and can diverge significantly from raw at- tention. In Figure Ta illustrate how the Pearson correlation between effective and raw attention decreases with sequence length. We use the same Wikipedia samples as in Clark et al. (2019) with maximum sequence length 128. This result is in line with theoretical finding in Eq. that states an increase in the dimension of the null space with the sequence length. Given a bigger null space, more of the raw attention becomes irrelevant, yielding a lower correlation between effective and raw attention. Notice how, for sequences with fewer than de 64 tokens, the associated null space dimension is zero, and hence attention and effective attention are identical (Pearson correlation of value 1). This loss of correlation with increased sequence length questions the use of attention as explanation practical models, where it not uncommon to use large sequence lengths. few ex- amples include: BERT for question answering (Alberti al. 2019 and XL-Net (Yang et al. 2019a) with dg = 512, or document translation (Junczys- Dooosss-Dowmtttt 2019) with de = 1000.
To illustrate the point further, Figure I shows in (b) raw attention A and (c) effective attention A+. using again the data olark et al. (2019). We compute the average attention of BERT and compare it to the corresponding average effective attention. Clark et al. (2019) conclude that the [CLS] token attracts more attention in early layers, the [SEP] tokens attract more in middle layers, and periods and commas do in deep layers. However, after gradient based investigation, they propose that attention to the [SEP] token generally a "no-op". The effective attention weights suggest a more consistent pattern: while periods and commas seem to generally attract more attention than [CLS] and [SEP], the peak of [SEP] token observed by raw attention has disappeared. Effective attention provides an explanation: the [SEP] token peak is irrelevant the computation of the output for middle layers; i.e., it is in the null space component of the corresponding attention vector. The same arguments also hold for the sharp peak of raw attention on punctuation tokens between layers 10 and 12. An additional example showing similar results can be found in Appendix A.2 See also Appendix [A.3] where we discuss in more depth a case where effective attention would support interpretive conclusions that differ from those might draw solely based on raw attention. In conclusion, effective attention can help discover interesting interactions encoded the attention weights which may be otherwise obfuscated by the null attention.
4 TOKEN IDENTIFIABILITY
We now study the other fundamental element of transformers; the hidden vector representations of tokens, contextual word embeddings. It is commonly assumed that contextual word embedding
5

---

Published as a conference paper at ICLR 2020
keeps its "identity", which is tied to the input word, it passes through the self-attention layers. Specifically, we identify three cases where this assumption is made implicitly without justification.
Visuliiztions/nneepettaiisss linking attention weights to attention between words, when in fact the attention is between embeddings. i.e., mixtures of multiple words (Vaswani
et al.,
2017, Devlin et al. 2019; Vig. 2019 Clark et al., 2019. Raganato & Tiedemann,
2018
Voita et al. 2019; Tang et al. 2018| Wangperawong 2018 Padigela et al., 2019
Baan et al. 2019 Dehghani et al. 2019 Zenkel et al. 2019).
Attention accumulation methods that sum the attention to a specific sequence position over layers and/or attention heads, when the given position might encode different mixture of
inputs in each layer (Clark et al. 2019 Baan et al. 2019 Klein & Nabi 2019: Coenen
et al. 2019).
Using classifiers to probe hidden embeddings for word-specific aspects without factoring in how much the word is still represented (Lin et al. 2019 Peters et al. 2018).
To investigate this assumption we introduce the concept of token identifiability,, as the existence of mapping assigning contextual embeddings to their corresponding input tokens. Formally, we state that an embedding el is identifiable there exists a classification function c(-) such that c(e)) = 24- For identifiability we only require c(e)) to recover 2; in a nearest neighbour sense within the same input sentence. Therefore, for each layer / we define cx(-) NN (-)), where NN is a 1-nearest neighbour lookup, and 9I : I2d - Rd is a continuous function mapping embeddings to vectors of real numbers. Since we cannot prove the existence of gi analytically, we instead use a function approximator ği (ee)) é;, trained on a dataset of (el, ax;) pairs. We then say that et is identifiable (e() = NN (ā)(e)) = x,, For evaluation we report the token identifiability rate defined as the percentage of correctly identified tokens.
4.1 SETUP
For the experiments in this and subsequent sections we use the development dataset from the Microsoft Research Paraphrase Corpus (MRPC) dataset (Dolan & Brockett 2005), while in Ap- pendix D we provide results on two additional datasets. The MRPC development set contains 408 examples with sequence length d, between 26 and 92 tokens, with 58 tokens on average. We pass all 408 sentences (21,723 tokens) through BERT and extract for each token the input embeddings æ; and the hidden embeddings e at all layers. We then train ğ on the regression task of predicting input tokens x; from hidden tokens e/. We experiment with two loss funetions and similarity measures for finding the nearest neighbour; cosine distance and L2 distance. We use 10-fold cross validation with 70/15/15 train/yalidationnnest splits per fold and ensure that tokens from the same sentence not split across sets. The validation set is used for early stopping. See Appendix B.] for details.
4.2 EXPERIMENTAL RESULTS AND DISCUSSION
In a first experiment, we use linear perceptron without bias and a non-linear MLP GML,, where training, validation and test data all come from layer 1. Figure 2a shows the test set token iden- tifiability rate of & for 1 = [1, (1.... 12). We also report naive baseline graive (e!) =e/ i.e., we directly retrieve the nearest neighbour of eş from the input tokens. The results for graive show that, according to both similarity measures, contextual embeddings in BERT stay close to their input embeddings up layer 4, followed by linear decrease in token identifiability rate. By training a transformation to recover the original embedding, we see that most of the identity information is still present in the contextualized embeddings. Specifically, a linear projection is enough to recover 93% of the tokens in the last layer based on a cosine distance nearest neighbour lookup.
This experiment shows that although the identifiablity rate decreases with depth, tokens remain mostly identifiable across layers. Furthermore, we find that cosine distance is more effective to recover identity than Le distance. Therefore, we conjecture that BERT encodes most of the identity information in the angle of the embeddings. Finally, Lin et al. (019)) show that BERT discards much of the positional information after layer 3. However, tokens remain largely identifiable throughout the model, indicating that BERT does not only rely on the positional embeddings to track token identity. To provide further insights into contextual word embeddings. Appendix B.5 shows results for recovering neighbouring input tokens.

---

Published as a conference paper at ICLR 2020
Identifiability Rate
1
Identifiability Rate
0.8
0.9
0.6
0.8
0.4
Scos,
LP
1=6
0.2
din
Greree
0.7
10
12
2
6
10
12
Layer
Layer
(a)
(b)
Figure 2: (a) Identifiability of contextual word embeddings at different layers. Here, is trained and tested on the same layer. (b) geos,1 trained on layer and tested on all layers.
In second experiment test how well the م trained only (el, a::) pairs from layer / generalizes to all layers, Figure 2b For / = 1, the token identifiability rate on subsequent layers drops quickly to below 70% at layers 11 and 12. Interestingly, for = 12 a very different pattern can be observed, where the identifiability is 94% for layer 12 and then almost monotonically increases when testing on earlier layers. Further, for / = 6 we see both patterns.
This experiment suggests that the nature of token identity changes tokens pass through the model, and patterns learned on data from later layers transfer well to earlier layers. The experiment also shows that layer 12 is behaving differently than the other layers. In particular, generalizing to layer 12 from layer 11 seems to be difficult, signified by sudden drop in token identifiability rate. We believe this is due task dependent parameter adaptation induced in the last layer by the next- sentence prediction task which only uses the CLS token (cf. Appendix B.4 for additional hints that M.PP and 9205.1
the last layer(s) behave differently). See Appendix B.3 for results of 9 912.1
Overall, the results of this section suggest that input token, for example for drawing conclusions based on (effective) attention weights. However, self-attention has the potential strongly mix tokens across multiple layers, and hence it is unclear whether token identifiability alone is enough to equate hidden embeddings with their input words, or whether we also need take into account exactly how much of the word is still contained in the hidden embedding. In order to address this question, we now study the degree of information mixing among embeddings, and introduce tool to track the contributions of tokens to embeddings throughout the model.
associate most hidden embeddings with their
5 ATTRIBUTION ANALYSIS TO IDENTIFY CONTEXT CONTRIBUTION
We consider the role of the contextual information in the hidden embeddings, which is accumulated through multiple paths in a multi-layer network. To shed more light on this process, introduce Hidden Token Attribution, context quantification method based on gradient attribution (Simonyan
al. 2014) to investigate the hidden tokens' sensitivity with respect the input tokens.
5.1 HIDDEN TOKEN ATTRIBUTION
Gradient based attribution approximates the neural network function f(X) around given sequence of input word embeddings X dgd. xd by the línear part of the Taylor expansion:
(X+ f(X) Vxf(X AX
(10)
With this, the network sensitivity is analyzed by looking how small changes AX the input correlate with changes at the output. Since the linear approximation this change is given by the
gradient Va.f f 5f(X) for change in the i-th input token æ, E IRd of X, the attribution of how much input token æ; affects the network output ((X) be approximated by the To2 norm

---

Published as a conference paper at ICLR 2020
Contribution [%
30
30
P [%)
20
20
10
10
0
0
2
3
4
5
6
8
9
10
12
1
2
3
4
5
6
7
8
9
10
11
12
Layer
Layer
(a)
(b)
Figure 3: (a) Contribution of the input token to the embedding at the same position. The orange line represents the median value and outliers not shown. (b) Percentage of tokens P that not the main contributors to their corresponding contextual embedding at each layer.
of the respective gradient: attr( VV fll2- Since interested in how much given hidden embedding e/, attributes to the imput tokens H., [1, dg]. define the relative input contribution d,, input æ; to output f(X) - e/ as
VV,lll CR /
5e
with
(11)
5x;
Since we normalize by dividing by the sum of the attribution values to all imput tokens, we obtain values between and 1 that represent the contribution of each input token æ; to the hidden em- bedding e/,- Hidden Token Attribution differs from the standard use of gradient attribution in that, instead of taking the gradients of the output the model with respect to the inputs order to explain the model's decision, we calculate the contribution of the inputs intermediate embeddings in order to track the mixing of information. Further details of this method are discussed in Appendix C.T
5.2 TOKEN MIXING: CONTRIBUTION OF INPUT TOKENS
We Hidden Token Attribution to extend the results of Section 4 showing how much of the input token is contained in a given hidden embedding. In Figure 3a we report the contribution c) of input tokens z, to their corresponding hidden embeddings e the same position for each layer After the first layer the median contribution of the input token is less than a third (30.6%). The contribution then decreases monotonically with depth; at layer 6 the median is only 14.4% and after the last layer is 10.7%. Appendix C.5 we provide detailed results by word type. Next, we study which input token is the largest contributor a given hidden embedding e/. The corresponding
input token æj generally has the largest contribution. Figure 3b shows the percentage P of tokens that are not the highest contributor their hidden embedding at each layer. the first three layers the original input 2, always contributes the most to the embedding e/, In subsequent layers, P increases monotonically, reaching 18% in the sixth layer and 30% in the last two layers.
These results show that, starting from layer three, self-attention strongly mixes the input information by aggregating context into the hidden embeddings. This is in line with the results from Section where we see decrease in token identifiability rate after layer three. Nevertheless, P is always higher than the token identifiability eπror the same layer, indicating that tokens are mixed in way that often permits recovering token identity even if the contribution of the original token is outweighed by other tokens. This suggests that there is some "identity information" that is preserved through the layers.
The strong mixing of information questions the common assumption that attention distributions can be interpreted "how much a word attends to another word". However, the fact that tokens remain identifiable despite information mixing opens a number new interesting questions to addressed by future research. In particular, this seeming contradiction may be solved by investigating the space in which hidden embeddings operate: is there a relation between semantics and geometric distance for hidden embeddings? Are some embedding dimensions more important than others?
8

---

Published as a conference paper at ICLR 2020
Rel. Contribution (%)
10
12
Total Contribution
6
8
0.5
6
Isl
3rd
4th and 5th
0
4
1011 12
-60
-40
-20
0
40
Layer
Neighbour
(a)
(b)
Figure 4: (a) Relative contribution per layer of neighbours different positions. (b) Total contribu-
tion per neighbour for the first, middle and last layers.
5.3 CONTRIBUTION CONTEXT TO HIDDEN TOKENS
In this section we study how context is aggregated into hidden embeddings. Figure 4a shows the relative contribution of neighbouring tokens at each layer for the relative positions: first, second, third, fourth and fifth together, sixth to 10th together, and the rest. The closest neighbours (Ist) contribute significantly more the first layers than in later layers. Conversely, the most distant neighbours (11th onwards) contribute the most in deeper layers (ef. Appendix (C.2). Despite the progressive increase in long-range dependencies, the context in the hidden embeddings remains mostly local. Figure 4b represents the normalized total contribution aggregated over all tokens from each of their neighbours at the first, middle and last layer. This figure shows that the closest neighbours consistently contribute the most to the contextual word embedding regardless of depth. On the other hand, we indeed observe an increase of distant contributions at later layers.
The results of this section suggest that BERT learns local operators from data in an unsupervised manner, in the absence of any such prior in the architecture. This behavior is not obvious, since attention highly non-local operator, and turn indicates the importance of local dependencies in natural language. While contribution is local average, we find that there are exceptions, such as the [CLS] token (cf. Appendix C.3). Furthermore, using our Hidden Token Attribution method, one can track how context is aggregated for specific tokens (cf. Appendix C.4
RELATED WORK
Input-output mappings play key role in NLP. For example, in machine translation, they were in- troduced in the form explicit alignments between source and target words (Brown et al.) 1993). Neural translation architectures re-introduced this concept in the form of atfention (Bahdanau al., 2013). The development of multi-head self-attention Vaswanni et al.) (207)) has led to many impres- sive results in NLP. As a consequence, much work has been devoted better understand what these models learn, with a particular focus on using attention to explain model decisions.
Jain & Wallace (2019) show that attention distributions of LSTM based encoder-decoder models not unique, and that adversarial attention distributions that do not change the model's decision can be constructed. They further show that attention distributions only correlate weakly moderately with dot-product based gradient attribution. WWiereeffe & Pinter (2019) also find that adversarial attention distributions can be easily found, but that these alternative distributions perform worse on a simple diagnostic task. [Serrano & Smith (2019) find that zero-ing out attention weights based on gradient attribution changes the output of a multi-class prediction task more quickly than zero-ing out based attention weights, thus showing that attention is not the best predictor of learned fea- ture importance. Pruthi et al. (2019) demonstrate that self-attention models can be manipulated to produce different attention masks with very little cost in accuracy. These papers differ in their ap- proaches, but they all provide empirical evidence showing that attention distributions are not unique with respect downstream parts of the model (e.g., output) and hence should be interpreted with

---

Published as a conference paper at ICLR 2020
care. Here, support these empirical findings by presenting theoretical proof of the identifia- language models with bility of attention weights. Further, while these works focus on self-attention models. Our single layer of attention, we instead consider multi-heat head multi-layer
token classification and token mixing experiments show that non-identifiable tokens increase with depth, further reinforeing the point that the factors that contribute to the mixing of information complex and deserve further study.
Voita et al. (2019) and Michel al. (2019) find that only a small number of heads in BERT have relevant effect on the output. These results are akin to ours about the non-identifiability of attention weights, showing that a significant part of attention weights do not affect downstream components. One line of work investigates the internal representations of transformers by attaching probing clas- sifiers to different parts of the model. Tenney et al. (2019) find that BERT has learned to perform steps from the classical NLP pipeline. Similarly, Jawahar et al. (2019) show that lower layers of BERT learn syntactic features, while higher layers learn semantic features. They also argue that long-range features are learned in later layers, which agrees with our attribution- basod experiments.
7 CONCLUSION
We used the notion of identifiability to gain a better understanding of transformers from different yet complementary angles. We started by proving that attention weights are non-identifiable when the sequence length is longer than the attention head dimension. Thus, infinitely many attention distri- butions can lead to the same internal representation and model output. As an alternative, we propose effective aftention, method that improves the interpretability of attention weights by projecting out the null space. Second, we show that tokens remain largely identifiable through a learned linear transformation followed by a nearest neighbor lookup based on cosine similarity. However, input to- kens gradually become less identifiable in later layers. Finally, we present Hidden Token Attribution, a wradient-based method to quantify information mixing. This method is general and can be used to investigate contextual embeddings in self-attention based models. In this work, we use it to demon- strate that input tokens mix heavily inside transformers. This result means that ittention-based interpretations, which suggest that word at some layer is attending to another word can be im- proved by accounting for how the tokens are mixed inside the model. We further show that context progressively aggregated into the hidden embeddings while some identity information is preserved. Moreover, we show that context aggregation is mostly local and that distant dependencies become relevant only in the last layers, which highlights the importance of local information for natural lan- guage understanding. Our results suggest that some of the conclusions in prior work (Vaswani et al.,
2017
Vig 2019; Marreekk & Rosa 2018 Clark et al. 2019 Raganato & Tiedemann] [2018] Voita
et al.
2019:
Tang et al.|_2018 Wangperawong 2018
Padigela et al. 2019 Baan et al. 2019 Lin
al.
2019
Dehghani et al.) 2019, Zenkel al.; 2019 Klein & Nabil 2019 Coenen et al.) 2019)
may be worth re-examining from this perspective.
There are still many open questions for future research. For one, by constraining effective atten- tion to the probability simplex. one could better compare it to standard attention, although in this case non-influencing parts would be included in the weights. More research needed to better un- derstand the differences between these formulations. Further, we find that tokens mix and remain largely identifiable. While these two conclusions are not necessarily at odds a token can both gather context information and still retain the essence of the original input word - we believe that the rela- tionship between mixing and identifiability warrants further investigation. Moreover, it is becoming increasingly difficult to compare all the new transformer variants, and it is hence important to gain a deeper understanding of this class of models. The concepts introduced in this paper could help in identifying fundamental differences and commonalities between variants of self-attention models.
ACKNOWLEDGEMENTS
We would like to thank the reviewers and area chairs for their thorough technical comments and valuable suggestions. We would also like to thank Jacob Devlin for feedback on an early draft. This research was supported with Cloud TPUs from Google's TensorFlow Research Cloud (TFRC).
10

---

Published as a conference paper at ICLR 2020
REFERENCES
Chris Alberti, Kenton Lee, and Michael Collins. A BERT Baseline for the Natural Questions.
https://arxi..org/b///10111.1 08634, 2019.
Joris Baan, Maartje ter Hoeve, Marlies van der Wees, Anne Schuth, and Maarten de Rijke. Do transformer attention heads provide transparency in abstractive summarization? CoRR, has/1907.005700 2019.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning align and translate. 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL
http: pttt:://axiinn /wtp/v.aiii//.a///a//e../...aa.. http:///xxvvvo..aaaaa/
www.oo/g//ao... abs///b///// 14014.19.00 404404733333 0473
R. Bellman and Karl Johan Aström. On structural identifiability. Mathematical Biosciences, 7: 329-339, 1970.
Peter F. Brown, Stephen A. Della Pietra, Vincent Della Pietra, and Robert L. Mercer. The math- ematies of statistical machine translation: Parameter estimation. Computational Linguistics, 19 (2):263-311, 1993. URL
org/ http://wwww......//..oooooooooa.
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. What does BERT look at? an analysis of bert's attention. CoRR, abs/ 1eds//0000..441,, 2019.
Andy Coenen, Emily Reif, Ann Yuan, Been Kim, Adam Pearce, Fernanda B. Viégas, and Martin Wattenberg. Visualizing and measuring the geometry of BERT. CoRR, abs/1906.027155, 2019.
Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. Universal transformers. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLL 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume I (Long and Short Papers), pp. 4171-4186, 2019.
William B. Dolan and Chris Brockett. Automatically constructing corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing, WWP@IJCNLA 2005, Jeju Island, Korea, October 2005, 2005, 2005.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings the Thirteenth International Conference on Artificial Intelligence and Statistics, AISTATS 2010, Chia Laguna Resort, Sardinia, Italy, May 13-15, 2010, pp. 249-256, 2010. CtL.tttp::/// ptctp::/prrocedinng: ml wwwpprsssvvie vtess/v9/qllrree 0a .ttml
Dan Hendrycks and Kevin Gimpel. Bridging nonlinearities and stochastic regularizers with gaussian error linear units. CoRR, abs/1606.084i55, 2016. URL http: http:////xxiv.vo wrtp:///.rrabb//5.....
08415
Sarthak Jain and Byron C. Wallace. Attention is not explanation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, 2019.
Volume 1 (Long and Short Papers), pp.
Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. What does BERT learn about the structure of language? In Proceedings of the 57th Conference of the Association for Computational Linguis- tics, ACL 2019, Florence, Italy, July 28- August 2019, Volume 1: Long Papers, pp. 3651-3657, 2019. URL www.l:ttpsp... www. ac web. org/ anthol oog./ 1356/
Marcin Junczys- Dowmunt. Microsoft translator at wmn 2019: Towards large- document-level neural machine translation. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pp. 225-233, 2019. 2019.

---

Published as a conference paper at ICLR 2020
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd Inter- national Conference on Learning Representations. ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. Uttp:tt/://axxi wtwp:///.rggaa.////.//a// 1412 6980
Tassilo Klein and Moin Nabi. Attention is (not) all you need for commonsense reasoning. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy July 28- August 2, 2019, Volume I: Long Pupers, pp. 4831-4836, 2019.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Sori- cut. Albert: A lite bert for self- self-supervised learning of language representations. In International Conference Learning Representations, 2020. URL WRRLhttps://ppeee /ttpa//////peeeeeee//e/ www.:eertffuuttmooo idd-HeeAAEtt
Yongjie Lin, Yi Chern Tan, and Robert Frank. Open sesame: Getting inside bert's linguistic knowl- edge. arXiv preprint arXiv: 190600080,, 2019.
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint 1907.1907..11990 2019.
David Marecek and Rudolf Rosa. Extracting syntactic trees from transformer encoder self- attentions. In Proceedings of the Workshop: Analyzing and Interpreting Neural Networks for NLP, BlackboxNLP@
2018, Brussels, Belgium, November 1, 2018, pp. 347-349, 2018.
Paul Michel, Omer Levy, and Graham Neubig. Are sixteen heads really better than one? CoRR,
abs/1905.10057t 2019, URL htttht/t///aae /ttp//aaaivvv///m///i org/ aaa/aab/s1111/ abs/1905. 10650
Harshith Padigela, Hamed Zamani, and W. Bruce Croft. Investigating the successes and failures of BERT for passage re-ranking. CoRR, 0ass/1905.01755 2019.
Matthew E. Peters, Mark Neumann, Luke Zettlemoyer, and Wen-tau Yih. Dissecting contextual word embeddings: Architecture and representation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium. October 31 November 4, 2018, pp. 1499-1509, 2018
Nina Pörner, Hinrich Schütze, and Benjamin Roth. Evaluating neural network explanation methods using hybrid documents and morphosyntactic agreement. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACI. 2018, Melbourne, Australia, July 15-20, 2018, Volume Long Papers, pp. 340-350, 2018
Danish Pruthi, Mansi Gupta, Bhuwan Dhingra, Graham Neubig, and Zachary C Lipton. Learning to deceive with attention-based explanations. arXiv preprini urXiv:1909.07900 2019.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language un- derstanding by generative pre-training. URL http:://s3-us-westt amazonaws. com/openai- understanding paper. pdf. 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl Blog, 1(8), 2019.
Alessandro Raganato and Jörg Tiedemann. An analysis of encoder representations in transformer- based machine translation. In Proceedings of the Workshop: Analyzing and Interpreting Neural Networks for NLP,, BlackboxNI.PGr EMNLP 2018, Brussels, Belgium, November 2018. pp. 287- 297, 2018
Sofia Serrano and Noah A. Smith. Is attention interpretable? In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers. pp. 2931-2955 2019. URL https::/ wwwww..wlwb.co.. org/
anthology/219 1282/
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. In 2nd International Conference Learning Representations, ICLR 2014, Banff. AB, Canada, April 14-16, 2014, Workshop Track Proceedings, 2014.
12

---

Published as a conference paper at ICLR 2020
Gongbo Tang, Rico Sennrich, and Joakim Nivre. An analysis of attention mechanisms: The case of word sense disambiguation in neural machine translation. In Proceedings of the Third Con- ference Machine Translation: Research Papers, WMT 2018, Belgium, Brussels, October 31 November 1, 2018, pp. 26-35, 2018.
Ian Tenney, Dipanjan Das, and Ellie Pavlick. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Conference of the Association for Computational Linguistics. ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pp. 4593-4601, 2019.
Kristina Toutanova, Dan Klein, Christopher D. Manning, and Yoram Singer. Feature-rich part-of- speech tagging with a cyclic dependency network. In Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, HLT-NAACL 2003, Edmonton, Canada, May 27- June 1, 2003, 2003.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances Neural Infor- mation Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA, pp. 5998-600 6608, 2017.
Jesse Vig. Visualizing attention in transformer-based language representation models. CoRR, Abs/1994.02669, 2019.
Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Pupers, pp. 5797-5808, 2019.
Artit Wangperawong. abs/1812.02822 2018
Attending to mathematical language with transformers.
CoRR,
Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. Neural network acceptability judgments. arXiv preprint arXiv: 1188805..244.. 2018
Sarah Wiegreffe and Yuval Pinter. Attention is not not explanation. CoRR, abs/1998.00422 2019.
URL htttthttt://:rttte org abs 110.. 04626
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sen- tence understanding through inference. In Proceedings the 2018 Conference the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long Papers), pp. 11 1112-1122, 122.. Association for Computational Linguisties, 2018. URL
http: http///cwleeee org/ ant ho cochllosh loollooy//// /N18- 1101
Baosong Yang. Zhaopeng Tu, Derek F. Wong, Fandong Meng, Lidia S. Chao, and Tong Zhang. Modeling localness for self-attention networks. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 Novem- ber 4, 2018 pp. 1449-4455 2018. URL http:///ccaatthoooyy.o wwwpaahhoooo....ooaaaaaaeeee wnwfopppprss//////
D18 1475/a18 1475
Zhilin Yang. Zihang Dai, Yiming Yang. Jaime Carbonell, Ruslan Salakhutdinov, and Quoe V. Le. Xlnet: Generalized autoregressive pretraining for language understanding. 2019a.
Zhilin Yang. Zihang Dai, Yiming Yang. Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in neural information processing systems, pp. 5754-5764, 2019b.
Thomas Zenkel, Joern Wuebker, and John DeNero. Adding interpretable attention to neural transla- tion models improves word alignment. CoRR, abs/19011.11355 2019.
13

---

Published as a conference paper at ICLR 2020
A IDENTIFIABILITY OF SELF-ATTENTION
A. BACKGROUND ON ATTENTION IDENTIFIABILITY
Often, the identifiability issue arises for a model with a large number of unknown parameters and limited observations. Taking a simple linear model T1 + I2 as an example, when there only one observation (y, F1, x2). model parameters 31 and $2 cannot be uniquely determined. Moreover, in the matrix form Y = X,3, by definition the parameter 3 is identifiable only if Y 3 and - X imply 31 = 32. if the null space contains only the zero solution (3 0J {0). i.e., X3, X ((1 32) = - 31 B2 0, the model is identifiable.
Therefore, the identifiability parameters in a linear model is linked to the dimension of the null space, which in the end is determined by the rank of X.
A.2 ADDITIONAL RESULTS OF THE EFFECTIVE ATTENTION VS. RAW ATTENTION RESULTS
In Figure we provide a recreation of the figure regarding the attention of tokens towards the [SEP] token found in (Clark et al. 2019 Figure 2) with average attention as well as average effective attention. Again, we see that most of the raw attention lies effectively in the null space, confirming the pattern of Figure The figures are produced using the code from Clark al. (2019).
Effective Attention
other + [SEP]
Raw Attention
[SEP] [SEP]
0.5
0.5
0
0
2
12
2
12
Layer
Layer
(a)
(b)
Figure 5: Effective attention (a) attention (b). (a) Each point represents the average effec- tive attention from token type to a token type. Solid lines are the average effective attention of corresponding points each layer. (b) is the corresponding figures using attention weights.
A.3 A CLOSER LOOK AT EFFECTIVE ATTENTION WEIGHTS
Here discuss example of how effective attention might lead to interpretive conclusions that differ from raw attention. Figure plots the attention weights (raw, effective, null) from one of the attention heads in BERT'S layer for the following passage:
"[CLS] research into military brats has consistently shown them to be better behaved than their civilian counterparts. [SEP] hypotheses as why brats are better behaved: firstly, military parents have a lower threshold for misbehavior in their children; secondly, the mobility of teenagers might make them less likely to attract attention to themselves, with their surroundings; and thirdly, normative constraints are greater, with brats knowing that their behavior under scrutiny and can affect the military member's career. teenage years are typically period when people establish independence by taking some risks away from their [SEP]".
many want fit and are less secure
For readability, on the y-axis, we consider just the sentence "the mobility of teenagers might make them less likely to attract attention to themselves, as many want to fit in and are less secure with their surroundings".
The following seems worth noticing:
Raw attention weights are by and large concentrated either on the structural components, [CLS] and [SEP), or the semi-monotonie, near diagonal alignments.
14

---

Published as a conference paper at ICLR 2020
Effective attention weights more uniform, in general. They still concentrated near the diagonal elements, although less so than raw attention. However, the attention on [CLS] and [SEP] has vanished. The collapse of the [CLS] and [SEP] weights brings to the surface other interesting things. As an example, we point out the highest weight on the attention matrix, that is not the diagonal. This involves (highlighted by means of the yellow lines) the main verb of the selected sentence, "make", whose object is "them" (teenagers), and the pronoun "them" (the direct object of the first sentence, "military brats", 48 positions away). The two are co-referential, as both refer to the main subject of the passage, military brats.
Null attention weights are also more uniform than raw attention ones. Interestingly, they seem to carry all the mass of the [CLS] and [SEP] tokens. There visible degree of redundancy between the null attention weights and the effective ones, but also clear com- plementary elements.
One should not extrapolate too much from a single observation. Further research needed on this topic. However, this example is proof of concept that raw and effective attention diverge qualitatively, in significant ways. It agrees with the hypothesis that the weights on the structural components may act as sinks, observed in (Clark et al. 2019), but also tells us how this happens. Furthermore, it indicates that attention in the null space can obfuscate other valuable interactions that may be recoverable by inspecting effective attention.
15

---

Published as a conference paper at ICLR 2020
16
Figure 6: Raw attention weights (top), Effective attention weights (middle) and Null attention weights (bottom).

---

Published as a conference paper at ICLR 2020
B TOKEN IDENTIFLABILITY EXPERIMENTS
B.1 EXPERIMENTAL SETUP AND TRAINING DETAILS
The linear perceptron and MLP are both trained by either minimizing the L2 or cosine distance loss using the ADAM optimizer Kingmaa & Ba 2015) with learning rate of 0.0001, f1 = 0.9 and 32 0.999. We use batch size 256. We monitor performance on the validation set and stop training if there is no improvement for 20 epochs. The input and output dimension of the models is = 768; the dimension of the contextual word embeddings. For both models we performed learn- ing rate search over the values weights are initialized with the Glorot Uniform initializer (Glorot & Bengio 2010 The MLP has one hidden layer with 1000 neurons and uses the gelu activation function (Hendryeks & Gimpel 2016), following the feed-forward layers BERT and GPT. We chose a hidden layer size of 1000 in order avoid bottleneck. We experimented with using larger hidden layer of size 3072 and adding dropout to more closely match the feed-forward layers in BERT. This only resulted in increased training times and hence deferred from further architecture search.
[0.003, 0.001, 0.0003, 0.0001, 0. 0.00003, 0. 0.00001, 0. 0.000003). The
We split the data by sentences into train/validation/tesst according to a 70/15/15 split. This way of splitting the data ensures that the models have never seen the test sentences (i.e., contexts) during training. In order to get a more robust estimate of performance we perform the experiments in Figure 2a using 10-fold cross validation. The variance, due the random assignment of sentences to train/validation/tsst sets, is small, and hence not shown.
B.2 GENERALIZATION ERROR
Figure shows the token identifiability rate for train and test set for both models, linear and MLP, when using I.2 distance. Both models overfitting the same degree. The fact that the linear model has about the same generalization error as the MLP suggests that more training data would significantly increase performance
the test Further, we trained the MLP on layer 11 using 50%, 80%, 90% and 100% of the training data set. The MLP achieved the following token identifiability rate on the test set: 0.74, 0.8, 0.81, 0.82. This indicates that the MLP would profit much from more data.
We do not report the generalization error for the models trained minimize cosine distance, as the linear and non-linear perceptrons perform almost equally.
I-NN Recovery Accuracy
Linear Train
0.9
Linear Test
MLP Train
0.8
MLP Test
2
4
6
10
12
Layer
Figure 7: Train and test token identifiability rates for the linear perceptron and MLP.
17

---

Published as a conference paper at ICLR 2020
B.3
ADDITIONAL RESULTS FOR FIGURE
Figure 2b in the main text only shows results of the linear perceptron trained to minimize cosine distance layers / = [1, 6, 11, 12] and tested all other layers. Figures 8 and 10 show the corresponding results for the linear perceptron trained to minimize cosine distance, and for the MLP trained minimize L2 and cosine distance respectively. Overall, all figures show the same qualitative trends as presented in Section 4 of the main text: Generalizing to later layers works considerably worse than the other way around. The linear pereeptrons seem to generalize better across layers, likely due to the MLPs overfitting more to the particular layers they are trained on.
Identifiability Rate
0.8
0.6
0.4
1=1
1=6
0.2
|=11
1=12
6
10
12
Layer
Figure 8: Linear Perceptron trained to minimize L2 distance generalizing to all layers.
Identifiability Rate
0.8
0.6
0.4
1=1
1=6
0.2
1=11
1=12
6
10
12
Layer
Figure 9: MLP trained to minimize L2 distance generalizing to all layers.
18

---

Published as a conference paper at ICLR 2020
Identifiability Rate
0.8
0.6
1=1
1=6
1=11
1=12
0.4
4
6
10
12
Layer
Figure 10: MLP trained to minimize cosine distance generalizing to all layers.
19

---

Published as a conference paper at ICLR 2020
B.4 TOKEN IDENTITY FROM HIDDEN TOKENS TO HIDDEN TOKENS
Figure [1 shows results for identifying tokens across single layers of BERT, i.e., the input to is (e). m)) in the first layer, and subsequently (e), e where = [1, ..... 12]. This experiment gives further insight into what kinds of transformations
applied by each transformer layer separately, as opposed to the cumulative transformations shown in Section ! of the main text. Interestingly, even the naive baselines perform well across single layers. This shows that BERT only applies small changes to the contextual word embeddings, whereas overall the angle (as indicated by the naive baseline using cosine distance) is affected less than the magnitude of the word embeddings (indicated by the naive baseline using L2 distance).
Figure shows that tokens are on average more difficult to identify across later layers. In the main text we hypothesize that the qualitative change seen in later layers could be due to task-specific parameter adaptation during the second (next sentence prediction) pre-training phase. possible reason is that during this tre-training-phase which qualitatively very different form the first (masked language modeling) pre-training phase, where potentially all the tokens needed in the last layer.
BERT only needs the [CLS] token in the last layer,
To further verify this hypothesis we experimented with BERT fine-tuned on two datasets, MRPC and CoLA (Warstadt et al. 2018). 1). During the fine-tuning phase, similar to the next sentence prediction pre-training phase, only the [CLS] token is needed at the last layer. If task-dependent parame- ter adaptation indeed has different influence the last layen(s) than earlier layers, then should be able to see a difference between the finetuned and non-finetuned cases. [2 T3 compare the naive baselines across single layers for BERT finetuned MRPC and CoLA, respec- tively. Indeed, one can see a remarkable decrease in identifiability across the last layer for L2-based nearest neighbour lookup, further indicating that the last layers are indeed more strongly affected by different fine-tuning objectives. Nearest neighbour lookup based on cosine distance is affected much less, indicating that in terms of token identifiability, the last layers are only slightly affected by fine-tuning.
Identifiailiity Rate
0.98
0.96
0.94
GMLP
Jin Icos,!
graine
912.1
912.1
922.1
0.92
2
3
4
5
6
8
10
11
12
Layer
Figure 11: Token identifiability across single layers. These results are for non fine-tuned BERT MRPC
20

---

Published as a conference paper at ICLR 2020
Tdentifiability Rate
0.8
J fine
912,1
0.6
anaive,fine
Jcor,1
1
2
3
10
11
12
Layer
Token identifiability across single layers, comparing non fine-tuned (dashed) BERT against BERT fine-tuned on MRPC (solid).
Figure 12:
Identifability Rate
0.95
0.9
annive gnaive
unioo,
gnaive,
0.85
2
3
4
5
10
12
Layer
Figure 13: Token identifiability across single layers, comparing non fine-tuned BERT (dashed)
against BERT fine-tuned on CoLA (solid).
21

---

Published as a conference paper at ICLR 2020
B.5 TOKEN IDENTITY RECOVER NEIGHBOURING INPUT TOKENS
In Section of the main text we show that tokens at position remain largely identifiable throughout the layers of BERT. In this section show results of a related experiment, where test how much information about tokens at neighbouring positions is contained
contextual word embedding. 11, 33. Thus, we try recover input 16 and 17 show the results of for Peor M.L.P.
More formally, the imput g is (e), T(tk). where E
token Li+h from hidden token e/. Figures 14
15
MLP and L22, respectively. In the figures, blue corresponds to "previous" tokens and red to "next" tokens.
From the figures can that tokens do contain information about neighbouring tokens that lets us recover the neighbouring tokens based on a transformation and subsequent nearest neigh- bour lookup. The identifiability rate drops both with increasing k, but also with increasing depth. Interestingly, recovering left (blue) and right (red) neighbours shows different behaviour, indicat- ing that BERT is treating left and right context differently, despite having been pre-trained using bi-directional language modeling objective.
Overall, neighbouring tokens be recovered to much lower degree than same-position tokens (cf. Sectior
i-l
i-2
Identifiability Rate
0.8
1-3
i+1
i42
i+3
0.6
0.4
0.2
1
2
3
4
5
6
8
9
10
11
12
Layer
Figure 14: Recovering neighbouring input tokens using geoe.t
i-l
i-2
Identifiability Rate
i-3
i+1
0.8
i+3
0.6
0.4
0.2
1
2
3
4
5
8
9
10
11
12
Layer
Figure 15: Recovering neighbouring input tokens using gmip
22

---

Published as a conference paper at ICLR 2020
i-1 ---
i-2
Identifiability Rate
0.6
1-3
i+1
i42
1+3
0.4
0.2
1
2
3
4
5
8
9
10
11
12
Layer
mlp Figure 16: Recovering neighbouring input tokens using 912.1"
0.6
i-1
i-2
Identifiability Rate
i-3
i+1
143
0.4
0.2
1
2
3
4
5
8
9
10
11
12
Layer
Figure 17: Recovering neighbouring input tokens using alin
23

---

Published as a conference paper at ICLR 2020
C CONTEXT CONTRIBUTION ANALYSIS
C.I HIDDEN TOKEN ATTRIBUTION: DETAILS
The attribution method proposed in Section 5.1 to calculate the contribution of input tokens to a given embedding does not look at the output of the model but at the intermediate hidden representa- tions and therefore is task independent. Since the contribution values do not depend on the task that is evaluated, we can compare these values directly
attention distributions, which are also task- independent. In this way, we can compare to other works in the literature (Vig, (019) Clark et al.] 2019 Klein & Nabi 2019; Coenen et al. 2019 Lin et al. 2019) by using the publicly available pretrained BERT model in our analyses without fine-tuning it to a specific task.
interested in analysing how the input affects the output of the model but in quantifying the absolute contribution of the input tokens to the hidden embeddings, we use the L2 norm of the gradients. If we were analyzing whether the input contributed positively or negatively to a given decision, the dot-product of the imput token embedding with the gradient would be the natural attribution choice (Pörner et al. 2018).
Furthermore, since
C.2 CONTEXT IDENTIHABILITY DETAILS
To calculate the relative contribution values shown in Figure 4a we firstly calculate the mean of the left and right neighbours for each of the groups of neighbours, i.c., first, second, third, fourth and fifth, sixth to 10th and, from 1 11th onwards. Then we aggregate the values averaging over all the tokens in the MRPC evaluation set. Finally, we normalize for each group so that the sum of the contribution values of each group is one. In this way, we can observe in which layer the contribution of given group of neighbours is the largest.
Our results on context identifiability from Section 5.3 complement some of the studies in previous literature. In (Jawahar et al. 2019) the authors observe that transformers learn local syntactic tasks in the first layers and long range semantic tasks in the last layers. We explain this behavior from the point of view of context aggregation by showing that distant context acquires more importance in the last layers (semantic tasks) while the first layers aggregate local context (syntactic tasks). Furthermore, the results showing that the context aggregation is mainly local, specially in the first layers, provide explanation for the increase in performance observed in (Yang et al.. al.] (018)) In that work, the authors enforce a locality constraint in the first layers of transformers, which pushes the model towards the local operators that it naturally tends to learn, as we show in Figure improving in this way the overall performance.
C.3 CONTEXT CONTRIBUTION TO CLS TOKEN
In this section [CLS] token, which is added to the beginning of the input sequence by the BERT pre-processing pipeline. This is an especially interesting token to look at because the decision of BERT for classification task is based on the output in the [CLS] token. Furthermore, like the [SEP] token, it does not correspond to a natural language word and its position the input sequence does not have any meaning. Therefore, the conclusion that context is on average predominantly local (cf. Section 5.3) is likely to not hold for [CLS].
Hidden Token Attribution to look at the contribution of the context to the
The second and final pre-training task that BERT is trained on is next sentence prediction. During this task, BERT receives two sentences separated by [SEP] token as input, and then has to predict whether the second sentence follows the first sentence or not. Therefore. it is expected that the context aggregated into the [CLS] token comes mainly from the tokens around the first [SEP] token, which marks the border between the first and second sentence in the input sequence. In Figure T8 we show the contribution to the [CLS] token from all of its neighbours averaged over all the examples in the MRPC evaluation set for the first, middle and last layers. In Figure [8a] the [CLS] token is placed at position and we see how the context contribution comes mainly from the tokens around position 30, which is roughly the middle of the input examples. In Figure 18b center the contribution around the first [SEP] token and indeed, it becomes clear that the (CLS) token is aggregating most of its context from the tokens around [SEP], i.e., from the junction between both
24

---

Published as a conference paper at ICLR 2020
sentences. In particular, the two tokens with the highest contribution the tokens directly before and after [SEP]. Also, it seems that the second sentence contributes more to [CLS] than the first one.
These results give insight what information BERT uses to solve next sentence prediction and serves as an illustrative example how Hidden Token Attribution can be used to analyze transform-
Total contribution to [CLS]
( 12
Total contribution to [CLS]
12
I
0.5
0.5
20
40
60
-40
-20
[SEP]
20
40
Neighbour
Neighbour
(b)
Figure 18: Normalized total contribution to the [CLS] token (a) centered around [CLS] at position
0 (b) centered around [SEP].
C.4 TRACKING CONTEXT CONTRIBUTION
Here we show examples of how Hidden Token Attribution can track how context is aggregated for given word at each layer. For reasons of space we show only few words of randomly picked sentence of the MRPC evaluation set, which is tokenized as follows:
[CLS] he said the foods #ter ##vic #ie pie business doesn fit the company
long term growth strategy [SEP] the foods #er #vvic ##e pie business does not fit our long term growth strategy [SEP]
[CLS]
0.07
3
0.06
0.05
Layer
0.03
0.02
11
0.01
[CLS]
he
said
the
##er ##e
business doesn
fit Hayyy
the
company
long
same Saaee
garo Straleg
ISEP!
the
foods
##er
##vic
##
gle.
business does
not
fit
our
Bong
tem Cravee
[e3s]
Figure 19: [CLS]: Aggregates context from all tokens but strongly from those around the first
[SEP] token. We hypothesize that this is due to the Next Sentence Prediction pre-training.
25

---

Published as a conference paper at ICLR 2020
he
0.10
0.08
Layer
0.06
0.04
0.02
11-
[CLS]
he
said
the
## ##vic
## pie
businesss doesn
fit
the ny
company
long som Saooe sereene
TStP!
the
foods
##er #ævic
pie ##0
business does not fit
ear bon
groee staeeo
[stp]
0.00
Figure 20: he: Aggregates most context from the main verb of the sentence, "said".
said
0.08
0.07
0.06
Layer
0.04
0.02
0.00
[CLS] he- said
the
##er #øvic ##e
pie
business doesn
fit the Cerpprie
s
Lamaji
(aas)
the soods #aet ## #as
ft
Guo)
1rmos6
[dap]
compon
Figure 21: said: Aggregates context mainly from its neighborhood. the main verb of the subordinate sentence and the border between the two input sentences.
fit
0.08
0.07
Layer
0.05
0.03
9-
11
0.01
the Sauaee
0.00
[CLS]
he
said
the
foods
##er
#*vic ##
pie
business doesn
ftt
Company
the
foods
##err
#vv
#e
pie
musinero does mon
fit our
Gong
[d3P]
Figure 22: fit: In the first layers it aggregates most context from neighborhood and towards the
last layers it gets the context from its direct object (strategy) and from the token with the same
meaning the second sentence.
26

---

Published as a conference paper at ICLR 2020
long
0.10
0.08
Layer
0.06
0.04
0.02
[CLS] he said
the foods
##er
pie #*
business doesn
tem
ABayens
[das]
toods
Pr
ssauisna
soep
sou
Köavens unMouu tem
[dass]
Figure 23: long: It is part of composed adjective (long-term) and aggregates most of its context from the other part of the adjective (term) as well as from the same tokens the second sentence. Interestingly, it mostly ignores the hyphen.
strategy
0.10
0.08
Layer
0.04
0.02
11
[CLS]
he
said the
foods
##er
#
pie
business mesinesss doesn
fit the Sanny
Company
long Jeme paro
the ISEP
foods
##er
##vic #se
pie. business
does not fit our long
term cooon Lesven
[d3s]]
Figure 24: strategy: Aggregates context from the word growth, which is the first one of the noun
phrase "growth strategy".
[SEP]
0.16
0.14
0.12
Layer
0.10
0.06
0.04
11
0.00
[CLS]
he
said
the
##er v#vvc
## pie
business doesn
itt the Seaaeeee
ABaseuns
(eass
"he ser
foods
##vic
#
pie
pess
d0eg
Yoo
Gong
[d3s]
aussing
Figure 25: [SEP]: This token that has no semantic meaning aggregates context mostly from [CLS]
and its own neighborhood.
27

---

Published as a conference paper at ICLR 2020
C.5 TOKEN CONTRIBUTIONS BY POS TAG
Here we show the contribution of input tokens to hidden representations in all layers split by part-of- speech (POS) tag (Toutanova al. 2003). The POS tags ordered according the contribution in layer 12.
0.40
Contributio
0.35
0.25
NIVS
VON
CD
ViG
RB6
m.
113
nB
ve
VBD
nbs.
Vgp
MD
PDT
WRB
V82
PRP
wh.
cc
WOT
PRPS
EX
SYM
DT
TO
SEP
Figure 26: Layer Most token types are equally mixed and have already less than 35% median
contribution from their corresponding input. The only exception are the [CLS] tokens, which remain with over 40% median original contribution.
0.325
0.300
Contaibttion
0.255
0.250
0.225
0.200
0.175
NN5
11-
VBN
V9G
RBR
PW
VB
veD
VBP
LS
MD
POT
wns
vBz
PRP
CLS
WP.
m
oc
PAPS
EX
STM
DT.
POS
SEP
Figure 27: Layer 2: Similar to the previous layer with less contribution over all and [SEP] behaving
similarly to [CLS].
0.325
Contribution
0.255
0.250
0.255
0,200
0.175
AINS
VBN
VBG
SBR
Rv-
ve
veo
nBs
vBP
is
NO.
PDT.
wns.
v9z.
cus
wr-
IN
cc
woT-
PRPs
Ex
SYM
or-
POs
SEP
Figure 28: Layer 3: Similar to layer 2 with decreasing contribution overall.
28

---

Published as a conference paper at ICLR 2020
0.300
0.275
0.250
Contribution
0.225
0.200
6.175
0.150
0.125
MWS
VBN
VBG
FV
RB
v9
VBD
VBP
L5
ND.
WRB
vBz
cLs
cc
wot
MRP
8X
SW
DT
TO
SEP
Figure 29: Layer 4: The original input contribution [CLS] and [SEP] falls significantly. The trend
that the word types will follow until the last layer is already clear: Most nouns (NNP, NNS, NN), verbs (VBN, VB. VBD, VBP), adjectives (JJ, JJS) and adverbs (RBR, RBS) keep more contribution from their corresponding input embeddings than words with "less" semantic meaning like Wh- pronouns and determiners (WP, WDT)), prepositons (IN), coordinating conjunctions (CC), symbols (SYM), possessives (PRPS, POS) or determiners (DT).
0.30
Contribution
0.15
0.05
VeN
vBG
JR
RBR
PV-
Na.
v8
VB0
nos-
VBP
nP-
LS
MD
NOT
WRg
V82
PRP
wp.
I
cc
WOT
ex.
GT.
TO
SEP
Figure 30: Layer 5: The trend started in the previous layer continues, with a reduction of internal
variability within those word types with less original contribution.
button
0.20
0.15
0.05
MWS
VeN
VBG
RBR
rw-
8a-
VBD
nAs-
VBP
mP-
Ls
MD
nor-
weB
vaz
PRP
C.s.
wr-
cc
woT
EX
STM
OT.
POS
SEP
Figure 31: Layer 6: Similar behavior as in the previous layer with minor evolution.
29

---

Published as a conference paper at ICLR 2020
Contribution
0.05
NAS
VON
VBG
RB8
no.
VBD
n8s
VBP
ND
PCT.
V82
PRP
cc
WITT
EX
SYM
DT
SEP
Figure 32: Layer 7: Minor changes with respect to Layer 6.
0.25
Contribution
0.20
6.10
0.05
VIN
CD
VIGG
ABR
PM
ns.
VBD
nes
MD
v82
PRP
a.s.
wP
cc
WaT
EX
SPM
DT
SEP
Figure 33: Layer 8: At this point there is clearly different behavior between the tokens with most contribution which present more intra-class variability, and those with less contribution, which are more uniform.
0.25
Contribution
0.15
0.05
VBG
RBR.
PW
nB
18
V80
RB5-
RDT
WRB
V82
PRP
WP
cc.
wDr
EX
SYM.
DT
TO
SEP
Figure 34: Layer 9: SEP changes increasing the contribution, while the rest stays similar.
30

---

Published as a conference paper at ICLR 2020
Contribution
0.15
0.10
NAX
VON
VBG
RBR
FW
us
no
vB
VBD
nes.
RP
nDT
WRB
VB2
WP
cc
wor
EX
DT
TO
SEP
Figure 35: Layer 10: The contribution evolves with same pattern as in previous layers.
0.25
Contribution
0.15
0.00
1005
U
VIN
VIG
RBS
ns
VBD
nes.
ND
PDT
VAZ
PRP
GLS
WP-
cc
WITT
SPM
DT
TO
SEP
Figure 36: Layer 11:The contribution evolves with the same pattern as in previous layers.
0.20
Contribuction
0.15
0.00
RNS
VON
VBG
RBR
PM
n6
18
VBD
RP-
MD
POT
WRB
VB2
wp.
oc.
woT
PAPS
EX
5mM
DT
10
SEP
Figure 37: Layer 12: Finally, nouns, verbs, adjectives, adverbs, receive more contribution from their
corresponding input than determiners, prepositions, pronouns, "to" words and symbols.
31

---

Published as a conference paper at ICLR 2020
D GENERALIZATION TO OTHER DATASETS
In this appendix we reproduce several experiments from the main text using the development sets of two additional datasets from the GLUE benchmark: The Corpus of Linguistic Acceptability (CoLA) (Warstadt et al., 2018), and the matched Multi-Genre Natural Language Inference corpus (MNLI-matched) (Williams et al. 2018). CoLA is dataset about grammatical acceptability of sentences and MNLI consists of pairs of sentences where the second sentence entails, contradicts or is neutral about the first one. These datasets differ significantly from MRPC. The development set of CoLa has 1043 examples with sequence length d, between 5 and 35 tokens, and 11 tokens on average. The development set of MNIJ-m consits of 9815 examples although we restrict the experiments the first 4000 examples without loss of generality. These contain total of 155964 tokens, with a sequence length comprised between and 128 tokens and an average per example.
tokens
The results presented in this appendix are qualitatively similar to those presented in the main text, which shows that empirical conclusions about BERT general across data domains.
D.1 TOKEN IDENTIFABILLTY
Here we reproduce the main token identifiability results of Section 4 on two additional datsets: CoLA and MNLI. Qualitatively, the results are line with those for MRPC. Note that random classifier would achieve accuracy of 1/d,, where d, denotes the average sentence length. Thus, the random guessing baselines for MRPC, CoLAA and MNLJ are 1.7%, 9% and 2.6% respectively.
D.1.1 CoLA
Figure
38
shows the token identifiability results for CoLA.
Identifiability Rate
0.8
0.6
MLP
Jin
0.4
Icos,
Icos,1
MLP
912,1
0.2
f122.
annive
912
2
3
4
5
9
10
11
12
Layer
Figure 38: Identifiability of contextual word embeddings different layers CoLAA.
32

---

Published as a conference paper at ICLR 2020
D.1.2
MNLI
Figure 39
shows the token identifiability results for the first 500 sentences (19,839 tokens) of MNLI- matched.
Identifiability Rate
0.8
0.6
MLP
Jin
0.4
Icos!
912,1
MLP
0.2
9L2,1
1
2
3
10
11
12
Layer
Figure 39: Identifiability of contextual word embeddings at different layers on the first 500 sen-
tences MNLI-matched (19,839 tokens).
33

---

Published as a conference paper at ICLR 2020
D.2 ATTRIBUTION ANALYSIS
D.2.1 COLA EXPERIMENTS
shows the token mixing analysis for the CoLA dataset. The behavior is very similar to MRPC with the only difference that both, the contribution of the original token and the percentage of tokens that are not maximum contributors to their embeddings are slightly larger across layers. However, this increase explained by CoLA consisting much shorter sequences on average than MRPC
Figure D.2.1
50
Contribution [%)
40
40
30
[%]
30
20
20
10
10
0
2
3
4
9 10 12
2
7 8 10
Layer
Layer
(a)
(b)
Figure 40: (a) Contribution the input token to the embedding at the same position. (b) Percentage of tokens that are not the main contributors to their corresponding contextual embedding at each layer.
Figure D presents the context aggregation for the CoLA development set. We observe the same general trend for MRPC, with the context being aggregated mostly locally and long range depen- dencies increasing in the later layers. The fact that examples in CoLA have an average sequence length of 11 tokens explains the smaller relative contribution of tokens beyond the 10th neighbour.
Rel. Contribution (%)
12 6
Total Contribution
8
0.5
6
3nd
4th and 5th
0
3
4
5
9 10 12
-40
-20
0
20
40
Layer
Neighbour
(a)
(b)
Figure 41: (a) Relative contribution per layer of neighbours at different positions. (b) Total contri- bution per neighbour for the first, middle and last layers.
34

---

Published as a conference paper at ICLR 2020
D.2.2 MNLI EXPERIMENTS
As shown by Figures D.2.2 and D.2.2 the results with the MNLI matched dataset are very similar
to the ones presented in the main text. No meaningful discrepancy exists in this case.
40
40
Contribution [%]
30
30
p 1%
20
20
10
10
0
2
3
4
5
6
8
9
10
11
12
1
2
3
5
6
7
8
9
10
1001100
12
Layer
Layer
(a)
(b)
Figure 42: (a) Contribution of the input token to the embedding at the same position. (b) Percentage of tokens P that are not the main contributors to their corresponding contextual embedding at each layer.
Rel. Contribution (%)
10
12
Total Contribution
6
8
0.5
1st
2nd
6
6th 1OCh
11th oowards
0
2 3
4
5
6
8
9 10 12
-60
-40
-20
0
20
40
60
Layer
Neighbour
(a)
(b)
Figure 43: (a) Relative contribution per layer of neighbours at different positions. (b) Total contri- bution per neighbour for the first, middle and last layers.
35


Reviews:
[{"review_id": "rygqy5YTtB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work studies the identifiability of attention, i.e., to what extent does the attention weight uniquely determine the output. The paper (1) formally establishes, using a rank argument, that at least for the cases where the attention head dimensions are smaller than the sources, infinite different attention weights can yield the same attention output. Based on such an observation, (2) a principled tool to extract the `effective` attention is introduced, with which several previous observations are challenged. This tool can certainly inspire future research. The paper further (3) explores how much token and context information is mixed at different layers of a transformer model, using both a carefully designed probing task and a gradient-based method.\n\nOverall I like the paper a lot. Its hypothesis and arguments are very clearly presented, which are then fleshed out by carefully designed experiments. I vote for an acceptance.\n\nI don't have any major complaint. Below are some questions and comments.\n\n- Since the paper only experimented with BERT, I think the authors mean \"BERT\" instead of the transformer models in general, whenever an empirical argument is drawn.\n\n- Section 4.1, can the authors justify the use of MRPC for this analysis? Adding onto this, do the authors think the conclusion here holds for other input text domain? What about other transformer models, e.g., GPT, or a transformer trained using MT objective?\n\n- Section 4.2, 2nd paragraph. I'm not sure why that g^{MLP} is better than g^{lin} can be interpreted as \"non-linearities play an important role in maintaining token identifiability.\" Could it be the case that the non-linearities make the token information more opaque, such that it can only be extracted with a more expressive model (MLP)? A better way to support this claim, in my opinion, is to show a better token identifiability in a BERT model trained without non-linearities.\n\n- In section 4.2, many conclusions are drawn that the last layer behaves differently than the rest. To eliminate the potential effect of depth, have the authors considered conducting a controlling experiment, where the probing task is to recover e_{i}^{l-1} (i.e., the representation from layer l-1, instead of the input token embedding) from e_{i}^{l}?\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "rkxDvlu0tr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "CONTRIBUTIONS:\nTopic: Analysis of self-attention in the standard Transformer, applied to models of language. \nNotation for review (see also definitions in C1-C4): cell(t,L) is the cell in layer L at input-string-position t (“corresponding to input token t”); Z(t,L) is the output vector for cell(t,L). Also, in the review, points are sometimes labeled by “(label)”, which is intended to label the point that immediately *follows* it in the text.\nC1. “Attention identifiability.” Uniquely identifying the attention-weight vector from the output of attention is not possible when D < T: For an individual head, the vector R returned by attention uniquely determines the attention-weight vector A only if the dimension D of the value vector equals or exceeds the length T of the input. \nC2. “Effective attention vector” A1: When D < T, there is a method for uniquely decomposing A as A = A1 + A2 where A2 has no effect on R. \nC3. “Token identifiability” method M. Identifying t from Z(t,L): A method M is proposed for this, and experimental results presented showing that, roughly, M succeeds at lower levels L but can fail at higher levels.\nC4. “Hidden token attribution” method: A gradient-attribution-based method is proposed to quantify C(t,t’), the proportional contribution of input token t’ to Z(t,L). C(t,t) is the “[same-]token contribution”; the “contextual contributions” are the C(t,t’) for t’ =! t. Experimental results are presented showing that after L = 1, 6, 12 the median token values are about 30%, 15%, 10%.\nMAIN ARGUMENT: Interpretation of the weight of cell(t,L)’s attention to cell(t’,L) as a measure of the influence of token t’ on the level-L representation of token t is (P0) frequent, but problematic, because (P1) the attention weight itself includes an irrelevant quantity (the A2 contribution, which should be omitted to get the effective attention [C2]) and because (P2) Z(t,L) is not properly viewed as representing token t, since (P2) it can be insufficient to even determine which token t it corresponds to [C3] as (P3) it can contain strong contributions from other tokens [C4]. \nRATING: Weak accept\nREASONS FOR RATING (SUMMARY). The paper raises important challenges to a standard practice for interpreting the knowledge in Transformer models (especially BERT) by examining attention weights. It proposes methods for reaching more theoretically justified interpretations. It introduces important new general interpretive concepts in the process. The main argument is sufficiently well supported by theoretical and empirical results to justify making the ICLR community aware of it, although there is a missing step in the argument: justification that the proposed means of resolving the unidentifiability of attention yields an optimal result.\nREVIEW\nWeaknesses.\nMinor complaint concerning exposition: The central result, C1 above, is actually rather obvious, unless I’m missing a subtlety (in which case it would be useful for that subtlety to be pointed out explicitly in the paper). (*) The dimension-D vector R is a linear combination of T vectors; if T > D, these T vectors cannot be linearly independent, so the weighting coefficients that yield R cannot be unique. The argument presented is rather overblown and may hide the obvious correctness of the result. However, I think over-deriving is clearly preferable to under-deriving, so this complaint truly is extremely minor. It might be worthwhile to add a sentence like (*) however to make sure that simple observation doesn’t get lost in the formalism. A further advantage of the more detailed derivation is that it sets up nicely the unique decomposition of A into the component A2 in the null space and the component A1 orthogonal to the null space (of the matrix containing the T vectors being linearly combined). \nA more substantial question here: What argument or evidence is there that this particular decomposition A = A1 + A2  is the one that yields a definition of “effective attention” that is best for interpretative purposes? Is there a metric of success that would validate this? Given that A is underdetermined, there is a whole subspace of attention-weight vectors that yield the same result R of attention. The proposed choice of A1 within that space is a natural one, but what makes it the *right* one? The incontrovertible point is that the A vector a model happens to use has a degree of arbitrariness in it that makes using it for interpretation problematic. But what is the right criterion for picking an alternative “effective” version of A, A1? Being orthogonal to the null space is one well-defined choice, but is it clearly the right choice? (I believe this choice is also the one with minimal L2 norm. Is that important?) For example, Sec. 3.3 shows that raw attention, but not effective attention, gives evidence that attention shifts from [CLS] to [SEP] to periods and commas as L increases. This shows they are different, but what is the argument that the raw-attention conclusion is less ‘correct’ than the effective-attention conclusion here? \nMore generally, it would strengthen the paper even further to document more concrete cases where effective attention yields different interpretive conclusions than raw attention, especially to the extent that it can be argued that the former interpretations are somehow better than that latter.\nStrengths.\nThe paper gives convincing evidence of the MAIN ARGUMENT’s propositions (P0) – (P3).\n(P0) [Interpretation of the attention weight as a measure of the influence of one token on another] is frequent] Sec 4, 2nd paragraph, cites 16 papers to illustrate the point. While I have not examined all of them, even allowing for some possible disagreements of interpretation, I am confident there is ample relevant evidence among these papers and probably others as well.\n(P1) [the attention weight itself includes an irrelevant quantity [C2]] The argument provided for this in Sec 3 seems correct to me, but as stated above as a ‘minor complaint concerning exposition’, it may not make as clear as possible the obviousness of the conclusion. \n(P2) [Z(t,L) can be insufficient to determine which token t it corresponds to [C3]] This point is operationalized in Sec 4.2 by picking a function-approximation architecture G, and attempting to use it to perform the map Z(t,L) -> t, for a given L and for all input strings and tokens t. A linear and a non-linear MLP G are used. An experiment applying the pre-trained BERT-base Transformer to the Microsoft Research Paraphrase Corpus gives success rates that fall to 73% and 82% by final layer 12. That 18% of tokens cannot be recovered from their corresponding layer-12 encoding is the basis of the claim that equating attention from cell(t,L) to cell(t’,L) with attention from token t to token t’ is not justified in general. \nThat the non-linear approximator does 9% better than the linear one is the basis of the claim that ‘the non-linearities in BERT play an important role in maintaining token identifiability’ (Sec. 4.2). This claim should be spelled out explicitly. It reads to me to say that if the non-linearities in BERT were removed, token identifiability would be reduced, but the experiment does not show that. Rather, it shows that, in the presence of the non-linearities in BERT, a linear approximation to the token-identifying map G is significantly less accurate than the non-linear approximation: hardly a surprising conclusion. This bit of the argument needs to be clarified.\n(P3) [Z(t,L) can contain strong contributions from tokens t’ =! t [C4]] As defined in Sec. 5, the contribution of token t’ to Z(t,L), for a given L, is proportional to the norm of the gradient of Z(t,L) with respect to the input from token t’; these are normalized to sum to 1. This is a transparent definition that resembles the previous gradient attribution proposal, but applied to internal rather than output representations. Using the same experimental set-up, it is shown that the average contribution of token t to Z(t,L) declines from 30% to 10% as L goes from 1 to 12. The proportion of values t such that the largest contribution to Z(t,L) is from a token t’ =! T rises from 0 for L = 1 through 3 to over 30% for L = 10 through 12. This is further evidence for the claim that equating attention between Z(t,L) and Z(t’,L) with attention between tokens t and t’ is problematic.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "SyejH3d6FH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This very interesting paper proposes new perspectives for investigating the self-attention distribution and contextual embedding and challenges many existing findings in the literature. The conclusion presented in this paper partially aligned with some concurrent works (Danish Pruthi et al. 2019, Serrano et al. 2019 and etc.)\nMany concrete findings in the paper make sense very much (not surprising), like 1) Input tokens retain their identity in the first a few hidden layers and then progressively become less identifiable, 2) Non-linear activations are crucial in preserving token identity and 3) Strong mixing of input information in the generation of contextual embedding.\n\nThe neat proof in the paper, though is based on some assumptions, formally shows that the multi-head self-attention is not always identifiable. The conclusion is that when the sequence length is larger than the size of attention head, there is redundant parameterization space for the attention.\nThe authors propose “effective attention” which is orthogonal to the null space as a diagnostic tool. In Figure one, the fact that the Pearson correlation between raw and effective attention weight decreases as the sequence length increases experimentally support the theorem. \n\nHowever, I am not sure what benefit identifiable attention would bring us. It seems that \"redundancy\" sometimes have some \"benefits\". As shown in the original Transformer paper, very large d_v or very small d_v both lead to worse performance. I am not sure how “effective attention” would ease the training, though I agree it is a good \"diagnostic tool\".\n\nAnother concern is about how the “non-identifiability” really hurts the model? A pre-trained transformer-based model can achieve very good performance for discriminative tasks after fine-tuning. This shows that a deep model, though there are many uncertain choices in the earlier and middle layers, they finally give us correct discriminative labels.\nAs the authors claim that contextual embeddings are strong mixing of input (which I agree), my question is \"will the non-identifiable attention weights, dramatically affect the mixing component of the contextual embeddings\"? Or, will the uncertain choices in earlier layers finally come to close contextual representations in layers closer to output?\n\nTowards the claim that “non-linear activations in preserving token identity”, the authors provide evidence in Figure 2.\nCan we also do experiments that using the embedding e^{l}_j (Layer l and position j), through non-linear activations to recover neighboring tokens of position j? I suspect that neighboring tokens can probably also being covered very well by MLP g^{MLP}_l.\n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper investigates the identifiability of attention distributions in the context of Transformer architectures. The main result is that, if the sentence length is long enough, difference choices of attention weights may result in the same contextual embeddings (i.e. the attention weights are not identifiable). A notion of "effective attention" is proposed that projects out the null space from attention weights. 

In the discussion period, there were some doubts about the technical correctness of the identifiability result that were clarified by the authors. The attention matrix A results from a softmax transformation, therefore each of its rows is constrained to be in the probability simplex -- i.e. we have A >= 0 (elementwise) and A1 = 1. In the present version of the paper, when analyzing the null space of T (Eqs. 4 and 5) this constraint on A is not taken into account. In particular, in Eq. 5 the existence of a \tilde{A} in the null space of T is not clear at all, since for (A + \tilde{A})T = AT to hold we would need to require, besides A >= 0 and A1 = 1, that A + \tilde{A} >= 0 and A1 + \tilde{A}1 = 1, i.e.

\tilde{A} <= A (elementwise)
\tilde{A}1 = 0

The present version of the paper does not make it clear that the intersection of the null space of T with these two constraints is non-empty in general -- which would be necessary for attention not to be identifiable, one of the main points of the paper. 

The authors acknowledged this concern and provided a proof. I suggest the following simplified version of their proof:

We're looking for a vector \tilde{A} satisfying

(1) \tilde{A}’*T = 0 (to be in the null space of T)

and 

(2) \tilde{A}’*1 = 0 
(3) \tilde{A} >= -A

(to make sure A + \tilde{A} are in the probability simplex).

Conditions (1) and (2) are equivalent to require \tilde{A} to be in the null space of [T; 1]. It is fine to assume this null space exists for a general T (it will be a linear subspace of dimension ds - dv - 1). 

To take into account condition (3) here’s a simpler proof: since A is a probability vector coming from a softmax transformation (hence it is strictly > 0 elementwise), there is some epsilon > 0 such that any point in the ball centered on 0 with radius epsilon is >= -A.
 
Since the null space of [T; 1] contains 0, any point \tilde{A} in the intersection of this null space with the epsilon-ball above satisfies (1), (2), and (3). This should work for any ds - dv > 1  and as long as A is not a one-hot distribution (otherwise it collapses to a single point \tilde{A} = 0).

I am less convinced about the justification to use an “effective attention” which is not in the probability simplex, though (not even in the null space of [T; 1] but only null(T)). That part deserves more clarification in the paper. 

I recommend acceptance of this paper provided these clarifications are provided and the proof is included in the final version. 


Decision: Accept (Poster)