Absolute Position Embedding Learns Sinusoid-like Waves for Attention Based on Relative Position
Authors: Yuji Yamamoto
Takuya Matsuzaki
Venue: EMNLP 2023 Main (2023)
Keywords: position embedding
attention mechanism
Transformer
BERT
RoBERTa

Abstract: Attention weight is a clue to interpret how a Transformer-based model makes an inference. In some attention heads, the attention focuses on the neighbors of each token. This allows the output vector of each token to depend on the surrounding tokens and contributes to make the inference context-dependent. We analyze the mechanism behind the concentration of attention on nearby tokens. We show that the phenomenon emerges as follows: (1) learned position embedding has sinusoid-like components, (2) such components are transmitted to the query and the key in the self-attention, (3) the attention head shifts the phases of the sinusoid-like components so that the attention concentrates on nearby tokens at specific relative positions. In other words, a certain type of Transformer-based model acquires the sinusoidal positional encoding to some extent on its own through Masked Language Modeling.

Absolute Position Embedding Learns Sinusoid-like Waves for Attention Based Relative Position
Yuji Yamamoto and Takuya Matsuzaki Tokyo University of Science 1423531@ed. www.::eeed.uu...eeeem. jc. matuzakiērs. tus.ac. jp
Abstract
Position embedding thus the key to make in- ferences Transformer context-dependent While Transformer uses deterministic sinusoidal position embedding, BERT (Devlin et al., 2019) uses a learnable absolute position embedding. The latter learns the positional representation only through word cooccurrences. Clark et al. (2019) and Kovaleva al. (2019) investigated the attention weights of the self-attention in BERT and found that the attention in some heads largely deter- mined by relative positions (Figure 1). This im- plies that even when the absolute position embed- ding is used, self-attention can make an inference depending on relative position.
Attention weight a clue to interpret how a Transformer-ber model makes infer- ence. In some attention heads, the attention fo- the neighbors of each token. This al- lows the oulput vector of each token to depend on the surrounding tokens and contributes to make the inference context-dependent. We analyze the mechanism behind the concentra- tion attention on nearby tokens. We show that the phenomenon emerges follows: (1) learned position embedding has sinusoid-like components, (2) such components are trans- mitted to the query and the key in the self- attention. (3) the attention head shifts the phases of the sinusoid-like components that the attention concentrates on nearby tokens at specific relative positions. other words, certain type of Transformer-basee model ac- quires the sinusoidal positional encoding to some extent on its own through Masked Lan- guage Modeling.
Wang et al. (2021) compared various position embedding methods based on the performance in downstream tasks. Their results showed that lo- cal translation invariance and asymmetric position bias with respect to the direction improved the performance. Ravishankar and Søgaard (2021) observed that some columns of absolute position embedding were periodic. Chang et al. (2022) showed that position representation was periodic even in hidden representations. However, it is not clear how the periodicity is used in the model.
1 Introduction
The architecture of Transformer (Vaswani et al., 2017) is symmetric with respect to the token po- sition and it captures word order only through the position embedding included in the input. Thanks this design, Transformer lationships between tokens while allowing paral- lelization. To derive representation of context, previous language models have used, for instance, recurrent units to process tokens the given token order, or convolution to aggregate tokens within a given range. In these models, the property of translation invariance of language has been cap- tured through the architecture of the models. In contrast, Transformer receives all tokens the same time and does restrict the positions of the tokens on which each token depends; in exchange for it, the translation invariance has to be learned rather than imposed by the architecture.
In this work, analyze how attention de- pends on relative position. As a result, we show that the relative positional dependence of attention emerges due to the following factors.
flexibly learn
The learnable absolute position embedding has sinusoid-like waves with several limited frequencies ($4.1.1).
Attention heads extract periodic components derived from position embedding from the hidden states. It is made explicit by applying Singular Value Decomposition to the parame- ters of the pre- pre-trained language model ($4.2).
The self-attention shifts the phase of the peri- odic components in the query and the key to decide the direction of attention ($4.3, $4.4).

---

Thus, it becomes partially clear how the self- attention equipped with learnable absolute posi- tion embeddings makes inferences based on con- text. However, is is suggested that, when the atten- tion is strongly concentrated on the adjacent to- kens, the word embedding is also a factor that en- ables inference based on relative position ($5).
BERT's position embedding is learnable pa- rameter. It is called the Absolute Position Embed-
ding (APE) because each row represents an abso- lute position in the input sequence.
RoBERTa (Liu et al., 2019) is identical to BERT in architecture. RoBERTa is pre-trained with 512 tokens for all steps whereas BERT is pre-trained with 128 tokens for 90% of the steps. Hence, RoBERTa's APE recieves the same number of up- dates regardless of the positions. Wang and Chen (2020) showed that RoBERTa's APE is more or- derly than BERT's. We thus use RoBERTa for all experiments to analyze the position dependence of inference.
2 Background
In this section, we review the multi- head self- attention mechanism and position embedding.
2.1
Multi-Head Self-Attention
The self-attention mixes the representations of to- kens different positions. The input to the 1-th attention layer is matrix X₁ Ixxd whose k- th row corresponds the hidden representation of the k-th token as a vector in IRd, The output of the self-attention layer is defined as follows:
3 Relative Position Dependence of Attention
In this section, we demonstrate that the attention depends on the relative position. Specifically, we analyze in how many heads the attention is de- (1) pendent the relative position, and the variation in the direction and concentration of the attention across the heads. The visualizations of the atten- tion weight Aur in Figure shows that, in (a) and (d), each token strongly attends only to the adja- cent token, while in (b) and (e), the attention is put on a few nearby tokens on the left or right side. We show that such pattern is largely invariant with respect to the inputs, by clustering the patterns of the attention different inputs.
XWW-O (X
Ayh softmax
th
Vd/n
Oi An 1h
(2)
MultiHead, =
(3)
where WOO th W W% e (d/(d/)) and WP E Raxd are the parameters, is the number of atten- tion heads per layer, and the subscripts / and h are the indices of the layer and the head, respectively. In this paper, we refer to the h-th head in the 1-th layer as "lead(1, h)" for short, and omit the sub- scripts I, h when the discussions are common to all layers and heads. The matrices XWQ, XW and A are called query, key, and attention weight, respectively. If the (i, j) clement of A is large, it is interpreted that the i-th token in the sentence attends the j-th token.
To focus the relation between the attention and the relative positions, we summarized the at- tention weight for each input in each head into a vector and applied k-means clustering to them. For that, we defined (-offset trace tr, as follows:
(A) (+2 0)
tre(1) =
(7)
(T) (t<
2.2 Position Embedding
Transformer's position embedding is defined as follows (Sinusoidal Position Embedding: SPE):
and transformed Ash to a vector:
SPE (2)),2)) = sin(pos//00000///g
(4)
am = [tr 10(AA), ..... trio(An:)] E R²
(8)
SPE 25+) cos(pos/100000//4)
(5)
These vectors represent the trend of attention Ayh with respect to relative position.
Vaswani al. (2017) hypothesized that it allows the model to easily learn attend by relative po- sitions, since the offset between two tokens can be expressed as a matrix product:
We input 100 sentences of length 512 cre- ated from wikitext-2 (Merity et al., 2017) into RoBERTa and computed vectors agh for every head. Figure 2 shows the results of applying the k-means to total of 100 12 x 12 vectors an when the number of clusters was set to 6. the clusters named leftward and next-to-left
coss 0 sin 0 sin 0 cos 0
[sin 24 cos
[ssin(ay + 0)
cos(a; + 0)]-
(6)

---

150
15.0
15
head
0.1
5
101112
3
(a) A
(b) 1
(c) 412.3
4
5
layer
7
8
0.0
9
(d) Ag,
(e) A7,3
(f) AL,
10
11
Figure I: Attention weights for the first 15 tokens of an input text.
12
next-to-left
leftward
itself
next-to-right
rightward
other
(resp. rightward and next-to-right), the at- tention is concentrated on the nearby tokens on the left (resp. right). Particularly, in the clus- ters named next-to-left and next-to-right, the attention strongly concentrated on the ad- jacent tokens. We found that, for each head, the vectors ath corresponding to the 100 inputs were grouped into one or two clusters. This means that there are some heads that always attended to the same direction for all the input.
For each head, a pie chart illustrates the per- centage of labels assigned to 100 vectors am (Eq. (7)).
Figure 2:
length T.
We computed the amplitude spectrum of each column of Epns by applying DFT:
spec; abss(DFT(EB))) Epos ei))
(9)
where E Fd is the i-th elementary unit vec- tor. The vector Ep is hence the i-th column of Fspos-
4 Attention to Nearby Tokens
Figure 3a shows the mean and the quartile range of the amplitude spectra of d 768 time series. The amplitude spectra peak at some frequencies, indicating that the learnable APE has acquired pe- riodicity through pre-training, even though is not explicitly defined using sinusoidal waves in Transformer. In contrast, there are no peaks in the amplitude spectrum of word embedding quence of sample text.' The periodicity is thus unique property of learned position embeddings.
In this section, we show that the attention depends relative position due periodic components in the position embeddings. First, we show that the learnable absolute position embeddings ac- quire several limited frequency components. Sec- ond, we show that some attention heads extract the periodicity derived from position embeddings. Fi- nally, we show that the concentration of attention on the nearby tokens is due to the periodicity.
We investigated whether similar properties are present in pre-trained models other than RoBERTa (Figure 3b and 13). The amplitude spectra the encoder-only models are similar to RoBERTa re- gardless of language, but ones of GPT-2 (Radford et al., 2019) are higher at lower frequencies.
4.1 Learned Representation of Positions
First, we show that APE includes sinusoid-like components using the Discrete Fourier Transform (DFT). Next, we show that the position embed- dings are confined relatively low-dimensional subspace (-15 dimensions) using Principal Com- ponent Analysis (PCA). Finally, show that the encoded positional information occupies simi- lar number of dimensions in the hidden states as in the position embedding, and the dimensionality becomes smaller in the higher layers using Canon- ical Correlation Analysis (CCA).
The decoder model probably can focus atten- tion the neighboring tokens of itself without periodicity-
mechanisms ($4.3). For exam- ple, attention is paid more strongly to the back- ward token, in BERT, attention is focused on the end of the sentence, but in a decoder with causal attention mask, attention is focused on itself. We leave it to future research to clarify whether this
4.1.1 APE Includes Sinusoid-like Waves We view the RoBERTa's position embedding Expos E Txx as a collection of d time series of
'The text taken from abstract and introduction of (Vaswani et al., 2017) was used as input to create the figures in this paper, unless otherwise stated.

---

5.0
leyer
amplitude
1.0
2.5
0.8
4
0.0
0.6
5
50 frequency (a) RoBERTa
100
6
0.4
0.2
5.0
10
amplitude
0.0
11
2.5
20
30
50 12
dimension
0.0
Figure 4: CCA coefficient for each layer compared to position embedding.
50
100
frequency (b) GPT-2
where 2x(z) is the response of the i-th neuron to input Zz.. We input 200 sentences of length 512 created from wikitext-2 into RoBERTa and col- lected the responses of each layer for the input of m 200 x 512 tokens. We then maximized their correlation coefficients Pk:
Figure Amplitude spectra of position embeddings. Blue line is the mean of spec, and blue area is the quar- úle range. Orange line is the spectrum word embed- dings for an input before adding position embeddings.
phenomenon really occurs in the causal model, and in this paper focus the case where self- attention symmetric with respect position.
Pk: max Corr( 7/' ak; 2 bg.by
(12)
where is chosen such that it is orthogonal to 01,-.. a:-1 and similarly for by. The CCA co- efficients in Figure 4 show that the hidden states of higher layers have lower correlations with the po- sition embeddings. This is in agreement with Lin et al. s (2019) result that BERT phases out posi- tional information as it passes through the layers. The CCA result indicates that the number components that are highly correlated with posi- tion embedding is only 5-20-dimensions, and the PCA result suggests that it is enough to accommo- date the positional representation. This indicates that the hidden states include positional represen- tation in low-dimensional subspace, similarly position embedding.
4.1.2 Dimensionality of Positional
Representation
We applied PCA to the position embeddings see how many dimensions were used to represent the positions. The cumulative principal component contribution rate of the position embedding was 50.51% % for the top 4 dimensions, and 92.23% for the top 12 dimensions (see Figure 12 in Appendix A for plot). This means that the positions are mostly encoded in low dimensional subspace.
We then employed CCA to show how much the input to the self-attention layers included the posi- tional representation. CCA a method for investi- gating the relation of two sets of variables by find- ing the parameters a and b that maximizes the cor- relation between two synthetic variables Xa and Yb given two inputs X and Y. Raghu et al. (2017) showed that CCA allows layer-to-layer compari- son of deep learning models. We used the repre- sentation of neurons and layers proposed by them and computed the correlation between the hidden layer and the position embeddings.
4.2 Positional Representation in Self-Attention
not at all clear how the positional representa- tion in the hidden states contribute to the inference in self-attention. We thus analyzed how attention weight is calculated, which is one of the most im- portant process in self-attention.
In this study, the i-th neuron 2 of layer I and the I-th layer Z are represented as follows:
4.2.1 Rethinking About Query and Key
The attention weight A is determined by the in- ner product of the rows of the query and the key (10) matrix. We thus expect that the relative position dependence of the attention can be explained by (11) analyzing the relationship between the query and
2 (2((x1) ..... . ((xm)
z! 2 2i...., 2/1

---

query
key
5.0
2.5
PE
2.5
0.0
0.0
150
2.5
query
2.5
125
|| 0.0
12
100
-2.5
0
20
40
60
key
token position
6-
75
Figure 5: Column of the query and the key in head(1, 1).
12
50
hidden
25
the key, We begin by rethinking about their defi- nition. The two parameter matrices Wº and W contribute the attention only through their prod- uet (W (Eq. (I)). Hence, they can be seen as a parametrization of rent-restritted x ma- trix, and we may consider another decomposition the product W WWWW")""
12
25
50
75
100
frequency
Figure 6: The frequency-wist maxima of the amplitude spectra. The [-th row of the heat map corresponds to max-spec; The top figure reiteration of Figure 3a.
We found that some sinusoid-like components were obtained from the query and the key by ap- plying Singular Value Decomposition (SVD) to (W(KK)" Specifically, the query and the key are redefined as the product of the hidden state and the singular vectors:
query. The relation between the query and the key is thus summarized in the matrix R. In addition, since the singular value matrix $ is diagonal, the product of the query and the key can be written as follows:
= -USSUUK
=
(13)
din
Q XUX, K -
(14)
X =QQT" Σ s.q.k (16)
where the matrix S E Iedxd is diagonal matrix diag(s1....,$///... .,.... 0) and each element s; is singular value. In the rest of this paper, refer and K defined above as the query and the key, respectively. As shown in Figure 5, the head in which the attention depends on relative position, sinusoid-like waves appear several columns of the redefined query Q and the key K. Moreover, a sine wave is paired with cosine wave of the same frequency, the 5-th and the 6-th column for hcad(1, 1) shown in Figure 5.
where qi and k; are the i-th columns Q and K, respectively. Eq. (16) says that the subsets of queries [q:] and keys [k:] corresponding to the top singular values [s:] are more important in the cal- culation of the attention distribution. We hereafter call the matrix QSKT atlention score (i.e., atten- tion before applying softmax).
4.2.2 Spectral Analysis of Query and Key
We computed the amplitude spectrum of each col- umn of the query Qu in each head, using the same procedure for the position embeddings in $4.1.1:
Furthermore, the introduction of SVD provides new theoretical interpretation of self-attention. Let R be the orthogonal matrix R (UQ)TK. Then U can written as UK U due to the orthogonality of the singular vectors. Thus, the key K can be written as:
spec abs abs(DFF((Qne.) max-spec, (mex((spe))a))l ol
(17)
(18)
By taking the maximum of spеc among the heads and the columns of Qin- we check if there are high peaks in the spectra of the queries the I-th layer (similarly for the keys and hidden states).
(15) That is to say. the rows of the key are the result an orthogonal transformation the rows of the
K X RURR-QR

---

Figure 8 and 9 show that it is actually the case. Figure 9 presents the cross-correlaioons of the query qj and the key kj for js correspond- ing to the top singular values. We can see how the concentration of attention on nearby tokens is formed by the weighted superposition of the cross- correlations. head(1, 1), the maxima of the cross-correlations near t = -1 stack up, while the maxima away from / = -1 cancel with the (21) minima of other components. Since there are mul- tiple periodic components with different frequen- cies (Figure 3a), the attention is not concentrated Figure 6 show that the query and key spectra peak at the same frequency bands as the posi- tion embeddings, indicating that attention heads extract periodic representations derived from the position embeddings. Furthermore, some of the peaks disappear in the higher layers (downward in the Figure 6), indicating that the periodic com- ponents derived from the position embeddings are not dominant in the higher layers. This is consis- tent with the result that the correlation between the position embeddings and the hiddem states of each layer gradually decreases ($4.1.2). also agrees with the result shown by Lin et al. (2019) 9) that posi- tion information is discarded between the 3rd and 4th layers.
i=5
-1 20 15
10
-5
0
5
10
15
20
Figure 7: The cross-correlation xcorr; (4) between the queries and the keys in Figure 5.
100
head(1.1)
head/8,9)
2.5
40
-20
40
-5
0
10
Figure The sum of cross-covariances weighted by singular values (Eq. (23))
4.3 Attention Based on Relative Position is due to the Phase Shift
head(1,1)
head/8, 9)
As shown in Figure 5, there are phase shifts in the sinusoid-like components of the query and the key. this subsection, we focus on the phase shifts, and clarify that the direction and the width of the phase shifts determine where the attention is con- centrated. We measure the direction and the width of the phase shift by cross-covariance and cross- correlation, defined as follows:
40
-0
0
20
40
15
-5
0
10 15
t
t
Figure 9: The cross-correlations xcorr, (1) correspond- ing to the top 10 singular values.
where tr, is the t-offset trace defined in Eq. (7) and $1,..., Sd/n are the singular values of W WWWKT. According to Eq. (21-23), the sum of the attention scores at relative position is equal to the weighted sum of the inner product of qj and kj shifted by t. Therefore if the xtov,(t)s corre- sponding to the top singular values attain maximal values at the same f, the attention is likely to be concentrated around relative position
E qi Kitki 2 0)
(t)
(19)
TH = hii
(t 0)
(corr,(1)= (t) - By [x((xcov((t)
(20)
|q, |k
For example, Figure shows the cross-correlation between the queries and keys in Figure Both (co and xcorra(() attain maximal value at t - - and t - 3. It means that the phase shift between the queries and the keys are approx- imately 2.5 tokens.
can be explained theoretically that the direc- tion of phase shift coincides with the direction of attention. The cross-covariance is related to the product query and key (Eq. (16)):
T-t QQSK) 1iH
tn tQS(T
Σ
Σ d/n
(22) away from each token. In contrast, in head(8. 9), some cross-correlations have narrow peaks only 1, and makes the attention be very con- (23) centrated only on the adjacent tokens. However, is unlikely that the cause of the narrow peaks
d/n Σ syxcov,(1)

---

is the sinusoid-like waves, because their period is approximately 8 tokens or more."
Aff
A
A(f,0)
head(1,1)
100
4.4 Phase Shift Width is the Same even if Frequencies are Different
We saw in the previous subsection that the main factor of the concentration of attention is that the phases of multiple sinusoid-like waves are all shifted by the same number of tokens. In this sub- section, we explain based on the property of the cigenvalues and the cigenvectors of the matrix R that relates the query Q to the key K.
h2ad(2,3)
2
25
50
75
100
Figure 10: 2D histogram of the function g(f, 8). The section at = 0g is the amplitude spectrum of Qp:.
Let p; E Cd and X E C be an eigenvector and an eigenvalue of the matrix R, respectively. Since is an orthogonal matrix, its cigenvalue can be expressed as A = ed where the imaginary unit. From Eq. (15), the following relation be- tween the query and the key holds:
appear around straight line. Specifically, the ra- tio A 8) of frequency phase defined as fol- lows is nearly constant:
T 2n
A(S. 0)
(27)
Kp; - QRp; Qup: Qp: co0
(24)
Hence, the two conditions are in fact approxi- mately hold. Thus, by multiplying the matrix Rt, the sinusoid-like components in the key offset by certain number of tokens relative to the query.
Let two conditions be assumed: (1) for each .,. the time series Qp; is sinusoidal with single frequency f. and (2) the ratio of the argu- ment 0g of the eigenvalue >; the frequency Si is constant regardless of Then Eq. (24) implies that the phases of Qp; and Kp; differ by con- stant number of tokens for any eigenvector p::
Attention to the Adjacent Tokens
Figure shows that, in head(8, 9). there are not only sinusoid-like waves but also spikes the rel- ative position = -1. In this section, we dive deeper into the fact that, in some heads, the atten- tion is focused only on the adjacent token. We an- alyze how the attention changes when modify either the position embedding the word embed- ding component. The following are the results on the modified inputs and insights from them.
(Qp.): = (KPi)LA
(25)
where A (T/27) (i:/fi). Furthermore, Q: K++A follows from Eq. (25). We provide the proofs Appendix D.
We verify whether the two conditions hold by analyzing the periodicity of Qp; and the ratio of the frequency to the argument 6; of the eigenvalue
Baseline The input is the sum of word embedding and position embedding as usual. This result is shown in Figure and 9.
>a To do so, define the bivariate functions for frequency and argument as follows:
Only position embedding The word embeddings replaced with zero vectors. Figure I a shows that spikes of cross-correlation do not appear for this case. This suggests that posi- tion embedding not the only cause the spikes.
g(f. 0.) = abs(DFF(Qp:))
(26)
This function is shown in Figure 10 as a 2D his- togram. Figure 10 shows that the spectrum of Qp, has peaks in different frequency bands according 8,, It means that the component along each eigenvector p; is fairly band-limited, namely they are linusoid-like. Furthermore, the spectral peaks
Only word embedding The position embeddings are replaced with zero vectors. Figure 11b shows that most of the cross-correlations are flat for this case. It suggests that word em- bedding contributes less to make the atten- tion dependent on relative position. However,
FFigure 3a shows that the maximum frequency of the po- sition embeddings is around 60. The minimum period hence 512/60 (> 8) tokens.

---

it is interesting that the cross-covariance -1 (i.c., left neighbor) relatively large even without position information.
x10*
-10
-20
Shuffling word embeddings The order of word embeddings is shuffled. Figure 1lc shows that spikes appear the relative position = -1 even for this case. It suggests that the contribution of position embedding is signifi- cant in determining the direction of attention, since the attention is focused on the preced- ing word regardless of what it actually is.
10
15
(a) Only position embedding
2
15-10
0
5
10
15
15
10
5
10
15
As mentioned at the end of Section 4.3 and also suggested by Figure la, it is unlikely that po- sition embedding is the only cause of the strong concentration of attention on the adjacent to- kens. However, if the word embeddings identi- fied the adjacent tokens (i.e., if the narrow peak cross-correlation appeared due to an interaction the word embeddings of frequently occurring bigram), the attention would have been moved to non-neighboring positions by the shuffling of word embeddings, but this not the case. It is thus suggested that the concept of adjacency in RoBERTa is built upon both word meaning and positional information.
(b) Only word embedding
110
0
-5
(c) Shuffling word emheddings
The cross-correlations (left) and the weighted sum cross-covariances (right) in head(8, 9) when we modify either the position embedding or the word embedding component of the input.
Figure 11:
Discriminant Analysis, i..., by identifying the axes separating the different position representations. We showed that sinusoid-like components could be extracted from the hidden states by applying SVD to the pre-trained parameters the self- attention even though SVD does not have the ob- jective separating positional representations.
6 Remark on the Learning Process of Position Embeddings
The training data of the masked language model- ing is only word sequences without positional in- formation. It suggests that the relative position de- pendence of attention acquired by the combina- tion of two factors: (I) the linguistic property that related words tend to appear nearby due to gram- matical rules and collocations, and (2) the property that attention is focused on the word that is syn- tactically semantically related and hence gives clue to fill the masked token. In appendix E, demonstrate that the position representation can be acquired with a much smaller amount of training data than pre-training.
Vaswani al. (2017) stated that the sinusoidal position embedding allowed position offsets be represented by rotation transformation (Eq, (6)), and this could prompt learning of attention that de- pends on relative position. We showed that the query and the key in the self-attention included sinusoid- like components and the learned param- eters in the self-attention shift the phase of the query and the key relatively. This means that the mechanism hypothesized by Vaswani al. is in fact acquired through pre-training by masked lan- guage modeling. However, the frequencies ac- quired by the position embedding of RoBERTa are only in the specific frequency bands, whereas si- nusoidal position embedding has d/2 frequencies (Eq. (4)-(5)). RoBERTa thus seems to have ac- quired more economical positon embedding than sinusoidal position embedding.
7 Related Works
Ravishankar and Søgaard (2021) observed that columns of absolute position embedding were periodic and sinusoidal position embedding was more effective in multilingual models than other embedding methods. Chang (2022) showed that position representation was periodic even in hidden representations by using Linear
Recently, there are many variants position

---

embedding and Dufter et al. (2022) surveyed them exhaustively. In particular, Rotary Position Em- bedding (RoPE) (Su et al., 2022), a type of Rel- ative Position Embedding (RPE) (Shaw et al., 2018), relates to the property that the self attetnion acquires the rotation matrix through pre-training To acquire relative positional dependence, RoPE widens the angle between query and key propor- tionally the relative position, while pre-trained self-attention rotates the hidden states containing absolute positional bias by the same angle regard- less of position. In other words, APE and self- attention, which are distant components, must ac- quire frequency and angle of rotation, respectively, to satisfy the relation Eq. (27). If translation in- variance is essential to language understanding, the cost of discovering this relation is possible reason why APEs are inefficient to learn compared RPEs.
self-attention), the non-linearity of softmax func- tion should be taken into account.
References
Tyler Chang, Zhuowen Tu, and Benjamin Bergen. 2022. The geometry of multilingual language model representations. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pages 119-136, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019. What does BERT look at? an analysis of BERT's attention. In Pro- ceedings of the 2019 ACL Workshop BlackbexNLP: Analyzing and Interpreting Neural Networks for NLP, pages 276-286, Florence, Italy. Association for Computational Linguistics.
Jacob Devlin, Ming-Wel Wei Chang. Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies. Volume I (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Associ- ation for Computational Linguistics.
8 Conclusion
We analyzed the concentration of attention based on relative position in the self-attention using the learnable absolute position embedding. As a re- sult, we showed that is due to relative phase shifts of the periodic components in the query and the key derived from position embeddings. Our re- sults explain in part that absolute position embed- ding allows inference based on relative position.
Philipp Dufter, Martin Schmitt, and Hinrich Schüitze 2022. Position information in transformers: An overview. Computational Linguistics, 48(3):733- 763.
Kazuki Irie, Albert Zeyer, Ralf Schlüter, and Hermann Ney. 2019. Language Modeling with Deep Trans- formers. In Proc. Interspeech 2019, pages 3905- 3909.
9 Limitations
As mentioned in $4.1.1, the positional representa- tion of the GPT-2 differs significantly from ones of the encoder models. Thus, it is impossible to interpret the inferences of the decoder-only model in the same way as in this study.
Amirhossein
Kazemnejad, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Payel Das, and Siva Reddy. 2023. The impact of positional encoding length generalization transformers. arXiv preprint 2305 19466.
Olga Kovaleva, Alexey Romanov, Anna Rogers, and Anna Rumshisky. 2019. Revealing the dark secrets of BERT. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Process- ing and 9th International Joint Conference on Natural Language Processing (EMNLP--JCNNLP) pages 4365 4374, Hong Kong, China. Association for Computational Linguistics.
In the tasks of predicting sentence structure (e.g., dependency parsing), the relative order of two tokens is important regardless of the distance between them. However, we analyzed the depen- dence of the output of each token only on the nearby tokens. Thus, it is unclear whether position embedding provides relative position information that helps determine which of the distant tokens precedes the other.
Mike Lewis, Yinhan Liu, Naman Goyal, Mar- jan Ghazvininejad, Abdelrahman Mohamed, Omer Levy. Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising pruuence-to-sqqueeee pre- training for natural language generation, translation. and comprehension. In Proceedings of the 58th An- nual Meeting of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computational Linguistics.
We obtained an interpretable representation by decomposing the attention scores before apply- ing softmax function (Eq. (16)). When analyz- ing the contribution of the decomposed represen- tations to downstream components (e.g., Value in

---

Yongjie Lin. Yi Chem Tan, and Robert Frank. 2019. Open sesame: Getting inside BERT's linguistic knowledge. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpret- ing Neural Networks for NLP, pages 241-253, Flo- rence, Italy, Association for Computational Linguis- tics.
In Proceedings of the 2020 Conference on Em- pirical Methods Natural Language Processing (EMNLP), pages 6840-6849, Online. Association for Computational Linguisties.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized bert pretraining ap- proach. arXiv preprint 1907.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017. Pointer sentinel mixture models. In International Conference Learning Representations.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever, 2019. Language models unsupervised multitask learners.
Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. 2017. SVCCA: Singular vector canonical correlation analysis for deep learn- ing dynamics and interpretability. In Advances in Neural Information Processing Systems, volume Curran Associates, Inc.
Vinit Ravishankar and Anders Søgaard. 2021. The im- pact of positional encodings multilingual com- pression. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process- ing, pages 763-777, Online and Punta Cana, Do- minican Republic. Association for Computational Linguistics.
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018. Self-attention with relative position represen- tations. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 2 (Short Papers), pages 464-468, New Orleans, Louisiana. Association for Computa- tional Linguistics.
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. 2022. Roformer: En- hanced transformer with rotary position embedding.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkorcit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. 2017. Attention all you need. Advances Neural Information Pro- cessing Systems, volume 30. Curran Associates. Inc.
Benyou Wang, Lifeng Shang, Christina Lioma, Xin Jiang, Hao Yang, Qun Liu, and Jakob Grue Simon- sen. 2021. On position embeddings in BERT. In International Conference on Learning Representa- tions.
Yu-An Wang and Yun-Nung Chen. 2020. What do position embeddings learn? an empirical study of pre-trained language model positional encoding.

---

A Cumulative Principal Component
2
bert-base-uncased
Contribution Rate
0
1.00
2
0.75
ja- BERT
0.50
0.25
0
0.00
5
5
10
15
20
xlm-roberta-base
dimension
Cumulative principal component contribu- tion rate of RoBERTa position embedding.
Figure 12:
0
5
roberta-base
B Amplitude Spectra of Various Models
0
We computed the amplitude spectra of the position embeddings of bert-base-uncased, cl-roookk/eerrr-hes---ppeeeeeeed
2
gpt2
-masking, xlm-roberta-base and gpt2 in the same way as Figure 3a. In Figure 13, the encoder models BERT and RoBERTa both have peaks in their amplitude spectra, and RoBERTa has higher peaks. On the other hand, the GPT-2 de- coder model has only low-frequency components. This suggests that the representation of absolute position embedding is similar between encoder models, regardless of language, but differs signifi- cantly between encoder models and decoder mod- els. In fact, Iric et al. (2019) and Kazemnejad et al. (2023) showed that the position embeddings is un- necessary for the decoder models.
0 0
50
100
150
200
250
frequency
Figure 13: Amplitude spectra of position embedding of various models.
concentrating attention on the backward tokens, which are masked in the subsequent processing. Figure 15 shows the result of clustering the atten-
tion scores of GPT-2 using k-means. The number of clusters was set to and the inputs to k-means were t-offset traces (E). from f = -3 30. The heads that depend on position are found in the lower layers.
C Comparing Different Architectures
This paper analyzed an encoder-only model (RoBERTa). In this section, we apply our methods to other transformer-based architectures: decoder- only and encoder-decoder. The target pre-trained models GPT-2 (gpt2) for decoder-only and BART (facebook/bart-basee (Lewis al., 2020) for encoder-decoder, both of which use absolute position embedding and are available on hugging- face. To reduce the gap between architectures, when analyzing the decoders, we use the attention scores, which is the matrix before causal masking and softmax function applicd, instead of the attention weights.
Along $4.4, we investigate the relationship be- tween the frequency of the hidden state and the angle of the rotation matrix inherent the param- eters of attention head. Figure 16a differs from the case of RoBERTa (Figure 10) in that the peaks appear horizontally rather than holding constant ratio between frequency and angle. Furthermore, according Figure 13, the dominant components in position embedding of GPT-2 are those with fre- quencies below 10, i.e., with periods longer than 51.2 (= 512/10) tokens. These differences from RoBERTa imply that even when the attention is focused within a few tokens in a particular head, cannot be attributed to the mechanism described in $4.3.
The attention matrices of GPT-2 show two main patterns: one related position and the other not. Figure 14a shows that the attention head of GPT- 2 pays stronger attention to the closer tokens by
In BART, the tendency the positional depen- dence of attention differs between encoder and

---

decoder. In encoder, the patterns of attention weights (Figure 17) and their trend (Figure 18), such as direction and strength, are similar to those RoBERTa (Figure and 2). On the other hand, comparing Figure 14 and Figure 20, the patterns of decoder is different from those of GPT-2. Such patterns in the BART decoder could not be clus- tered based on the strength or direction of attention using k-means. Like in GPT-2, Figure/ 21 shows that the peaks appear horizontally in the decoder.
Figures for BART encoder
0.10
0.05
15
15
15
15
0
15
(a) A1.4
(b) A2.4
(c) As,1
0.10
0.05
Figures for GPT-2
15 0
15
15
0.00
15
20
(d) A1.0
(c) A4.o
(f) As.3
50-
Figure 17: Attention weights of BART encoder for the
150
150
9
first 15 tokens of input text.
200
200
-10
head
300
100
200
300
100
200
300
2
4
6
8 101112
1
head(1.4)
(b) 12,)(
2
layer
Figure 14: Attention scores of GPT-2 for the first 300
3
tokens of imput text.
4
5
6
head
2
3
6
8 9 101112
next-to-left
leftward
itself
next-to-right
rightward
other
3
Figure 18:
Trend of attention weights in BART
4
coder.
layer
6
7
0.005
21,.8
8
D1f.0
0.004
9
10
0.022
11
00001
217,0)
12
00
25
76
25
75
0.000
rightward
other
(a) (3,d(3,4)
(b) head(1,4)
Figure 15: Trend of attention scores in GPT-2.
Figure 19: 2D histogram of the function g(f.0) 0)
BART encoder.
A) -20
Figures for BART decoder
0.0
0.02
0.01
0
0
6
6
2
-10
(a) (1ad(1.4)
(b) head(12, 1)
200
(a) (2,5(2
(b) head(3.6)
(c) head(6.6)
Figure 16: 2D histogram of the function g(f.0) in GPT-2
Figure 20: Attention scores of BART decoder for the first 300 tokens of an input text.

---

1. Qp: is sinusoidal with a single frequency fü- offset of A tokens between the query and the key.
-11,01- 10
That is,
0.2
EP E Cilxd 8.1. |P| # and (QP); = (KP))+
(33)
0.00
5
VB Cdxn (QB), = (KB)++
(a) (2,5(
(b) hcad(6,6)
(34)
Figure 21: 2D histogram of the function 9(ʃ.0) in BART decoder.
where Q, E Rxd are the query and the key respectively and n E N is less than or equal to The subscript indicates the t-th row of the matrix.
Theorems About the Phase Shift Between the Query and the Key
Theorem The following two assumptions are given:
Proof. Let B be an arbitrary matrix. Then,
(QB) (QPP-1B)) B))
(35)
((P))
(36)
2. The ratio of the argument O, of an eigenvalue X; to the frequency fi is constant regardless of
k=1
Σ (KP) (KK))++A, (P)++A((PP B) (: Eq. (33))
this case, Eq. (25) derived from Eq. (24):
(37)
Kp. Qp; -0,
(24)
-KKPP-B))u B)
(38)
= (Qp.), = (Kp,)++
(25)
= -KBB)++A
(39)
Proof. Since Qp: is sinusoidal, the t-th element in polar form is as follows:
(Qp:). ftppj2mftt/
(28)
In particular, when B the equation Q: KI+A obtained.
where ry the absolute value of (Qpi)c- Then the 1-th element of Kp; can be written as follows from Eq. (24):
E
How the Relative Position Dependence of Attention Emerges
(Kpi); exp (etpppjj2fftt///m exp[j0,)]
(29)
How does a masked language model acquire the (30) concept of "neighborhood" even though absolute position embedding learns without the informa- tion about the order of the tokens? In this sec- tion, demonstrate the process of acquiring the ability foeus attention nearby tokens by re- learning the position embedding under the masked language modeling.
=r:expli exexppi 0,)]
( A.)/T]
(31)
where A; = (9/2m))(9./f1)) Now, since the ratio the argument 0; to the frequency fü constant, A; is independent of i, namely, A; can be replaced with a constant A. Thus, (QP:); (Kp.):ta holds from Eq. (28) and Eq. (31).
Suppose that the following sentence is input into pre-trained model:
Let the matrix following equation holds:
[pi, Pa). then the
"This <unk> is <unk> an <unk> <mask>
(QP); (KP)
(32)
Since this sentence collapsed with many UNK tokens, the model cannot fill the MASK correctly. If we re-learn the position embedding, will the model be able to fill
Theorem 2. When, in a given basis, the position of each token in the key is offset by A Z) tokens relative to the query, then in any basis, there is an

---

training, the models can predict them. The at- tention weights during inference are visualized in Figure Since the attentions are concentrated every fourth token, the model with relearned posi- tion embeddings recognizes the relative positions of the non-UNK tokens. This result suggests that the attention is not focused the tokens that co- occur frequently, but on the tokens that are infor- mative to fill the MASK token. Thus, the relative position dependence of attention is simply caused by the fact that related words appear nearby.
(a) As.9
(b) A10.1
(c) As.s
(d) A7.6
Figure 22:
Attention weight with relearned position
embedding
PE relearned
predicted top-S tokens
exercise essay experiment article adventure and the
In summary, this experiment indicates that the relative position dependence attention is caused by the combination of two factors: (1) the lin- guistic property that related words tend to appear nearby due to grammatical rules and collocations, and (2) the property that attention is focused words that are related in some sense.
RoBERTa base
to
Table I: Top 5 MASK candidates when 3 UNKs are put between each token in "This is an MASK."
E.1 Experiment
We additionally train RoBERTa with the following configurations.
Dataset We used 8645 samples consisting of more than 128 tokens from wikitext-2.
Putting UNK token Three UNK tokens were in- serted after each token in all sentences; the token sequence [t1, t2..., f12s] was expanded to 14, unk, unk, unk, l2, unk, unk, unk. 1128, unk, unk, 3nk].3
Freezing parameters We freezed parame- ters except position embeddings. This encourages the position embeddings to learn to focus attention only on position t 1,5, 9, 13, and prevents position- independent shortcut solutions (c.g., shorten- ing the UNK embedding ignore it).
Position embedding The position embeddings were initialized with random values follow- ing normal distribution N (0, 0.02?) before training.
E.2 Results
Table shows that, before the additional training, RoBERTa cannot fill MASK with the appropriate tokens (in this case, nouns beginning with a vowel) when UNKs are inserted, but after the additional
'Sinee Transformeris is symmetric with respect to position, this process is similar to padding. However, the PAD token of RoBERTa published on hugging face is zero-vector, which is inconvenient for analyzing interactions hetween word em- heddings. Thus, we choose UNK tokens.


Reviews:
[{"review_id": "n1Zz9dJ9U6", "reviewer": "Reviewer_fPAJ", "summary": "", "questions": "- A. What is the substantial contribution to the community from these findings? For example, could they lead to any ideas to improve the Transformer architecture?\n- B. Clark+'19 reported that most heads pay little attention to themselves and that there are heads that focus heavily on adjacent tokens, especially in the earlier layers. However, this paper shows that there are several heads that focus on themselves and that heads focusing heavily on adjacent token heads were found in various layers. What is the reason for this discrepancy?\n- C. Nowadays, relative position embedding is often used instead of APE. Can this paper provide insights into the reasons for the superiority of relative position embedding? For example, can we interpret that relative position embedding can capture other language information richer than APE because there is no need to extract relative positions in the query or key transformation matrices?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We first answer to question B and C and then answer to question A.\n\n> B. Clark+'19 reported that most heads pay little attention to themselves and that there are heads that focus heavily on adjacent tokens, especially in the earlier layers. However, this paper shows that there are several heads that focus on themselves and that heads focusing heavily on adjacent token heads were found in various layers. What is the reason for this discrepancy?\n\nClark+'19 used bert-base-uncased, while we used the roberta-base. We created a bert-base-uncased version of Figure 2. The heads in which 90% of the attention matrices was assigned to the same cluster were as follows.\n\n| cluster       | heads (notation is `<layer>-<head_number>` as in Clark+'19)    |\n| :------------ | ---------------------------------------------------------------- |\n| next-to-left  | 2-5, 4-6, 7-12, 8-5                                              |\n| leftward      | 1-3, 1-4, 3-2, 4-12, 5-8, 5-11, 6-11, 7-4, 7-5, 7-10, 8-10, 8-12 |\n| next-to-right | 1-11, 2-2, 3-1, 3-10, 4-10, 5-12, 6-10, 7-7, 8-3                 |\n| rightward     | 2-7, 3-12, 4-4, 5-6, 6-1, 7-6, 8-2, 8-9, 8-11, 9-6, 9-10         |\n| itself        | 3-7, 10-7, 11-10, 11-11, 12-9                                    |\n\nFor comparison, we quote the relevant part of Clark+'19: \"In particular four attention heads (in layers 2, 4, 7, and 8) on average put >50% of their attention on the previous token and five attention heads (in layers 1, 2, 2, 3, and 6) put >50% of their attention on the next token\".\n\nNext-to-left seems to be in agreement. However, there is still a discrepancy in that our method finds more heads in which each token focuses attention on itself. A possible reason is that **the t-offset trace (Eq. 7) emphasizes attention patterns that depend on relative positions**.\n\nFor example, consider the attention matrix $A$ of size 10x10 shown below.\nThis can be regarded as a case of mixed attention to relative positions and attention to specific tokens (e.g., special tokens or period).\n\n$$\nA = \\begin{bmatrix}\n0.1 & 0 & 0 & 0 & 0 & 0.9 & 0 & 0 & 0 & 0 \\\\\\\\\n0 & 0.1 & 0 & 0 & 0 & 0.9 & 0 & 0 & 0 & 0 \\\\\\\\\n0 & 0 & 0.1 & 0 & 0 & 0.9 & 0 & 0 & 0 & 0 \\\\\\\\\n0 & 0 & 0 & 0.1 & 0 & 0.9 & 0 & 0 & 0 & 0 \\\\\\\\\n0 & 0 & 0 & 0 & 0.1 & 0.9 & 0 & 0 & 0 & 0 \\\\\\\\\n0 & 0 & 0 & 0 & 0 & 1 & 0 & 0 & 0 & 0 \\\\\\\\\n0 & 0 & 0 & 0 & 0 & 0.9 & 0.1 & 0 & 0 & 0 \\\\\\\\\n0 & 0 & 0 & 0 & 0 & 0.9 & 0 & 0.1 & 0 & 0 \\\\\\\\\n0 & 0 & 0 & 0 & 0 & 0.9 & 0 & 0 & 0.1 & 0 \\\\\\\\\n0 & 0 & 0 & 0 & 0 & 0.9 & 0 & 0 & 0 & 0.1 \\\\\\\\\n\\end{bmatrix}\n$$\n\nThe t-offset trace of this matrix takes its maximum value at $t=0$.\n\nAs the number of input tokens is increased, the attention that depends on the relative positions becomes more emphatic.\nThus, even if the average attention to itself is small, the t-offset trace can be maximized at relative position $t=0$.\n\nThis effect is true for any relative position, not just $t=0$.\nThat is, our method can detect the heads for which the actual value of attention is small, but for which attention consistently depends on a certain relative position.\n\n> C. Nowadays, relative position embedding is often used instead of APE. Can this paper provide insights into the reasons for the superiority of relative position embedding? For example, can we interpret that relative position embedding can capture other language information richer than APE because there is no need to extract relative positions in the query or key transformation matrices?\n\nIn our paper, we showed that inference with APE also depended on relative positions. In other words, the goal of APE and RPE is the same: to acquire translation invariance. However, APE requires the entire model to acquire translation invariance, whereas RPE directly models the property. Thus, APEs are at a disadvantage if the inference requires translation invariance.\n\nSpecifically, in order to shift the position where attentions are focused by a specific number of tokens, the APE and parameters must learn the frequency and the rotation angle, respectively, to follow Eq. 27 (Fig. 10).\n\n> A. What is the substantial contribution to the community from these findings? For example, could they lead to any ideas to improve the Transformer architecture?\n\nThis could be one of the reasons why APE is not a good choice for position embedding.\n\nAssuming that the acquisition of translation invariance of the model is essential for language processing, APEs are structurally disadvantaged in acquiring this property, as noted in the response to Question C.\n\n---\n\n> Findings are interesting, but no statement of what the contribution is and how practical impact on the community or practical use. (Question A).\n\nI completely agree that it is valuable to apply the accumulated technology to the real world. However, we believe that the goal of NLP also includes the clarification of the mechanisms by which machines process language, i.e., \"Interpretability, Interactivity and Analysis of Models for NLP,\" the name of this track. If practicality is a requirement for acceptance, we fear that research other than proposals for new models, architectures, etc. will be at a disadvantage unreasonably.\n\n> Most of the experiments (excluding Section 4.1.1) are limited to RoBERTa-base only, and it is unclear if the results can be generalized to other models adopting learnable APEs. It is important to investigate whether the results can be generalized to differences in model size, objective function, and architecture (i.e., encoder, encoder-decoder, or decoder). In particular, it is worthwhile to include more analysis and discussion for GPT-2. For example, I would like to see the results of Figure 2 for GPT-2.\n\nThis paper targets the phenomenon that, **despite that self-attention without causal mask is symmetric with respect to position, its inference is dependent on position**. It is interesting to see how decoders, which pay attention only to the preceding tokens, handle position, but this is a topic out of the scope of this paper, since the principles of causal modeling and bidirectional modeling are completely distinct.\n\n> Introduction says to analyze \"why attention depends on relative position,\" but I cannot find content that adequately answers this question.\n\nWe will correct \"how and why\" in L062 to \"how\".", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Analyses use convincing and solid mathematical tools.\n- Mechanism for realizing relative position-dependent attention patterns from absolute position embedding is really interesting.", "reasons_to_reject": "- Limited Experiments\n  - Most of the experiments (excluding Section 4.1.1) are limited to RoBERTa-base only, and it is unclear if the results can be generalized to other models adopting learnable APEs. It is important to investigate whether the results can be generalized to differences in model size, objective function, and architecture (i.e., encoder, encoder-decoder, or decoder). In particular, it is worthwhile to include more analysis and discussion for GPT-2. For example, I would like to see the results of Figure 2 for GPT-2.\n  - The input for the analysis is limited to only 100 or 200 samples from wikitext-2. It would be desirable to experiment with a larger number of samples or with datasets from various domains.\n- Findings are interesting, but no statement of what the contribution is and how practical impact on the community or practical use. (Question A).\n- Results contradicting those reported in existing studies (Clark+'19) are observed but not discussed (Question B).\n- I do not really agree with the argument in Section 5 that word embedding contributes to relative position-dependent attention patterns. The target head is in layer 8, and the changes caused by large deviations from the input, such as only position embedding, are quite large at layer 8. It is likely that the behavior is not such that it can be discussed to explain the behavior under normal conditions. Word embeddings may be the only prerequisites for the model to work properly rather than playing an important role in certain attention patterns.\n- Introduction says to analyze \"why attention depends on relative position,\" but I cannot find content that adequately answers this question.\n- There is no connection or discussion of relative position embedding, which is typically employed in recent Transformer models in place of learnable APE (Question C).", "questions_to_authors": "- A. What is the substantial contribution to the community from these findings? For example, could they lead to any ideas to improve the Transformer architecture?\n- B. Clark+'19 reported that most heads pay little attention to themselves and that there are heads that focus heavily on adjacent tokens, especially in the earlier layers. However, this paper shows that there are several heads that focus on themselves and that heads focusing heavily on adjacent token heads were found in various layers. What is the reason for this discrepancy?\n- C. Nowadays, relative position embedding is often used instead of APE. Can this paper provide insights into the reasons for the superiority of relative position embedding? For example, can we interpret that relative position embedding can capture other language information richer than APE because there is no need to extract relative positions in the query or key transformation matrices?", "paper_topic_and_main_contributions": "Positional encoding is important in Transformer architecture, and until a few years ago, learnable absolute position embedding (APE) was often used (e.g., BERT, RoBERTa, GPT-2).\n\nClark+'19 reported that some attention heads in BERT attend to context words according to their relative positions; Ravishankar&Søgaard+'21 reported that some columns of absolute position embeddings are periodic; Chang+'22 reported that the position information is encoded in a hidden representation while remaining periodic.\nHowever, it is not clear how periodicity is used in models, which is what this paper attacks.\n\nSpecifically, this paper showed that several attention heads in RoBERTa realize an attention pattern that depends on relative position by extracting the periodic components derived from APE from the hidden representation with shifting the phase in query and key transformation.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "- In addition to Clark+'19, Kovaleva+'19 should be mentioned as a representative attention analysis\n    - Kovaleva+, Revealing the Dark Secrets of BERT (EMNLP2019)"}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Caption of Figure 1: Attention weights of the first 15 tokens -> Attention weights for the first 15 tokens of an input text? If it is a special input text, please explain explicitly.\n- ll.223-227: I cannot follow what you meant. Please provide a clearer explanation.\n- ll.263-265: Should mention the agreement with Lin+'19 results here\n- (minor) l.200: Consider changing the variable name PE, since PE appears to be the product of two matrices P and E."}}, {"review_id": "aQtWnkoUAX", "reviewer": "Reviewer_5Jmf", "summary": "", "questions": "\n\n", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We will fix the problems mentioned in the Reasons To Reject section in camera-ready version.\n\n> The relationships between different sections are not clear and authors should give a clear description of the relationship in the introduction. In the Introduction, these factors are put in Section 4. However, Section 3 also shows the factors of RPE.\n\nIn the camera-ready version, an description of the standpoint in Section 3 will be added to the Introduction.\n\nSection 3 demonstrates the subject of analysis in this paper. It is a phenomenon in which the type of attention using Absolute Position Embedding depends on the relative positions. The mechanism is then analyzed in sections 4 and 5.\n\n> Besides Section 4 focuses on nearby tokens and Section 5 focuses on adjacent tokens, while adjacent tokens can be viewed as one part of nearby tokens. It is better to combine Section 4 and 5.\n\nAdjacent tokens are indeed included in the nearby tokens, but the mechanism by which attention is focused on each may be different, as shown in §4.3 and Fig. 9.\n\n> Authors should make the equation more prescriptive. Line 244, $Xa$ -> $X_a$, $Yb$ -> $Y_b$.\n\nSince $a$ and $b$ are weight vectors that compose the synthetic variables $Xa$ (and $Yb$ ) as the product of $X$ and $a$ (and $Y$ and $b$ ), the original notation is correct.\nIn the camera-ready version, we refer to $a$ and $b$ as parameters.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper introduces an interesting work on the relationship between self-attention and position embeddings. The factors proposed in the paper are based on several experiments and the results are persuasive. ", "reasons_to_reject": "There are some problems in this paper\n\n1.  The relationships between different sections are not clear and authors should give a clear description of the relationship in the introduction. In the Introduction, these factors are put in Section 4. However, Section 3 also shows the factors of RPE. Besides Section 4 focuses on nearby tokens and Section 5 focuses on adjacent tokens, while adjacent tokens can be viewed as one part of nearby tokens. It is better to combine Section 4 and 5.\n\n2. Authors should make the equation more prescriptive. Line 244, $X$a->$X_a$, $Y$b->$Y_b$. Line 329, $q$s->$q=[q_n]$, $k$s->$k=[k_n]$ ", "questions_to_authors": "\n\n", "paper_topic_and_main_contributions": "This paper analyzes the mechanism of relative positional embeddings and shows that the relative positional dependence of attention emerges due to some factors. Besides, the word embedding is also a factor that enables inference based on relative position for the attention strongly concentrated on the adjacent tokens. ", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "1oyU2ZDkEj", "reviewer": "Reviewer_8B3x", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Thorough analysis of attention mechanisms. The paper's findings on the impact of position embeddings on attention is particularly valuable. \n2. Clear presentation of experimental results and findings. \n3. The authors' use of theoretical interpretations and visualization techniques to illustrate attention behavior is particularly helpful in conveying their findings to readers.", "reasons_to_reject": "1. The authors' findings are related to the models and training objectives used in their study, and may not be entirely generalizable. However, the detection and analysis methods proposed by the authors can be applied to other models.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper mainly discusses the behavior of attention mechanisms in pre-trained language models, specifically focusing on the impact of position embeddings on attention. The authors conduct a series of experiments to analyze the relationship between attention and position embeddings, and also investigate the effect of different model architectures and languages on attention behavior. \nFor example，the authors find that 1) learnable absolute position embedding contains sinusoid-like waves, 2) attention heads are able to extract periodic components from the position embedding in the hidden states, and 3) the self-attention mechanism is responsible for adjusting the phase of the periodic components in both the query and the key, thus influencing the direction of attention.\nThese findings provide a better understanding of the mechanisms underlying position encoding in attention, which can inform future model design and development.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper tries to understand how Transformer models with learned positional encodings manage to understand relative position (as seen, e.g., by relative position attention heads). The authors use DFT and PCA to show that positional embeddings encode sinusoid-like waves, and CCA along with spectral analysis for query and key in attention to show that attention to previous/next token is possible because the phases of these waves are shifted by the same number tokens. The paper’s experiments are only focused on RoBERTa models and the results are applicable only to MLM training objectives.

Overall, reviewers agreed that the discovered mechanisms are interesting and the used analysis tools look solid. The main concerns are that the experiments are limited to a single model, a very small subset of examples used in the analysis (100-200 samples from wikitext-2), very weak discussion of connections to other work. I agree with these concerns and they do not seem to be addressed in the rebuttal.
Decision: Accept-Main