ON POSITION EMBEDDINGS IN BERT
Year: 2021

Abstract:
Various Position Embeddings (PEs) have been proposed in Transformer based architectures (e.g. BERT) to model word order. These are empirically-driven and perform well, but no formal framework exists to systematically study them. To address this, we present three properties of PEs that capture word distance in vector space: translation invariance, monotonicity, and symmetry. These properties formally capture the behaviour of PEs and allow us to reinterpret sinusoidal PEs in a principled way. Moreover, we propose a new probing test (called ‘identical word probing’) and mathematical indicators to quantitatively detect the general attention patterns with respect to the above properties. An empirical evaluation of seven PEs (and their combinations) for classification (GLUE) and span prediction (SQuAD) shows that: (1) both classification and span prediction benefit from translation invariance and local monotonicity, while symmetry slightly decreases performance; (2) The fully-learnable absolute PE performs better in classification, while relative PEs perform better in span prediction. We contribute the first formal and quantitative analysis of desiderata for PEs, and a principled discussion about their correlation to the performance of typical downstream tasks.

heading: 1 INTRODUCTION
text: Position embeddings (PEs) are crucial in Transformer-based architectures for capturing word order; without them, the representation is bag-of-words. Fully learnable absolute position embeddings (APEs) were first proposed by Gehring et al. (2017) to capture word position in Convolutional Seq2seq architectures. Sinusoidal functions were also used with Transformers to parameterize PEs in a fixed ad hoc way (Vaswani et al., 2017). Recently, Shaw et al. (2018) used relative position embedding (RPEs) with Transformers for machine translation. More recently, in Transformer pretrained language models, BERT (Devlin et al., 2018; Liu et al., 2019) and GPT (Radford et al., 2018) used fully learnable PEs. Yang et al. (2019) modified RPEs and used them in the XLNet pre-trained language model. To our knowledge, the fundamental differences between the various PEs have not been studied in a principled way.
We posit that the aim of PEs is to capture the sequential nature of positions in vector space, or technically, to bridge the distances in N (for positions) and RD (for position vectors). We therefore propose three expected properties for PEs: monotonicity, translation invariance, and symmetry 1. Using these properties, we formally reinterpret existing PEs and show the limitations of sinusoidal
1Informally, as positions are originally positive integers, one may expect position vectors in vector space to have the following properties: 1) neighboring positions are embedded closer than faraway ones; 2) distances of two arbitrary m-offset position vectors are identical; 3) the metric (distance) itself is symmetric.
PEs (Vaswani et al., 2017): they cannot adaptively meet the monotonicity property – thus we propose learnable sinusoidal PEs.
We benchmark 13 PEs (including APEs, RPEs, and their combinations) in GLUE and SQuAD, in a total of 11 individual tasks. Several indicators are devised to quantitatively measure translation invariance, monotonicity, and symmetry, which can be further used to calculate their statistical correlations with empirical performance in downstream tasks. We empirically find that both text classification tasks (in GLUE) and span prediction tasks (SQuAD V1.0 and V 2.0) can benefit from monotonicity (in nearby offset) and translation invariance (in particular without considering special tokens like [CLS]), but symmetry decreases performance since it can not deal with directions between query vectors and key vectors when calculating attentions. Plus, models with unbalanced attention regarding directions (generally attending more to preceding tokens than to succeeding tokens) slightly correlate with better performance (especially for span prediction tasks).
Experiments also show that the fully-learnable APE performs better in classification, while RPEs perform better in span prediction tasks. This is explained by our proposed properties as follows: RPEs perform better in span prediction tasks since they meet better translation invariance, monotonicity , and asymmetry; the fully-learnable APE which does not strictly have the translation invariance and monotonicity properties during parameterizations (as it also performed worse in measuring translation invariance and local monotonicity than other APEs and all RPEs) still performs well because it can flexibly deal with special tokens (especially, unshiftable [CLS]).
Regarding the newly-proposed learnable sinusoidal PEs, the learnable sinusoidal APE satisfies the three properties to a greater extent than other APE variants, and the learnable sinusoidal RPE exhibits better direction awareness than other PE variants. Experiments show that BERT with sinusoidal APEs slightly outperforms the fully-learnable APE in span prediction, but underperforms in classification tasks. Both for APEs and RPEs, learning frequencies in sinusoidal PEs appears to be beneficial. Lastly, sinusoidal PEs can be generalized to treat longer documents because they completely satisfy the translation invariance property, while the fully-learnable APE does not.
The contributions of this paper are summarised below: 1) We propose three principled properties for PEs that are either formally examined or empirically evaluated by quantitative indicators in a novel Identical Word Probing test; 2) We benchmark 13 PEs (including APEs, RPEs and their combinations) in GLUE, SQuAD V1.1 and SQuAD V2.0, in a total of 11 individual tasks; 3) we experimentally evaluate how the performance in individual tasks benefits from the above properties; 4) We propose two new PEs to extend sinusoidal PEs to learnable versions for APEs/RPEs.

heading: 2 PROPERTIES OF POSITION EMBEDDINGS
text: Gehring et al. (2017); Vaswani et al. (2017) use absolute word positions as additional features in neural networks. Positions x ∈ N are distributively represented as an embedding of x as an element ~x ∈ RD in some Euclidean space. By standard methods in representation learning, similarity between embedded objects ~x and ~y is typically expressed by an inner product 〈~x, ~y〉, for instance the dot product gives rise to the usual cosine similarity between ~x and ~y. Generally, if words appear close to each other in a text (i.e., their positions are nearby), they are more likely to determine the (local) semantics together, than if they occurred far apart. Hence, positional proximity of words x and y should result in proximity of their embedded representations ~x and ~y. One common way of formalizing this is that an embedding should preserve the order of distances among positions 2. We denote φ(·, ·) as a function to calculate closeness/proximity between embedded positions, and any inner product can be a special case of φ(·, ·) with good properties. We can express preservation of the order of distances as: For every x, y, z ∈ N,
|x− y| > |x− z| =⇒ φ(~x, ~y) < φ(~x, ~z) (1)
Note that on the underlying space, the property in Eq. (1) has been studied for almost 60 years (Shepard, 1962), in both algorithmics (Bilu & Linial, 2005; Badoiu et al., 2008; Maehara, 2013), and machine learning (Terada & Luxburg, 2014; Jain et al., 2016) under the name ordinal embedding. As we are interested in the simple case of positions from N, Eq. (1) reduces to the following property:
2Theoretical evidence for this is nontrivial unless we assume more about the particular non-linear functions. We empirically find that all learned PEs can preserve the order of distance
Property 1. Monotonicity: The proximity of embedded positions decreases when positions are further apart:
∀x,m, n ∈ N : m > n⇐⇒ φ(~x,−−−−→x+m) < φ(~x,−−−→x+ n) (2) A priori, a position embedding might treat every element N individually. However, considering pairs of positions based on their relative proximity (rather than the absolute value of the positions), can lead to simplified and efficient position embeddings (Wang et al., 2020). Such embeddings satisfy translation invariance:
Property 2. Translation invariance: The proximity of embedded positions are translation invariant:
∀x1, . . . , xn,m ∈ N : φ(~x1, −−−−→ x1 +m) = φ(~x2, −−−−→ x2 +m) = · · · = φ(~xn, −−−−−→ xn +m) (3)
Finally, since the inner product is symmetric, we also consider whether φ(·, ·) is symmetric: Property 3. Symmetry: The proximity of embedded positions is symmetric,
∀x, y ∈ N : φ(~x, ~y) = φ(~y, ~x) (4)
There is no generally accepted standard set of properties for position embeddings; based on prior work as described above, we posit that the above properties are important, and now examine several existing PEs in relation to these properties, either formally (in Sec. 3) or empirically (in Sec. 4).

heading: 3 UNDERSTANDING PES VIA THE PROPERTIES
text: PEs come in two variants: absolute PEs (APEs) where single positions are mapped to elements of the representation space, and relative PEs (RPEs) where the difference between positions (i.e., x−y for x, y ∈ N) is mapped to elements of the embedding space. For Transformer-based architectures, the difference between APEs and RPEs manifests itself in the attention mechanism, in particular how the matrices of query, key, and value weights WQ, WK , and WV are used to calculate attention in each attention head. Consider two positions x, y ∈ N, let WEx be the word embedding of the word at position x, and let Px and Px−y be the embeddings of the position x and relative position x− y, respectively. The query-key-value vector for the word at position x is typically calculated as below for APEs and RPEs3 respectively:
APE: QxKx Vx  = (WEx + Px) WQWK WV  ; RPE: QxKx Vx  = WEx WQWK WV  +  0Px−y Px−y  (5) Observe that while the APEs calculation is linear in (WQ,WK ,WV ) with the word and position embeddings merged into the coefficient, the RPEs calculation is affine, with the relative position embedding Px−y acting as an offset independent of the word embedding WEx.
In Transformers, the resulting representation is a sum of value vectors with weights depending on A = QKT , that is, Attention(Q,K, V ) = softmax(QKT / √ dk)V . In the rest of the paper, we examine PEs in the above architecture with respect to the properties introduced in Section 2. In particular, we study four well-known variants of PEs: (1) the fully learnable APE (Gehring et al., 2017), (2) the fixed sinusoidal APE (Vaswani et al., 2017), (3) the fully learnable RPE (Shaw et al., 2018), and (4) the fixed sinusoidal RPE (Wei et al., 2019).

heading: 3.1 UNDERSTANDING SINUSOIDAL PES
text: With a sinusoidal parameterization in PEs, we may use a specific proximity, i.e., an efficient inner product like a dot product, to check if the sinusoidal form of PEs meets the above properties. The dot product between any two position vectors is
Ax,y = 〈~x, ~y〉 = sum   sin(ω1x) cos(ω1x) · · ·
sin(ωD 2 x) cos(ωD 2 x)
  sin(ω1y) cos(ω1y) · · ·
sin(ωD 2 y) cos(ωD 2 y)
  = sum   sin(ω1x) sin(ω1y) cos(ω1x) cos(ω1y) · · · sin(ωD
2 x) sin(ωD 2 y)
cos(ωD 2 x) cos(ωD 2 y)
  = D 2∑
i=0
cos(ωi(x− y)) (6)
3There are many variants of RPEs (e.g., (Dai et al., 2019)). As selecting RPEs is not the main concern in this paper, we give the original (and typical) RPEs only. One can easily extend this work to other RPE variants.
Note that sinusoidal PEs satisfy both Property 2 (translation invariance) because the inner product is only associated with its position difference x − y, and Property 3 (symmetry), because the dot product itself is symmetric: 〈~x, ~y〉 = 〈~y, ~x〉. Note also that checking Property 1 is equivalent to checking monotonicity of the map ψ(m) = ∑D/2 i=1 cos(ωim). ψ(m) is monotone on intervals where
its first order derivative ψ′(m) = ∑D/2
i=1 −ωi sin(ωim) does not change sign, and these intervals depend on the choice of ωi. With fixed frequencies ωi = (1/10000)2i/D, it is monotonous when m is roughly between 0 and 50, indicating that it can only strictly perceive a maximum distance of 50 and it is insensitive to faraway distances (e.g. longer than 50).
Although sinusoidal PEs with fixed frequencies (i.e., ωi = (1/10000)2i/D) are common in APEs and RPEs, we argue that learning these frequencies is useful because it can adaptively adjust intervals of monotonicity (they do not have to be 0-50 as in the fixed sinusoidal APE) 4. With trainable frequencies, we can adaptively allocate a number of frequencies in a data-driven way. App. A.2 explains the expressive power of sinusoidal PEs with trainable frequencies from the perspective of the Fourier series. Extending existing fixed sinusoidal PEs to a learnable version with learnable frequencies gives two variants: a learnable sinusoidal APE and a learnable sinusoidal RPE.

heading: 3.2 UNDERSTANDING RPES
text: RPEs ignore the absolute position of words and directly encode their relative distance. The RPEs expression adheres to the translation invariance property during parameterization, since relative distance with the same offset will be embedded as the same embedding, namely, Px1−y1 = Px2−y2 if x1−y1 = x2−y2. Plus, RPEs that separately embed forward and backward relative embeddings, i.e., Pi−j 6= Pj−i, do not meet symmetry during parameterization. Sinusoidal RPEs can also embed neighboring relative position in close vectors with a local monotonicity, similarly to sinusoidal APEs. Note that the dot products between two sinusoidal relative position vectors with the same offset, without distinguishing positive negative relative position vectors, should be identical 5. This makes it hardly perceive of the border between preceding and succeeding relative position vectors.

heading: 4 EXAMINING PE PROPERTIES IN PRE-TRAINED LANGUAGE MODEL
text: We train BERT with six basic PEs as in Tab. 1 and their combination variants, and conduct a probing test to check to which degree they satisfy the properties.
Pre-training The pre-trained “BERT-base-uncased” checkpoint (Devlin et al., 2018) is used to train by replacing the original absolute PE module with a new PE variant (including APEs and RPEs). We train the new models with a sequence length of 128 for 5 epochs and then 512 for another 2 epochs. The training is the same as in the original BERT, i.e., BooksCorpus and Wikipedia (16G raw documents) with whole word masking. To be fair, the BERT with the original fully-learnable
4See App. A to intuitively understand specific functions of each frequency ωi 5Namely, 〈Px1−y1 , Px2−y2〉 = 〈Px3−y3 , Px4−y4〉 if (x1−y1)− (x2−y2) = (x3−y3)− (x4−y4) = m,
in both x− y > 0 and x− y < 0.
APE is also further trained in the same way. All models have about 110M parameters corresponding to a typical base setting, with minor differences solely depending on the parameterization in Tab. 1.

heading: 4.1 DOT PRODUCT BETWEEN POSITION VECTORS
text: APEs We calculate dot products between two arbitrary position vectors for APEs and RPEs (see Fig. 1). For APEs, neighboring position vectors are generally closer compared to faraway ones. This trend is clearer in the learnable sinusoidal APE, which imposes a strict sinusoidal regularization for PEs. Note that additionally adopting RPEs does not affect too much PE patterns, as can be seen by comparing Fig. 1(a) and 1(b), or Fig. 1(c) and 1(d).
RPEs In the fully-learnable RPE setting, the vertical and horizontal bright bands in 1(e) and 1(f) show that the relative position vectors for small offsets (e.g., {P−5, · · · , P0, · · ·P5} ) are notably different to other relative position vectors; it indicates that the relative position vectors with small offsets are more distinguishable than faraway relative position vectors. The four dark corners in 1(e) and 1(f) means that relative position vectors with longer offset than 20, i.e., from -64 to -20 and from 20 to 64, are very close, showing that the fully-learnable RPE does not significantly distinguish far-distant RPEs. This suggests that truncating RPEs into a fixed distance (e.g. 64 in (Shaw et al., 2018)), is reasonable. This effect is further explained in App. D.

heading: 4.2 IDENTICAL WORD PROBING
text: In APEs, the attention matrix (A = softmax(QKT )) is related to individual words and their positions, an element of (inactivated) A in the first layer is given by:
aij = (wi + pi)W Q,1((wj + pj)W K,1)T
= wiW Q,1(WK,1)TwTj︸ ︷︷ ︸
word-word correspondence
+wiW Q,1(WK,1)T pTj︸ ︷︷ ︸
word-position correspondence
+ piW Q,1(WK,1)TwTj︸ ︷︷ ︸
word-position correspondence
+ piW Q,1(WK,1)T pTj︸ ︷︷ ︸
position-position correspondence
(7)
Identical word probing for PEs To study the effect of only PEs in A without considering individual words, we use identical word probing: feed many repeated identical words (can be arbitrary,
6In all figures we only show the first 128 positions instead of 512 positions since they are in principle compatible. Practically in BERT, there is a minor discrepancy between the first 128 positions and the remaining positions due to the typical training strategy (first training on 128-length input and then 512-length input).
denoted as w̄) as a sentence to BERT to check the attention values Ā(1), with each element ā1ij(w̄) = (w̄ + pi)W Q,1((w̄ + pj)W K,1)T (8)
As we take an average of Ā(1) over many randomly-selected words w̄, the general patterns of Ā(1) will not be affected by any particular word. Namely, Ā(1) is word-free and only related to learned PEs. Thus, Ā(1) can be treated as a general attention bias and can also implicitly convey positionwise proximity in Transformers. Note that the probing test could also be applied to RPEs.

heading: 4.2.1 QUALITATIVE ANALYSIS
text: Fig. 2 shows the average attention weights among all heads in the first layer. BERT without PE nearly treats all words uniformly (bag-of-words). Almost all APEs and RPEs have a clear pattern of translation invariance, local monotonicity in a neighboring window, and symmetry. Note that this is nontrivial since no specific constraints or priors were imposed on fully-learnable APEs/RPEs 7.
BERT with APEs does not show any direction awareness since Fig. 2(b) and 2(c) are nearly symmetrical. As seen from Fig. 2(f,h), BERT with learnable sinusoidal RPE generally attends more on forward tokens than backward tokens, which cannot be clearly found in fully-learnable RPE and fixed sinusoidal RPE. Interestingly, the white bands along the diagonal in Fig. 2 (d, f, g) suggest that some words generally do not attend to themselves, as previously observed in (Clark et al., 2019) 8 .

heading: 4.2.2 QUANTITATIVE ANALYSIS
text: Using the activated attention values Ā(1) in Eq. 8 9 we adopt three quantitative indicators to measure to which extent BERT models with individual PEs satisfy the three properties and their derivative indicators (see App. B for details of calculating these indicators) in Tab. 2. Basically, all APEs and RPEs satisfy monotonicity in small offsets and translation invariance compared to BERT without PE; All PEs nearly satisfy symmetry except for the learnable sinusoidal RPE and its combinations.
APEs and RPEs The learnable sinusoidal APE better satisfies all three properties than fully learnable APE and fixed sinusoidal APE; this is due to its sinusoidal parameterization and flexible frequencies. RPEs satisfy translation invariance to a higher degree than APEs, because they directly
7See App. G for an example of the evolution of PE patterns in fully learnable APE which starts from random initialization and ends with some patterns reflecting the properties.
8See App. J for more details about the white band effect along the diagonal. 9300 random-selected words were used to calculate average Ā(1), as we empirically found that adopting
more words almost does not change Ā(1).
satisfy translation invariance during parameterization. In the last column, direction balance values of all PEs except for the fixed sin. APE are larger than one, which indicates that BERT models with all PEs generally attend more to preceding tokens than succeeding tokens, and this phenomenon appears to be stronger in learnable sinusoidal RPEs than others.
The fully learnable APE and [CLS] Fully learnable APE generally performs worse in translation invariance (see the 4-th column) as it has to deal with the unshiftable [CLS] which is always in the first position. Without considering [CLS] and [SEP] (see the 5-th column), the fully learnable APE satisfies translation invariance better than other APEs, showing that the fully learnable APE can flexibly deal with both special tokens and normal positions. The fully learnable APE also could handle the mismatch between special tokens and normal positions in the monotonicity property.

heading: 5 PES IN DOWNSTREAM TASKS
text: We empirically compare the performance of PEs in classification and span prediction tasks.
Fine-tuning The fine-tuning on GLUE and SQuAD is the same as in the Huggingface website as per Wolf et al. (2019), see App. E for details. We report the average values of five runs per dataset. For classification, we use the GLUE (Wang et al., 2018) benchmark, which includes datasets for both single document classification and sentence pair classification. For span prediction, we use the SQuAD V1.1 and V2.0 datasets consisting of 100k crowdsourced question/answer pairs (Rajpurkar et al., 2016). Given a question and a passage from Wikipedia containing the answer, the task is to predict the answer text span in the passage. In V2.0, it is possible that no short answer exists in the passage since it additionally has 50,000 unanswerable questions written adversarially by crowdworkers (Rajpurkar et al., 2018).

heading: 5.1 EXPERIMENTAL RESULTS FOR DOWNSTREAM TASKS
text: GLUE Tab. 3 shows that the fully-learnable APE (a.k.a, BERT-style APE) performs well in GLUE. No PE variants, especially BERT with solely APEs or RPEs, notably outperform the fullylearnable APE. BRRT models with a combination of an APE and an RPE do not always boost the performance of the model with solely the APE or RPE.
SQuAD Tab. 4 shows that nearly all BERT models with RPEs significantly outperform the fully learnable APE. The learnable sinusoidal APE is slightly better than the fully learnable APE in most cases. Both the best-performed models in SQuAD V1.1 and V2.0 adopt the fully-learnable RPE.
10‘monotonicity ’ (second column) refers to monotonicity calculating in all relative distance, while ‘monotonicity (first 20 offsets)’ (third column) refers to monotonicity calculating within a relative distance of 20 (see App. C for monotonicity in other offsets.); the later matters since neighboring words in a small window are crucial in natural language. For translation invariance, we also adopt a new indicator without considering special tokens ([CLS] and [SEP]) to measure a purely position-aware translation invariance.
As demonstrated in Tab. 2, the fully learnable APE can flexibly deal with [CLS] and translation invariance in normal positions, thus it performs well in classification tasks (GLUE) which heavily relies on the unshiftable [CLS] token for inference. Span prediction tasks which do not infer from [CLS] can benefit from strict translation invariance during parameterization (e.g., sinusoidal APEs and RPEs), see Tab. 5 in Sec. 6.1 for the correlations between performance of SQuAD and the translation invariance property. Removing PEs (BERT without PE) dramatically decreases performance in SQuAD V1.1 and V2.0, and slightly harms performance on GLUE, showing that PEs are more important in SQuAD than GLUE.
Learnable sinusoidal PEs The sinusoidal APEs outperform fully-learnable APE in span prediction but underperform it in classification tasks. The learnable sinusoidal APE/RPE outperforms fixed sinusoidal APE/RPE in GLUE and SQuADs, showing the expressive power of flexible frequencies.
Complementarity of APEs and RPEs In SQuAD, jointly adopting APEs and RPEs can slightly boost performance in some cases. For instance, BERT with learnable sinusoidal APE + APE + fully RPE achieves the best EM score in both SQuADs. However, this complementary effect is relatively weaker in GLUE, where the fully-learnable APE performs strongly.

heading: 6 DISCUSSIONS ON PES

heading: 6.1 HOW DO THE PROPERTIES CORRELATE TO INDIVIDUAL TASKS?
text: We conduct a correlation analysis between the properties and the performance on individual tasks 11, as shown in Tab. 5. The results show that violating monotonicity in relatively-small offsets (e.g., 20) and translation invariance is harmful since it is negatively correlated to the performance on
11Pearson correlations are calculated between the property indicators and the performance of each individual task for 12 PEs. BERT without PE was not considered, since its property indicators are significantly different with other PEs and its performance is much worse; it therefore unexpectedly increases correlation values.
Table 5: Pearson correlations between the properties and evaluated tasks, evaluating on BERT models with 13 position embeddings. The positive (negative) numbers denote to which degree the performance of the task positively (negatively) correlate(s) to violating the property. This shows that
violating local monotonicity and translation invariance is harmful, while violating symmetry (and direction-balance) is beneficial. Best correlation values are in bold for each row.
Properties CoLA SST-2 MNLI QQP GLUE SQuAD V1.1 SQuAD V2.0
monotonicity all offsets 0.44 0.43 0.56 0.32 0.48 -0.31 -0.27first 20 offsets -0.18 0.44 -0.24 -0.42 -0.21 -0.91 -0.86 translation invariance w/ [CLS]/[SEP] 0.48 0.52 0.04 -0.07 0.42 -0.63 -0.57w/o [CLS]/[SEP] -0.47 0.01 -0.69 -0.68 -0.61 -0.51 -0.58 symmetry 0.17 0.24 0.40 0.09 0.31 0.15 0.16 direction balance 0.32 0.16 0.63 0.35 0.48 0.32 0.37
GLUE and SQuAD. However, violating symmetry (and direction-balance) is slightly beneficial. This shows that many tasks require BERT models to distinguish preceding and succeeding tokens, especially to attend more on preceding tokens. See Fig. 5b in App. C, the correlations between the direction balance indicators and the performance of downstream tasks will be much higher when only considering a few neighboring tokens for calculating the indicator.

heading: 6.2 MORE DISCUSSIONS ON THE PROPOSED PROPERTIES
text: Monotonicity Monotonicity holds locally in a small neighboring window (usually in 5-20 offsets) for all PE variants, see Fig.2. This shows that BERT models generally are not sensitive to longerdistance attendance patterns, also evidenced by the fact that performance in downstream tasks correlates more highly with monotonicity in middle-distance offsets (e.g., 20 in the second row of Tab. 5) than longer offsets (see App. C). To check monotonicity guided by learned frequencies of learnable sinusoidal APEs in individual tasks, see App. A.3
Translation invariance In BERT, we argue that absolute positions of words are uninformative since (1) absolute positions of the second segment depend on the length of the first sentence; (2) words are randomly truncated in the beginning or end if a sentence exceeds the expected maximum length, which may shift absolute positions of all tokens with an unexpected offset (Devlin et al., 2018). That is, absolute positions of words in pre-trained language models are arbitrarily replaceable, and thus adopting translation invariance is generally reasonable. Models with strict Translation invariance (all RPEs and sinusoidal APEs) naturally make PEs generalize to longer documents than the documents used in the pre-training phase, see App. F for some empirical evidence.
Symmetry APEs (especially sinusoidal APEs) express symmetry patterns without distinguishing the direction as shown in Fig 2. As seen from Eq. 7, it is nontrivial to model directions in two linearly-transformed query vectors and key vectors. This limits its performance in direction-sensitive downstream tasks. RPEs could behave better on direction perception, since forward and backward relative embeddings are separately embedded (see Tab. 1); Especially, learnable sinusoidal RPE or combination variants including it have more unbalanced attending patterns (see the last column in Tab. 2), as shown in Fig. 2 (f) and (h),

heading: 7 CONCLUSION
text: To theoretically and empirically understand position embeddings (PEs), we have defined three properties (translation invariance, monotonicity, and symmetry) inspired by distance mappings between the original domain of positions in N and their PEs in RD. A probing test has been proposed to quantitatively examine these properties using appropriate mathematical indicators. Our probing test has shown that these PEs nearly satisfy most properties even when they are fully-learnable without constraints. Experimental results have shown that violating local monotonicity and translation invariance decreases performance in downstream tasks (classification and span prediction tasks), and that violating symmetry benefits downstream tasks because of direction awareness. We also find that the fully-learnable absolute PE in general results in better performance for classification, and that relative PEs result in better performance for span prediction tasks, which can be explained by the connections between their properties and task characteristics.

heading: ACKNOWLEDGMENTS
text: The work is supported by the Quantum Access and Retrieval Theory (QUARTZ) project, which has received funding from the European Union‘s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement No. 721321.

heading: A UNDERSTANDING FREQUENCIES
text: A.1 UNDERSTAND INDIVIDUAL FREQUENCIES
We argue in this paper that a learning schema for such frequencies will be useful in a sense it could adaptively adjust frequencies to meet different functions, see Fig. 3.
A.2 EXPRESSIVE POWER OF LEARNABLE SINUSOIDAL PES
In Transformers, linear transformation is commonly-used, for example query, key, and value transformations on word representations. Let ri be the word representation paramertezied by the sum of word embeddings and position embeddings (like the learnable sinusoidal APEs). Then, each element in ri
ri,k(t) = ei,k + pk(t) =
{ ei,k + sin(ω k
2 t), if k is even
ei,k + cos(ω k−1 2 t), if k is odd
(9)
After a linear transformation parameterized by w (e.g., the key transformation WK in the first Transformer layer), ri is linearly transformed as hi(t) = wri (hi(t) can be one of query/key/value vectors Qx,Kx, Vx in t-th position) with each element
hi,k(t) = D∑ k=1 wj,kei,k + D/2∑ k=1 (wj,2k sin(ω2kt) + wj,2k+1 cos(ω2k+1t)) (10)
The RHS is a typical Fourier series with a base term ∑ wj,kei,k and Fourier coefficients {wj,2k, wj,2k+1}. It is customarily assumed in physics and signal processing (Arfken & Weber, 1999) that the RHS in Eq. 10 with infinite D and appropriate frequencies could approximate any continuous function on a given interval.
As using an infinite D is not practical, dynamic allocation of a limited number of frequencies in a data-driven way could be beneficial for general approximation. The predefined frequencies ωi =
(1/10000)2i/D in the Transformer (Vaswani et al., 2017) can be considered as a special case when it enumerates various frequencies ranging from 1/10000 to 1 under a specific distribution.
A.3 LEARNED FREQUENCIES OF LEARNABLE SINUSOIDAL APE
The learned frequencies are shown in Fig. 4a. Observe that the learned frequencies are generally smaller than the pre-defined ones from (Vaswani et al., 2017) (i.e., ωi = (1/10000)2i/D). The learned frequencies are close to the learned one since we use ωi = (1/10000)2i/D as initialization.
As shown in Fig. 4b, the patterns of dot products between positions for learnable frequencies are quite different to the predefined ones (denoted as ‘default’ in the figure); indeed, the former appears more predisposed to deeming remote positions similar. Moreover, fine-tuned models for span prediction tasks (including SQuAD and SQuAD2) satisfy strict monotonicity in larger windows than for classification tasks. Observe also that the patterns in pre-training language models seem more similar to those in classification tasks than span prediction tasks.

heading: B QUANTITATIVELY MEASURING THE PROPERTIES.
text: To quantitatively measure the primary properties we treat in this paper, we propose multiple criteria, described below.
Assume a position-wise attention matrix Ā(1) (denoted as A since there is no risk for confusion), in which each element is the (softmax) activated attention value from the i-th query token to j-th key token (all elements are positive).
Average In-group Variance (AIV) for translation invariance Let l be an offset between two positions; we denote by τ(l) the set of l-offset attention values {Ai,j , j − i = l}; for example, τ(1) = {A1,2, A2,3, · · · , AL−1,L}. Translation invariance requires that all elements in each group τ(l) should be identical, the smaller variance each τ(l) has, it is closer to translation invariance. The Average In-group Variance (AIV) is defined as a weighted average over in-group variances of all {τ(l)}L−1−L+1, namely:
AIV(A) = ∑L−1 l=−L+1 var (τ(l)) · |τ(l)|∑L−1
l=−L+1 |τ(l)| where | · | is the number of elements in the set. For normalization, this metric is further divided by the overall variance (i.e., var(A)).
Ordered Pair Ratio (OPR) for monotonicity For a word in i-th position, based on the increasing distance to the i-th position, there a forward attention sequence Si,+ = {Ai,i, Ai,i+1, · · · , Ai,L} and a backward attention sequence Si,− = {Ai,i, Ai,i−1, · · · , Ai,1}. This results in 2L sequences denoted as S = {S1,+, S1,−, S2,+, S2,−, . . . , SL,+, SL,−}. The ideal (decreasing) monotonicity requires that each S (an element in S) is totally ordered as s0 > s1 > · · · > sL−1. We define the Ordered Ratio of S by:
OPR(S) =
∑ sj ,si∈S,i 6=j sign ((si − sj)(i− j))
|S|2 − |S| We define sign(x) = 1 if x > 0, and sign(x) = 0 otherwise. Ideally, the OPR of a totally ordered decreasing (increasing) sequence S should be zero (one). The expected OPR of a randomly-ordered sequence (average OPR of the set of all such sequences) should be 0.5.
Finally, we get a weighted sum of OPRs of all sequences in S. OPR(A) = ∑
S∈S OPR (S) · |S|∑ S∈S |S|
In the paper, we also consider a version of monotonicity within a offset of k (e.g., ‘monotonicity (first 20 offsets)’ in Tab. 2), which OPR is calculated in first k elements of each S ∈ S.
Symmetrical Discrepancy for symmetry We define the Symmetrical Discrepancy (SD) by:
SD(A) = ∑ i,j,i<j |Ai,j −Aj,i| L× (L− 1)/2
Direction Balance We define the Direction Balance (DB) as the ratio between the sum of the lower (left) triangle and the upper (right) triangle of A. Note that all elements are positive in A, DB(A) in l-offset range is always positive.
DBl(A) = ∑ i,j;i<j,|i−j|<=lAi,j∑ i,j;i>j,|i−j|<=lAi,j
In Tab. 5 and 2, we report DB for a offset range of 20, see Fig. 5b for the performance correlations with other offset ranges.

heading: C MEASURING CORRELATIONS BETWEEN PROPERTIES AND DOWNSTREAM TASKS.
text: In Tab. 5, monotonicity in 20 offsets and the correlations with performance in downstream tasks was reported; here we show how different ranges of the monotonicity correlate to performance in downstream tasks. Among all tasks, we choose all single sentence classification tasks (CoLA and SST-2), two biggest sentence pair classification tasks (MNLI and QQP tasks have more training samples than others), average performance in GLUE, and in SQuAd (F1 metrics nearly have identical trends with EM metrics).
As shown in Fig. 5a, the monotonicity indicators in nearly 20-55 offsets are highest correlated to the performance of span prediction tasks (with Pearson correlation larger than 90%). Note that some classification tasks (especially SST-2) also show opposite correlations comparing to span prediction tasks, probably due to the unshiftable [CLS] on which classification tasks rely for interference does not need monotonicity.
In Fig. 5b, the performance correlates more to the direction balance indicators when considering neighboring tokens. For instance, the direction balance indicators within a small offset has correlation bigger than 0.5, this tends to be smaller with increasing offsets.

heading: D RELATIVE POSITION EMBEDDING WITH LONG OFFSETS
text: The dot product between two position embeddings are shown in Fig. 1(e) and (f). To analyze the behaviour, we replace the raw dot product with the cosine similarity (as the latter is normalized and
thus easier to interpret). When the cosine similarity is one, the two vectors are perfectly colinear and share the same direction. For the purposes of this investigation, we arbitrarily pick 0.95 as a threshold for the cosine similarity, denote that two vectors are not significantly different.
From Fig. 6 we observe the following for all PE variants with fully-learnable RPE: (1) There is no significant difference between relative position vectors with longer than 20-25 offsets; (2) forward relative position vectors are slightly more similar to forward relative position vectors instead of backward relative position vectors, and vice versa (see the central left-lower/right-upper white parts).

heading: E DETAILED EXPERIMENTAL SETTING
text: We train BERT base and BERT medium with both masked language prediction and next sentence prediction tasks; most parameters are listed in Tab. 6, with the remaining parameters set as in the original paper. Note that we share RPE in different heads and layers. Like (Shaw et al., 2018) RPE are truncated from −64 to 64.
We perform five runs for SQuAD and GLUE benchmark. The results in GLUE are for the last checkpoint during fine-tuning while SQuAD takes the best one for every 1000 steps. Finally, we calculate the average over 5 runs. All these settings are the same for all PEs. We use Mismatched MNLI. In GLUE (Wang et al., 2018), the train and dev are somewhat adversarial: training samples (in train and dev) containing the same sentence usually have opposite labels. Models may get worse when it overfits in the train set, resulting in unexpected results. Therefore, we exclude WNLI to calculate average in the last column in Tab. 3. The fine-tuning parameters are using default values in Huggingface project Wolf et al. (2019).

heading: F GENERALIZATION TO LONGER SENTENCES IN DOWNSTREAM TASKS
text: To fairly compare all models, we train a medium setting (8-layer transformer) on 128-length input in the first 10 epochs and 512-length input in the last 2 epochs from scratch. Fig. 7 shows that before 512-length pre-trained (like the 10-th epoch 128-length pre-trained) learnable sinusoidal APEs and RPEs perform better than BERT-style (without sinusoidal parameterization) in both SQuADs. This happens because PEs with translation invariance (learnable sinusoidal APEs and RPEs) generalize into longer positions 12, while position vectors between 128-512 positions are not trained in fullylearnable PEs and they are randomly initialized and finetuned in the downstream.

heading: G THE EVOLUTION OF DOT PRODUCTS BETWEEN POSITION VECTORS
text: We exhibit dot products between position vectors during training a BERT-medium, as shown in Fig. 8. There is seemingly no pattern in the beginning, but as the number of training steps increase, a regular pattern with translation invariance and local monotonicity emerges.
12In practice, the document length of some tasks, like summarization, document-level translation, etc. may be much longer than the maximum length typical BERT models can deal with, i.e., 512. Then, learnable sinusoidal PEs or RPEs would be beneficial. Note that they also save parameters compared to typical BERT models, especially when document length is very long like (Beltagy et al., 2020).

heading: H DISCUSSIONS ON RELATED WORKS
text: Complementary effect between APE and RPE The complementary effect between APE and RPE was demonstrated to be effective in (Wang et al., 2019) for machine translation. In the pretrained language model, Ke et al. (2020) propose that combining APE and RPE could be beneficial for classification tasks (GLUE), which in this paper, this complementary effect is not significant since most PE combinations (APE and RPE) do not outperform the BERT-style fully-learnable APE on classification. Instead, we empirically conclude that most PE combinations boost the performance in span prediction tasks. The benefit in classification tasks in (Ke et al., 2020) may come from other modifications, for example, it unties the [CLS] symbol from other positions. Moreover, in the paper, it adopts a special relative position embedding like (Raffel et al., 2019) (as this paper also suggests to do so): a simplified form of PE that each “embedding” is simply a scalar bias added to the corresponding logits when computing the attention weights. The fundamental difference between the ‘position bias’ and position embedding is unknown from now.
Study on attention visualization. Many works are focusing on understanding attention patterns in individual heads. For example Vig (2019) introduced a tool for visualizing attention in the Transformer at multiple scales; Rogers et al. (2020) suggest attention mechanisms like Vertical, Diagonal, Vertical + diagonal, Block, and Heterogeneous. Clark et al. (2019) found some attention mechanisms like attending broadly, to next, to [CLS] or [SEP], attend to punctuation. While our paper focuses on the general attention introduced by PEs from an average point of view, without considering any specific attention head.
Asymmetry in sequential labeling Yan et al. (2019) suggested asymmetry of position embedding in named-entity recognition task (without involving pre-trained language models) which is a kind of sequential labeling tasks like span prediction (SQuAD) in this paper. Their conclusion is generally compatible with ours, but we question its assumption that ‘the property of distance-awareness disappears when query and key projection are conducted’. As shown in Fig. 9, we could slightly see some distance-awareness by directly taking the average position-position correspondence in the first layer among many heads (i.e., PWQ,1(WK,1)TPT ).
Functional parameterization of PEs Xu et al. (2019) proposes various variants of sinusoidal positional encodings inspired by functional analysis. Wang et al. (2020) proposed a sinusoid-like complex word embedding to encode word order. Both (Xu et al., 2019) and (Wang et al., 2020) assume that PEs should satisfy the translation invariance property, but they induce different types of sinusoidal PE parameterization either in real or complex vector space. Moreover, Liu et al. (2020) use a neural ODE component to parameterize position encoding as a continuous dynamical model, which could learn suitable PEs in neural networks. All of these PEs are inspiring. Since selecting
the suitable parameterization type is not the main concern in this paper, we adopted the typical ones, namely, the fully-learnable, (learnable or fixed) sinusoidal APEs/RPEs. The fundamental difference between these PE parameterizations needs further investigation. More recently, (Wang & Chen, 2020) empirically study the behaviour of many position embeddings and their performance in Transformers for various NLP tasks.

heading: I THE THREE PROPERTIES IN OTHER MODELS
text: By using the proposed identical probing test, we also check the properties of other trained Transformer models with decoder components in Tab. 7 and Fig. 10. The machine translation model 13 is a typical encoder-decoder architecture using multiple-layer Transformers. GPT2 (Radford et al., 2019) adopts a purely decoder architecture; 12-layer base setting is used in this work.
Monotonicity Compared to BERT and the machine translation, GPT2 satisfies monotonicity (especially in the first 20 offsets) better than other models, showing capturing distance between neighboring tokens matters in the language model.
Translation invariance As seen from the translation invariance indicators in Tab. 7, GPT2 satisfies translation invariance poorer than other models, since tokens in it also additionally attend to a few beginning tokens no matter how far the attended tokens are.
Symmetry GPT2 shows the biggest symmetrical discrepancy, since GPT2, which aims to predict the next word, adopts an attention mask of succeeding tokens to avoid information leakage. Plus, the machine translation encoder slightly attends more to the succeeding tokens while BERT attends more on the preceding tokens than succeeding tokens.

heading: J WHITE BAND EFFECTS ALONG THE DIAGONAL
text: In order to analyze the white band effects along the diagonal, we show all results of identical word probing (average attention values in the first layer of identical word probing with respect to 100
13An English-to-French machine translation model downloaded from https://github.com/ pytorch/fairseq/tree/master/examples/translation (Vaswani et al., 2017; Ott et al., 2018; 2019)
(a) Encoder (w/ Decoder): Machine translation (Vaswani et al., 2017) (b) Decoder : Generative language model (GPT2) (Radford et al., 2019) (c) Encoder : Masked language model (BERT) (Devlin et al., 2018)
Figure 10: Identical word probing with different types of trained models.
randomly-selected words). This effect is more clear for fully-learnable RPE, learnable sinusoidal RPE and any combination variants including them (see. Fig. 11 (d,f,g,i,j,l)). To show the obvious differences between these PEs, in this paper, we use average unnormalized attention weights matrix for probing, but all indicators are calculated using normalized attention values for better quantitative comparison.

heading: K THE REPLACEABLE PROPERTY ABOUT ABSOLUTE POSITIONS OF WORDS
text: For example (we do not consider subword tokenization for simplicity), we have two sentences for next sentence predictions (As BERT did)
sentence1 : Deadlines are the No.1 productive forces .
sentence2 : I think , therefore I am .
By adding three special tokens, we will have a example with 17 tokens as
[CLS] Deadlines are the No.1 productive forces . [SEP] I think , therefore I am .[SEP]
with absolute positions in the bracket as
[CLS](1) Deadlines(2) are(3) the(4) No.1(5) productive(6) forces(7) .(8) [SEP](9) I(10) think(11) ,(12) therefore(13) I(14) am(15) .(16) [SEP](17)
Assume that the expected maximum sequence length is 16 (actually 128 or 512 in BERT), we need to randomly remove the first token of the first sentence (i.e., Deadlines ) as
valid sample: I: [CLS](1) are(2) the(3) No.1(4) productive(5) forces(6) .(7) [SEP](8) I(9) think(10) ,(11) therefore(12) I(13) am(14) .(15) [SEP](16)
or last token of the second sentence (i.e., . )
valid sample: II: [CLS](1) Deadlines(2) are(3) the(4) No.1(5) productive(6) forces(7) .(8) [SEP](9) I(10) think(11) ,(12) therefore(13) I(14) am(15) [SEP](16)
Both the above two sentences are valid for training. If we replaced the first sentence with another shorter sentence (i.e., Publish/Launch or Perish ?), the sample would be
valid sample: III: [CLS](1) Publish(2) or(3) Perish(4) ?(5) [SEP](6) I(7) think(8) ,(9) therefore(10) I(11) am(12) [SEP](13) [PAD](14) [PAD](15) [PAD](16)
The three samples I,II,III are valid, but its absolute position indexes are not shiftable. Especially, the first sentence of the second sentence could be 9, 10, and 7, respectively, depending on the random seed for dropping and the length of the first sentence.
(a) fully-learnable APE (b) fixed sinusoidal APE (c) learnable sin. APE
(d) fully-learnable RPE (e) fixed sinusoidal RPE (f) learnable sinusoidal RPE
(g) learnable sinusoidal APE + fullylearnable RPE (h) learnable sinusoidal APE + fixed sinusoidal RPE (i) learnable sinusoidal APE + learnable sinusoidal RPE
(j) BERT-style APE + fullylearnable RPE (k) BERT-style APE + fixed sinusoidal RPE (l) BERT-style APE+ learnable sinusoidal RPE
Figure 11: Identical word probing (models with more PEs are shown here comparing to Fig. 2). Darker in the i-th row and j-th column means that the i-th words generally attend more on the j-th words.

Review summaries:
The paper presents a systematic analysis of approaches used to encode position information in transformers and in particular BERT-based models. The paper investigates absolute and relative position embedding strategies that use either fixed/learnable sinusoidal or fully learnable position embeddings. These embeddings are characterized based on different properties that are either inherent from their formulation or observed empirically such as monotonicity, translation invariance, and symmetry. Interestingly these properties appear to emerge naturally when having learnable parameters in APEs and RPEs.