On Position Embeddings in BERT
Authors: Benyou Wang
Lifeng Shang
Christina Lioma
Xin Jiang
Hao Yang
Qun Liu
Jakob Grue Simonsen
Venue: ICLR 2021 Poster (2021)
Keywords: Position Embedding
BERT
pretrained language model.

Abstract: Various Position Embeddings (PEs) have been proposed in Transformer based architectures~(e.g. BERT) to model word order. These are empirically-driven and perform well, but no formal framework exists to systematically study them. To address this, we present three properties of PEs that capture word distance in vector space:  translation invariance, monotonicity, and  symmetry. These properties formally capture the behaviour of PEs and allow us to reinterpret sinusoidal PEs in a principled way.
Moreover, we propose a new probing test (called `identical word probing') and  mathematical  indicators to quantitatively detect the general  attention patterns with respect to the above properties. An empirical evaluation of seven PEs (and their combinations) for classification (GLUE) and span prediction (SQuAD) shows that: (1) both  classification and span prediction benefit from  translation invariance and local monotonicity, while symmetry slightly decreases performance;
(2) The fully-learnable absolute PE performs better in classification, while relative PEs perform better in span prediction.  We contribute the first formal and quantitative analysis of desiderata for PEs, and  a principled discussion about their correlation to the performance of typical downstream tasks.

Published as a conference paper at ICLR 2021
ON POSITION EMBEDDINGS IN BERT
Benyou Wang
Lifeng Shang
Christina Lioma University of Copenhagen
University of Padova wangddei. unipa.it
Huawei Noah's s Ark Lab Shang.
com
c. ..lioma@di. ku. dk
Xin Jiang
Hao Yang
Qun Liu
Huawei Noah's Ark Lab Ji Jiang. Xin@huawer com
Huawei Technologies Co., Ltd. yanghao300huawx com
Huawei Noah's Ark Lab q iu@hhawe com
Jakob Grue Simonsen University of Copenhagen simonsen& di. ku. dk
ABSTRACT
Various Position Embeddings (PEs) have been proposed in Transformer based ar- chitectures (e.g. BERT) model word order. These empirically-drive driven and perform well, but no formal framework exists to systematically study them. To address this, present three properties of PEs that capture word distance in vec- tor space: translation invariance, monotonicity, and symmetry. These properties formally capture the behaviour PEs and allow in principled way. Moreover, we propose new probing test (called 'identical word probing') and mathematical indicators to quantitatively detect the general attention patterns with respect to the above properties. An empirical evaluation seven PEs (and their combinations) for classification (GLUE) and span predic- tion (SQuAD) shows that: (1) both classification and span prediction benefit from translation invariance and local monotonicity, while symmetry slightly decreases performance; (2) The fully-lcarnabee absolute PE performs better in classification, while relative PEs perform better in span prediction. We contribute the first formal and quantitative analysis of desiderata for PEs, and principled discussion about their correlation to the performance of typical downstream tasks.
reinterpret sinusoidal PEs
INTRODUCTION
Position embeddings (PEs) are crucial in Transformer-based architectures for capturing word or- der; without them, the representation is bag-of- Fully learnable absolute position embed- dings (APEs) were first proposed by Gehring et al. (2017) to capture word position in Convolutional Seq2seq architectures. Sinusoidal functions were also used with Transformers to parameterize PEs in a fixed ad hoc way (Vaswani et al., 2017). Recently, [haw et al. (2018) used relative position embedding (RPEs) with Transformers for machine translation. More recently, in Transformer pre- trained language models, BERT (Devlin et al. 2018 Liu et al. (219)) and GPT (Radford et al. 2018) used fully learnable PEs. Yang et al. (2019) modified RPEs and used them in the XL.Net pre-trained language model. To our knowledge, the fundamental differences hetween the various PEs have not been studied principled way.
We posit that the aim of PEs is to capture the sequential nature of positions in vector space, or technically, to bridge the distances in (for positions) and P (for position vectors). We therefore propose three expected properties for PEs: monotonicity, translation invariance, and symmetry Using these properties, we formally reinterpret existing PEs and show the limitations of sinusoidal
'Informally, positions are originally positive integers, one may expect position vectors in vector space to have the following properties: 1) neighboring positions are embedded closer than faraway ones; 2) distances of arbitrary m-ooffet position vectors are identical; 3) the metric (distance) itself is symmetric.

---

Published as a conference paper at ICLR 2021
PEs (Vaswani et al. 2017): they cannot adaptively meet the monotonicity property thus we propose learnable sinusoidal PEs.
We benchmark 13 PEs (including APEs, RPEs, and their combinations) in GLUE and SQuAD, in a total of 11 individual tasks. Several indicators are devised to quantitatively measure transla- tion invariance, monotonicity, and symmetry, which can be further used to calculate their statistical correlations with empirical performance in downstream tasks. We empirically find that both text classification tasks (in GLUE) and span prediction tasks (SQuAD V1.0 and V 2.0) can benefit from monotonicity (in nearby offset) and translation invariance (in particular without considering spe- cial tokens like [CLS]), ) but symmetry decreases performance since it can not deal with directions between query vectors and key vectors when calculating attentions. Plus, models with unbalanced attention regarding directions (generally attending more to preceding tokens than to succeeding to- kens) slightly correlate with better performance (especially for span prediction tasks).
Experiments also show that the fully-learnable APE performs better in classification, while RPEs perform better in span prediction tasks. This is explained by our proposed properties follows: RPEs perform better in span prediction tasks since they meet better translation invariance, mono- tonicity. and asymmetry; the fully-learnable APE which does not strictly have the translation in- variance and monotonicity properties during parameterizations (as it also performed worse in mea- suring translation invariance and local monotonicity than other APEs and all RPEs) still performs well because it can flexibly deal with special tokens (especially, unshiftable [CLS]).
Regarding the newly- swwly-proposed learnable sinusoidal PEs, the learnable sinusoidal APE satisfies the three properties to a greater extent than other APE variants, and the learnable sinusoidal RPE ex- hibits better direction awareness than other PE variants. Experiments show that BERT with sinu- soidal APES slightly outperforms the fully-learnable APE in span prediction, but underperforms in classification tasks. Both for APEs and RPEs, learning frequencies in sinusoidal PEs appears to be beneficial. Lastly, sinusoidal PEs can be generalized to treat longer documents because they completely satisfy the translation invariance property, while the fully-learnable APE does not.
1) We propose three principled properties for PEs that are either formally examined or empirically evaluated by quantitative indicators in a novel Identical Word Probing test; 2) We benchmark 13 PEs (including APEs, RPEs and their combinations) in GLUE, SQuAD V1.1 and SQuAD V2.0, in a total of 11 individual tasks; 3) we experimentally evaluate how the performance in individual tasks benefits from the above properties; 4) We propose two new PEs to extend sinusoidal PEs to learnable versions for APEs/RPEs.
The contributions of this paper are summarised below:
2 PROPERTIES OF POSITION EMBEDDINGS
Gehring et al. (2017); Vaswani et al. (2117) use absolute word positions as additional features in neural networks. Positions x E N are distributively represented as an embedding of x as an element E E RD in some Euclidean space. By standard methods in representation learning, similarity be- tween embedded objects T and ÿ is typically expressed by an inner product (I", 5), for instance the dot product gives rise to the usual cosine similarity between F and J. Generally, if words appear close to each other in a text (i.e., their positions are nearby), they are more likely to determine the (local) semanties together, than if they occurred far apart. Hence, positional proximity of words and y should result in proximity of their embedded representations H and y. One common way of formalizing this is that an embedding should preserve the order of distances among positions P We denote -(),-) as a function to calculate doseness/proximiig between embedded positions, and any inner product can be a special case of ((-.-) with good properties. We can express preservation of the order of distances as: For every I, y, E N,
J 2| = (( 3) < 2(F,2)
Note that on the underlying space, the property in Eq. T has been studied for almost 60 years (Shepard 1962), in both algorithmics (Bilu & Linial 2005 Badoiu al. 2008 Maehara 2013). and machine learning (Terada & Luxburg 2014
J0inJJin& Jain et al. (016)) under the name ordinal embedding. As we are interested in the simple case of positions from N. Eq. reduces to the following property:
Thhoreeiccal evidence for this is nontrivial unless we assume more about the particular non-linear functions. We empirically find that all learned PEs can preserve the order of distance
2

---

Published as a conference paper at ICLR 2021
Property 1. further apart:
Monotonicity:
The proximity of embedded positions decreases when positions
Ye, m, E = o < ((5
A priori, a position embedding might treat every element individually. However, considering pairs of positions based their relative proximity (rather than the absolute value of the positions),
can lead to simplified and efficient position embeddings (Wang et al. 2020). Such embeddings satisfy translation invariance:
Property 2. variant:
Translation invariance: The proximity of embedded positions are translation in-
VIi,..., m E N: (1 = 6652, m) =... 6(Tn, Tn m)
(3)
Finally, since the inner product is symmetric, we also consider whether o(.,)) is symmetric: Property 3. Symmetry: The proximity of embedded positions is symmetric,
V2,yE 6(Z, Ø7) 6(ū,2)
(4)
There is no generally accepted standard of properties for position embeddings; based prior work as described above, we posit that the above properties are important, and now examine several existing PEs relation these properties, either formally (in Sec. 33 or empirically (in Sec. 4
UNDERSTANDING PES VIA THE PROPERTIES
PEs come in two variants: absolute PDs (APEs) where single positions are mapped to elements of the representation space, and relative PEs (RPEs) where the difference between positions (i.e., x for y E N) is mapped to elements of the embedding space. For Transformer- based architectures, the difference between APEs and RPEs manifests itself in the attention mechanism, in particular how the matrices of query, key; and value weights Wº, WKK and WV used calculate attention in each attention head. Consider two positions v, y E N, let WEg be the word embedding of the word position x, and let Pa and Pa-y be the embeddings of the position æ and relative position æ y. respectively. The query-key-value vector for the word at position æ is typically calculated as below for APEs and RPE4 respectively:
Q~
Wº
TQ= K2 V=
W W W
APE:
K V
= (WE, + Pa)
W W
RPE:
O WE,
+
Pa-w
(S)
Pa
Observe that while the APES calculation linear in (WQ, WK WV) with the word and position embeddings merged into the coefficient, the RPEs calculation is affine, with the relative position embedding Pa-y acting as an offset independent the word embedding WE
In Transformers, the resulting representation is sum of value vectors with weights depending = QK", that is, Attention K, V) = softmax QK") V. In the rest of the paper, we examine PEs in the above architecture with respect the properties introduced in Section 2 In particular, we study four well-known variants of PEs: (1) the fully learnable APE (Gehring et al.)
2017), (2) the fixed sinusoidal APE (Vaswani al.] 2017), (3) the fully learnable RPE (Shaw
et al. 2018), and (4) the fixed sinusoidal RPE (Wei et al. 2019
3.1 UNDERSTANDING SINUSOIDAL PES
With sinusoidal parameterization in PEs, we may use a specific proximity, i.e., an efficient inner product like dot product, to check if the sinusoidal form of PEs meets the above properties. The dot product between any two position vectors is
sin(=12)) cos(w]a)
sin(wiy) coo(w11) v)
sin(w] 2r) sin(w19) cos(wi cos(w19)
Ar.y = (FF Ø) sum
sin(w p x] nos(w Øa
sin(w q v)
sin(w p sin & v) sum
Σ cos(ws (xx)x)x))))))) (6)
cos(w o
cost cos
Thhee are many variants of RPFs (e.g., DDai et al. 2019 As selecting RPEs is not the main concern in this paper, we give the original (and typical) RPHs only.
One can easily extend this work other RPE variants.

---

Published as a conference paper at ICLR 2021
Table 1: Overview of PEs. Ps or P(x) is the z-th absolute/relative position vector (the latter is parameterized by sinusoidal functions). The newly-prov newly-proposed PEs this paper are in bold.
PBs
formulation
parameter scale
Jully Iesmmable APE
Ps 2
(Gelwing
2017
Exed sinasodaal APE
P(a) [.-. sin(w, coolwra)), .P.
Varsaaii or al
2017
w; = (1/10000)
learmblle sinusoidal APE
#; W
P(a) p sin((urs) cos(wra)) -a +1]]
2
Jully learnalle RPE
Pg E 2
1xx
(Shaw et al. 2018
sted simuoocal
P()) = (+-. cin((r cos(wrar)....
My (1/10000)3:/D
lenraalle sinasoidal RPE
P(z) (r sin(ww)) costura))...
Note that sinusoidal PEs satisfy both Property 2 (translation invariance) because the inner product is only associated with its position difference I y, and Property 3 (symmetryy) because the dot product itself is symmetric: (ī, 3) = (T. i³). Note also that checking Property is equivalent to checking monotonicity of the map (m) Ens
2/2 cos(w,m). (mm) is monotone intervals where
its first order derivative </mmm)= = 2/2 w; (w;m) does not change sign, and these intervals depend on the choice of wy. With fixed frequencies w, = (1/10000) it is monotonous when m roughly between 0 and 50, indicating that it can only strictly perceive a maximum distance of 50 and it is insensitive to faraway distances (e.g. longer than 50).
Although sinusoidal PEs with fixed frequencies (i.e., W; = (1/10000)2x/7) 24/)) are common in APEs and RPEs, we argue that learning these frequencies is useful because it can adaptively adjust intervals of monotonicity (they do not have to be 0-50 as in the fixed sinusoidal APE) [ With trainable frequencies, we can adaptively allocate a number of frequencies in a data-driven way. App. A.2 explains the expressive power of sinusoidal PEs with trainable frequencies from the perspective of the Fourier series. Extending existing fixed sinusoidal PEs to learnable version with learnable frequencies gives two variants: a learnable sinusoidal APE and a learnable sinusoidal RPE.
3.2 UNDERSTANDING RPES
RPEs ignore the absolute position of words and directly encode their relative distance. The RPEs expression adheres to the !ranslation invariance property during parameterization, since relative distance with the same offset will be embedded as the same embedding, namely, P2,-91 Pa.- if Φ 1/ = I2 1/2 Plus, RPEs that separately embed forward and backward relative embeddings, i.e., Pj-j + Pj-s, do not meet symmetry during parameterization.
Sinusoidal RPEs can also embed neighboring relative position in close vectors with local mono- tonicity, similarly to sinusoidal APEs. Note that the dot products between two sinusoidal relative position vectors with the same offset, without distinguishing positive negative relative position vec- tors, should be identical This makes hardly perceive of the border between preceding and succeeding relative position vectors.
4 EXAMINING PE PROPERTIES IN PRE-TRAINED LANGUAGE MODEL
We train BERT with six basic PES as in Tab. and their combination variants, and conduct a probing test to check which degree they satisfy the properties.
Pre-training The pre-trained "BERT-base-unanne train by replacing the original absolute PE module with a new PE variant (including APEs and RPEs). We train the new models with sequence length 128 for 5 epochs and then 512 for another 2 epochs. The training is the same as in the original BERT, i.e., BooksCorpus and Wikipedia (16G raw documents) with whole word masking. To be fair, the BERT with the original felly-learnable
checkpoint (Devlin et al.) 2018) used to
*See App. A to intuitively understand specific functions of each frequency wi
NNaeely, (P (Pz.-o:: Pa, Paz-wz) = (Ps--wa, PE., Pra-ua) if (3r1 v1) (x2 1/2) (xz -ya) (x V1) = m,
in both y 0 and y < 0.
4

---

Published as a conference paper at ICLR 2021
23
30
30
80
32
(a) fully-learnable APE (b) fully-learnable APE (c) learnable sin. APE (d) learnable sin. APE
(w/o RPE)
(w/ lealn-lea RPE)
(w/o RPE)
(w/ fully-learnable RPE)
23
(c) felly-learnable RPE
(f) fully-learnable RPE (g) learnable sin. RPE
(h) learnable sin. RPE
(w/o APE)
(w/ leannale APE) (w/o APE)
(w/ fella-leammable APE)
Figure 1: Dot products between absolute position vectors Stop row) and relative position vectors
(bottom row). Darker means the two position vectors are closer.
APE is also further trained in the same way. All models have about 110M parameters corresponding to typical base setting, with minor differences solely depending on the parameterization in Tab.
4.1 DOT PRODUCT BETWEEN POSITION VECTORS
We calculate dot products between two arbitrary position vectors for APEs and RPEs (see Fig. D.. For APES, neighboring position vectors are generally closer compared to faraway ones. This trend is clearer in the learnable sinusoidal APE, which imposes strict sinusoidal regularization for PEs. Note that additionally adopting RPEs does not affect too much PE patterns, as can be seen by b), or Fig. Tc) and d).
APEs
comparing Fig. a) and
RPES In the fully-learnable RPE setting, the vertical and horizontal bright bands in (e) and I show that the relative position vectors for small offsets (e.g., {P-s. Po. ... P3) are notably different to other relative position vectors; it indicates that the relative position vectors with small offsets are more distinguishable than faraway relative position vectors. The four dark corners in [e) and [(f) means that relative position vectors with longer offset than 20, i.e., from -64 to -20 and from 20 64, are very close, showing that the fully-learnable RPE does not significantly distinguish far-distant RPEs. This suggests that truncating RPEs into a fixed distance (c.g. 64 in (Shaw et al., 2018) is reasonable. This effect is further explained in App. D
4.2 IDENTICAL WORD PROBING
In APEs, the attention matrix (A softmax(QK")) is related to individual words and their posi- tions, an element of (inactivated) A in the first layer is given by:
ag = (w, + Pp) ((wy + K.)
w. (w ( + w. w. (w + p.w (w + p. (w
word-word corresondence
word-positin corresponence
wond-gesstion correspondence
positio-possiion corresponeene
Identical word probing for PEs
To study the effect of only PEs in A without considering indi- vidual words, we use identical word probing: feed many repeated identical words (can be arbitrary,
'In all figures we only show the first 128 positions instead of 512 positions since they are in principle compatible. Practically in BERT, there is minor discrepancy hetween the first 128 positions and the remaining positions due the typical training strategy (first training on 128-Iength input and then 512-length input).
5

---

Published as a conference paper at ICLR 2021
20
20
20
180
120
20
120
180
120
-22
180
120
(a) BERT without PE
(b) Jealyllearn APE
(c) learnable sin APE
(d) learnale RPE
24-
33-
01
100
13
170
23
120
(c) fixed sin. RPE
(f) Icarnable sin. RPE
(g) learnable sin. APE + (h) learnable sin.
APE
fully-learnable RPE learnable sin. RPE
Figure 2: Identical word probing, Darker in the i-th row and j-th column means that the i-th words generally attend more on the j-th words.
denoted as W) as a sentence to BERT to check the attention values A with each element ر (0) (00 p. ((z + K..
(8)
As we take average of A over many maudomly-selectte words w. the general patterss of A will not be affected by any particular word. Namely, A()) is word-free and only related to learned PEs. Thus, A(1) can be treated as a general attention bias and can also implicitly convey position-
wise proximity in Transformers. Note that the probing test could also be applied to RPEs.
4.2.1 1 QUALITATIVE ANALYSIS
Fig. 2 shows the average attention weights among all heads in the first layer. BERT without PE nearly treats all words uniformly (bag-of-words) Almost all APES and RPEs have a clear pattern of translation invariance, local monotonicity in neighboring window, and symmetry. Note that this is nontrivial since no specific constraints or priors were imposed on fella-le APEs/RPEs
BERT with APEs does not show any direction awareness since Fig. ((b) and metrical. As seen from Fig. [f..h). BERT with learnable sinusoidal RPE generally attends more on forward tokens than backward tokens, which cannot be clearly found in fully-learnable RPE and fixed sinusoidal RPE. Interestingly, the white bands along the diagonal in Fig.
and (c) are nearly sym-
(d, f, g) suggest that
some words generally do not attend themselves, as previously observed in (Clark et al. 2019
4.2.2 QUANTITATIVE ANALYSIS
Using the activated attention values A in Eq. 8 we adopt three quantitative indicators to measure to which extent BERT models with individual PEs satisfy the three properties and their derivative indicators (see App- B for details of calculating these indicators) in Tab. 2 Basically, all APES and RPEs satisfy monotonicity in small offsets and translation invariance compared to BERT without PE; All PEs nearly satisfy symmetry except for the learnable sinusoidal RPE and its combinations.
APEs and RPEs The learnable sinusoidal APE better satisfies all three properties than fully learn- able APE and fixed sinusoidal APE; this is due to its sinusoidal parameterization and flexible fre- quencies. RPEs satisfy translation invariance to a higher degree than APEs, because they directly
See App. G for an example of the evolution of PE patterns in fully learnable APE which starts from random initialization and ends with some patterns reflecting the properties.
SSee App. | for more details about the white band effect along the diagonal. 9300 random-selectedd words were used to calculate average A1)), as we empirically found that adopting more words almost does not change (1))

---

Published as a conference paper at ICLR 2021
Table 2: Quantitative measurement of the properties (monotonicity, translation invariance, symme- try, and direction balance D For these property indicators, the smaller the number, the better the property is met. 0 denotes that the property is ideally satisfied. Direction balance denotes the ra- tio between the sum of attention values for forward attending and backward attending. means it is fully-balanced in directions. We have indicated the numbers that most closely correspond to satisfaction properties and direction balance for each group in bold.
PE:
all offsets
monotonicity
first otsets
Inandationn invariance
[C1B]
(C1S]
symmerry
dinection balunce
BERT without PE
0.5430
0,1593
0,0005
1.0136
BERI-stle APE
0.2461
0.01205
0,5030
0.01443
0.0012
1,1940
lised sin, APE
0.1937
0,0190
0.2552
0,2143
0.0010
1/0266
leamable sin. APE
0.1936
0.0237
0,0653
0.0578
0,0004
1,0281
felly-leamable RPF
0.1576
0,0048
0.1178
0.0007
0.0007
1,1930
Risdd sin. RPF
0.1273
0.0054
0,0924
0,0020
0.0007
1,1565
lesmable sin. RPE
0.3157
0,0057
0.1397
0.0038
0,0014
1,3223
BLRT--ylC style
faull--kuaaaaa Joumnable RFL
0,0071
1.197L
BEIKT-stle APE fixed sin. RPE
0.1579
0.0143
0.1376
0.0072
0,0007
1.1302
BERII-style APEe losmaall sin, RPE
0.2364
0.0158
0.2534
0,D0SS
0.0014
1,5804
leanahll sin, APF fully- fully--kammible RPF
0.0065
0.0238
1,1196
leuraalle sin. APE fixed kin, RPE learsable sin, APE + learnable sin. RPF
0.0746
0.0040
0.0243
0,0168
0.0007
1,0733
0.1796
0.0052
0.0399
0.0252
0.0027
1.5722
satisfy translation invariance during parameterization. In the last column, direction balance values of all PEs except for the fixed sin. APE are larger than one, which indicates that BERT models with all PEs generally attend more to preceding tokens than succeeding tokens, and this phenomenon appears to be stronger in learnable sinusoidal RPEs than others.
The fully learnable APE and [CLS] Fully learnable APE generally performs worse in transla- tion invariance (see the 4-th column) as it has to deal with the unshiftable [CLS] J which is always in the first position. Without considering [CLS] and [SEP] ] (see the 5-th column), the fully learnable APE satisfies translation invariance better than other APEs, showing that the fully learnable APE can flexibly deal with both special tokens and normal positions. The Tulfllly learnable APE also could handle the mismatch between special tokens and normal positions in the monotonicity property.
PES IN DOWNSTREAM TASKS
We empirically compare the performance of PEs in classification and span prediction tasks.
Fine-tuning The fine-tuning on GLUE and SQuAD is the same as in the Huggingface website as per Wolf et al. (2019), see App. E for details. We report the average values of five runs per dataset. For classification, we use the GLUE (Wang et a (0018) benchmark, which includes datasets for both single document classification and sentence pair classification. For span prediction, we use the SQuAD V1.1 and V2.0 datasets consisting of 100k crowdsourced question/answer pairs (Rajpurkar et al. 2016). Given a question and a passage from Wikipedia containing the answer, the task is to predict the answer text span in the passage. In V2.0, it is possible that no short answer exists in the passage since it additionally has 50,000 unanswerable questions written adversarially by crowdworkers (Rajpurkar al., 2018).
5.1 EXPERIMENTAL RESULTS FOR DOWNSTREAM TASKS
GLUE Tab. shows that the fully-learnable APE (a.k.a, BERT-style APE) performs well in GLUE. No PE variants, especially BERT with solely APEs or RPEs, notably outperform the fully- learnable APE. BRRT models with a combination of an APE and an RPE do not always boost the performance of the model with solely the APE or RPE.
SQUAD Tab. 4 shows that nearly all BERT models with RPEs significantly outperform the fully learnable APE. The learnable sinusoidal APE is slightly better than the fully learnable APE in most cases. Both the best-performed models in SQuAD VI.I and V2.0 adopt the fully-learnable RPE.
10 mmmnotonicctyy" (second column) refers to monotonicity calculating all relative distance, while 'mono- tonicity (first 20 offsets)) (third column) refers to monotonicity calculating within a relative distance of 20 (see App./Coor monotonicity in other offsets.): ): the later matters since neighboring words in small window are cru- cial in natural language. For translation invariance, also adopt new indicator without considering special tokens (ICUS] and [SKP]) measure purely position-aware translation invariance.
7

---

Published as a conference paper at ICLR 2021
Table 3: Experiments on GLUE. The evaluation metrics are following the official GLUE benchmark Wang et 2018). The best performance of each task is bold.
single senience
sentenic pair
PEs
CoLA
SST-2
MNLI
MRPC
QNLI
OOP
STS-B
WNLI BCC
acc
spear, cor.
macan
BERT
86.5
86.2
87.4
Tuily learnbll (BeKi-syyR) APE
00.2
93.0
54.35
89.4
88.7
87.8
65.11
38.6
37,5
82-2 :
Bend sin. APF
57.1
92.6
84.3
89.0
88.1
87.5
58.4
86.9
45.1
80.5 + 0.71
leanaale APE
92.8
87.0
fully-learable RPE
92.6
84.9
90.5
6018
38.6
50.4
81.7
Bied sin. RPF
60.4
92.2
84.8
89.5
88.8
83.0
52.9
88.1
45.1
81 0.53
lcantable sin. RPE
00.3
92.6
85.2
90.3
89.1
S8.1
63.5
88.3
49.9
82.2 0.40
fuily leuriblle APE fully- fuily learmable APE fixed sin. RPE
92.8
59.2
92.4
84.8
89.9
88.8
87.9
51.0
88.3
48.2
81.5 0.20
fully learmable APE- laaaable sin. RPE
61.1
92.8
85.2
90.5
89.5
87.9
65.1
88.2
49.6
82.5 1: 0.44
leamable APE
Jeamaabl RPE
88.0
0.44
leumable sin. APE fixed
81.5
87.4
leumable sin. APE + Jearmable sin. RPE
57.7
92.7
85.0
89.6
88.7
87.8
62.3
87.5
50.1
81.4 0.33
Table 4: Performance (average and standard deviation in 5 runs) on dev of SQuAD V1.1 1 and V2.0.
i indicates stat. significance over fully learnable APEs using a two-sided test with p-value 0.05.
SQaAD VLI
SQLAD V2.0
BM
PE
36.47
Tully learnable (BERI-syle) APE
89.44 +: 0.08
8192 = 0.11
76.43 + 0.33
73.07 + 0.63
fixed sin, APE
89.45 + 0.07
$1.93 + 0.11
76.12 + 0.48
72.75 + 0.55
learmable sin, APE
89.65 0.1.1
80.24
0.17
77.24
0.43
73.93
0.44
fully leamable RPE
90.50
83.38
0.11
79.85
0.27
76.68
0.49
fixed sin, RPE
90.30 1: 0.07
83.24
0.08
78,761
+ 0.29
75.381
= 0.28
90,45
79.60
0.37
76.14
fully learnable APE leanalle RPE
90.57
+ 0.04
0,10
80.31
+ 0.10
76.94
fully learnable APE fixed sin. RPE
90.24 0.07
83.06
0.21
78.74
+
75.401
= 0.52
fully learnable APE+ learnable
77,62
74
lenmaalle sin. APE fully-leartable RPE
91.72
0.13
836
0.27
80.24
0.35
76.98
+ 0.34
lesenablc APF
90.36
+
0.10
78.811
0.33
75 0.28
leamable sin. APE leamable RPE
90,49"
1 0.14
83,59
0.14
79.93
0.34
76.691 0.39
As demonstrated in Tab. 2 the fully learnable APE can flexibly deal with [CLS] ] and translation invariance in normal positions, thus it performs well in classification tasks (GLUE) which heavily relies on the unshiftable [CLS] token for inference. Span prediction tasks which do not infer from [CLS] can benefit from strict translation invariance during parameterization (e.g., sinusoidal APEs and RPEs), see Tab. 5 in Sec./6.TTa the correlations between performance of SQuAD and the trans- lation invariance property. Removing PEs (BERT without PE) dramatically decreases performance in SQuAD V1.1 and V2.0, and slightly harms performance on GLUE, showing that PEs are more important in SQuAD than GLUE.
Learnable sinusoidal PFs The sinusoidal APEs outperform fully-learnable APE in span predic- tion but underperform it in classification tasks. The learnable sinusoidal APE/RPE outperforms fixed sinusoidal APE/RPE in GLUE and SQuADs, showing the expressive power of flexible frequencies.
Complementarity of APEs and RPEs In SQuAD, jointly adopting APEs and RPEs can slightly boost performance in some cases. For instance, BERT with learnable sinusoidal APE + APE + fully RPE achieves the best EM score in both SQuADs. However, this complementary effect is relatively weaker in GLUE, where the fully-leamable APE performs strongly.
6 DISCUSSIONS ON PES
6.1 HOW DO THE PROPERTIES CORRELATE INDIVIDUAL TASKS?
We conduct correlation analysis between the properties and the performance on individual tasks as shown Tab. 5 The results show that violating monotonicity in relatively-small offsets (e.g..
20) and translation invariance is harmful since it is negatively correlated to the performance
Pearson correlations are calculated between the property indicators and the performance of each individual task for 12 PFs. BERT without PE was not considered, since its property indicators are significantly different with other PEs and its performance is much worse; it therefore unexpectedly increases correlation values.
8

---

Published as a conference paper at ICLR 2021
Table 5: Pearson correlations between the properties and evaluated tasks, evaluating on BERT mod- els with 13 position embeddings. The positive (negative) numbers denote to which degree the per- formance of the task positively (negatively) correlate(s) to violating the property. This shows that violating local monotonicity and translation invariance is harmful, while violating symmetry (and Irrectionbbllncee is beneficial. Best correlation values are in bold for each row.
Properties
CoLA
MNLI
QQP
GLUE
SQUAD V1.1
SQUAD
all officess
0.44
0.56
0.48
-0.27
monototicity
offiets
-0.18
0.4
-0.4
-042
4.21
-0.91
-0.86
trandaation invariance
W [CLS]]]
(CLS [CPP)
-0.67
0.48
0.32
0.01
-0.69
0.01
-0.68
-307
-061
0.42
-0.63
-0.51
-0.58
+0.57
symmetry
0.17
0.40
0331
0:15
0.16
direction balance
0.32
0.16
0.63
0.35
0.48
0.32
0.37
GLUE and SQuAD. However, violating symmelry (and direction-bulance) is slightly beneficial. This shows that many tasks require BERT models distinguish preceding and succeeding tokens, especially to attend more on preceding tokens. See Fig. [56iin App- [ the correlations between the direction balance indicators and the performance of downstream tasks will be much higher when only considering few neighboring tokens for calculating the indicator.
6.2 MORE DISCUSSIONS ON THE PROPOSED PROPERTIES
Monotonicity Monotonicity holds locally in small neighboring window (usually in 5-20 offsets) for all PE variants, see Fig 2 This shows that BERT models generally not sensitive to longer- distance attendance patterns, also evidenced by the fact that performance in downstream tasks corre- lates more highly with monotonicity in middle-distance offsets (e.g.. g.. 20 in the second row of Tab. 5
than longer offsets (see App. C To check monotonicity guided by learned frequencies of learnable sinusoidal APEs in individual tasks, see App. A.3
Translation invariance In BERT, we argue that absolute positions of words are uninformative since (1) absolute positions of the second segment depend on the length of the first sentence; (2) words are randomly truncated in the beginning or end if a sentence exceeds the expected maximum length, which may shift absolute positions of all tokens with unexpected offset (Devlin et al.) 2018). That is, absolute positions of words in pre-trained language models are arbitrarily replace- able, and thus adopting translation invariance is generally reasonable. Models with striet Translation invariance (all RPEs and sinusoidal APEs) naturally make PEs generalize to longer documents than the documents used in the pre-training phase, see App. F for some empirical evidence.
Symmetry APEs (especially sinusoidal APEs) express symmetry patterns without distinguishing the direction as shown in Fig 2 Ennearlyttrasffrmmn query vectors and key vectors. This limits its performance in ftrection-sensitiv downstream tasks. RPEs could behave better on direction perception, since forward and backward relative embeddings are separately embedded (see Tab. [ Especially, learnable sinusoidal RPE or combination variants including it have more unbalanced attending patterns (see the last column in Tab. 2 as shown in Fig. (1) and (h),
As seen from Eq. 7 it is nontrivial to model directions in two
7 CONCLUSION
To theoretically and empirically understand position embeddings (PEs), we have defined three prop- erties (translation invariance. monotonicity, and symmetry) inspired by distance mappings between the original domain of positions in N and their PEs in RD. A probing test has been proposed to quantitatively examine these properties using appropriate mathematical indicators. Our probing test has shown that these PEs nearly satisfy most properties even when they are fully-learnable without constraints. Experimental results have shown that violating local monotonicity and translation in- variance decreases performance in downstream tasks (classification and span prediction tasks), and that violating symmetry benefits downstream tasks because of direction awareness. We also find that the lealaelea absolute PE in general results in better performance for classification, and that relative PEs result in better performance for span prediction tasks, which can be explained by the connections between their properties and task characteristics.

---

Published as a conference paper at ICLR 2021
ACKNOWLEDGMENTS
The work supported by the Quantum Access and Retrieval Theory (QUARTZ) project, which has received funding from the European Union's 's Horizon 2020 research and innovation programme under the Marie Sktodowska-Cun grant agreement No. 721321.
REFERENCES
George Arfken and Hans J Weber. Mathematical methods for physicists, 1999.
Mihai Badoiu, Erik D. Demaine, MohammadTaghhi Hajiaghayi, Anastasios Sidiropoulos, and Morteza Zadimoghaddam. Ordinal embedding: Approximation algorithms and dimensionality reduction. In Ashish Goel, Klaus Jansen, José D. P. Rolim, and Ronitt Rubinfeld (eds.), Ap- proximation, Randomization and Combinatorial Optimization. Algorithms and Techniques, 11th International Workshop. APPROX 2008, and 12th International Workshop, RANDOM 2008, vol- ume 5171 of Lecture Notes in Computer Science, pp. 21-34. Springer, 2008
Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020.
Yonatan Bilu and Nathan Linial. Monotone maps, sphericity and bounded second eigenvalue. J. Comb. Theory, Ser. B, 95(2):283-299, 2005.
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning. What does bert look at? an analysis of bert's attention. arXiv preprint arXiv:1906.0445 04341, 2019.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdi- nov. Transformer-xl: Attentive language models beyond fixed-length context. arXiv preprint arXiv: 028660, 2019.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04800 2018.
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. Convolutional sequence sequence learning. arXiv preprint arXiv: 1705.03122, 2017.
Lalit Jain, Kevin G. Jamieson, and Robert D. Nowak. Finite sample prediction and recovery bounds for ordinal embedding. In Daniel D. Lee, Masashi Sugiyama. Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 29: An- nual Conference Neural Information Processing Systems, pp. 2703-2711, 2016.
Guolin Ke, Di He, and Tie- Liu. Rethinking positional encoding in language pre-training. arXiv preprint arXiv:2006. 15522000..555.... 2020.
Xuanqing Liu, Hsiang-Fu Yu, Inderjit Dhillon, and Cho-Jui Hsieh. Learning to encode position for transformer with continuous dynamical model. arXiv preprint arXiv:2003.09229 2020.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprini arXiv: 1692, 2019.
Hiroshi Maehara. Euclidean embeddings of finite metric spaces. Discrete Mathematics, 313(23):
2848 - 2856, 2013.
Myle Out, Sergey Edunov, David Grangier, and Michael Auli. Scaling neural machine translation. arXiv preprint arXiv: 1 1806.0017, 2018
Myle Ott. Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations, 2019.
Alec Radford, Kanthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing by generative pre-training, 2018.
10

---

Published as a conference paper at ICLR 2021
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv: 1910. 10683, 2019.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprini arXiv: 05250, 2016.
Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't 't know: Unanswerable questions for squad. arXiv preprint arXiv: 03822,, 2018.
Anna Rogers, Olga Kovaleva, and Anna Rumshisky, A primer in bertology: What we know about how bert works. arXiv preprint arXiv:2002.12327, 2020.
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attentics with relative position representa- tions. arXiv preprint arXiv:1803.0213: 2018.
Roger N Shepard. The analysis of proximities: Multidimensional scaling with an unknown distance function. Psychometrika, 27(2):125-1444 1962.
Yoshikazu Terada and Ulrike Luxburg. Local ordinal embedding. volume 32 of Proceedings of Machine Learning Research, pp. 847-855. PMLR, 2014.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Jesse Vig. A multiscale visualization attention in the transformer model. arXiv preprint arXiv:1906.05774. 2019.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman.
Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arxiv:1804.07445 2018.
Benyou Wang, Donghao Zhao, Christina Lioma, Qiuchi Li, Peng Zhang, and Jakob Grue Simonsen. Encoding word order in complex embeddings. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
Xing Wang, Zhaopeng Tu, Longyue Wang, and Shuming Shi. Self-attention with structural position representations. arXiv preprint 1909 (00,03399,,0383, 2019.
Yu-An Wang and Yun-Nung Chen. What do position embeddings learn? an empirical study of pre- trained language model positional encoding. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 2840-6849, 2020.
Junqiu Wei, Xiaozhe Ren, Xiaoguang Li, Wenyong Huang, Yi Liao, Yasheng Wang, Jiashu Lin, Xin Jiang, Xiao Chen, and Qun Liu. Nezha: Neural contextualized representation for chinese language understanding. arXiv preprint arXiv: /909 0000000 2019.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowiez, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Huggingface' 's transformers: State- of-the-art natural language processing. ArXiv, abs/ 1b0.//99.. 2019.
Da Xu, Chuanwei Ruan, Evren Korpeoglu, Sushant Kumar, and Kannan Achan. Self-attention with functional time representation learning. In Advances in Neural Information Processing Systems, pp. 15915-15925, 2019.
Hang Yan, Bocao Deng, Xiaonan Li, and Xipeng Qiu. Tener: Adapting transformer encoder for name entity recognition. arXiv preprint 19Xiv:1911.04444 2019.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in neural information processing systems, pp. 5753-5763, 2019.
11

---

Published as a conference paper at ICLR 2021
A UNDERSTANDING FREQUENCIES
A.1 UNDERSTAND INDIVIDUAL FREQUENCIES
We argue in this paper that learning schema for such frequencies will be useful in sense it could adaptively adjust frequencies to meet different functions, see Fig. 3
350
300
250
200
w
100
100
200
300
400
500
(a) Examples of some cosine functions
(b) ((m). a sum of cosine functions with frequencies wi (1/10000)2/p,
Figure 3: d(m) in (b) is sum of many cosine functions of individual frequencies with increasing m, which determines the closeness between arbitrary two m-distance position vectors. As shown in (a), each frequency could play different roles: 1) the extremely small frequencies have few effects on the overall word representation (WE, + P2) in Eq. 5 since it makes such position embedding being almost identical with increasing positions; 2) some smaller frequencies can be beneficial to guarantee Property if wi < 11, 3) some bigger frequencies would promote the locally attending mechanism since such cos functions in Eq. [odrop dramatically in the beginning if w; is great enough; 4) Some big frequencies which wi Il would be smooth factors for the overall pattern since it would be randomly impose a bias to all positions.
T,
A.2 EXPRESSIVE POWER OF LEARNABLE SINUSOIDAL PES
In Transformers, linear transformation is commonly-used for example query, key, and value trans- formations on word representations. Let r; be the word representation paramertezied by the sum of word embeddings and position embeddings (like the learnable sinusoidal APEs). Then, each element in ri
ee, + sin(wyt), (e) = e,, (P)(t)
if l is even
(9)
e,, + cos(we=1 f),
if l is odd
After a linear transformation parameterized by w (e.g., the key transformation WK in the first Transformer layer), is linearly transformed as h;(t) wr, (h,(t) can be one of query/key/value vectors Qz, K2, V₂ in t-th position) with each element
D/2
(t)x = W.KKEk +
Σ (10 sin(wxx)) + 21,24+1 cos cos(w22+1t))
(10)
The RHS is a typical Fourier series with a base term Σ E and Fourier coefficients (w.22,, W1,28+1)- It is customarily assumed in physics and signal processing (Arfken & Weber 1999) that the RHS in Eq. 10 with infinite D and appropriate frequencies could approximate any continuous function
given interval.
As using an infinite is not practical, dynamic allocation of a limited number frequencies in a data-driven way could be beneficial for general approximation. The predefined frequencies w;
12

---

Published as a conference paper at ICLR 2021
10-
REAT
SQUAD1
SQUAD1
SQUAD2
350
SQUAD2
CoLA
551-2
MNU
MNU
0.6
default
defaul
0.2
20
40
60
100
120
(a) The learned frequencies of pre-trained BERT and (b) Dot products between two absolute positions with fine-tuned BERT in different downstream tasks. The increasing offset, indicates neighboring APES predefined frequencies in (Vaswani al. (017) (i.e., bedded together. axe refers to offset between posi- tions.
Wi (1/100 (1/10000) 2:/D is denoted as "default"
Figure 4: The learned frequencies in learnable sinusoidal APE in the pre-trained language model and downstream tasks
(1/10000)2*/) 2:/D in the Transformer (Vaswani et al. 2017) can be considered as a special case when it enumerates various frequencies ranging from 1/ 10000 to under specific distribution.
A.3 LEARNED FREQUENCIES OF LEARNABLE SINUSOIDAL APE
The learned frequencies are shown in Fig. 4a Observe that the learned frequencies are generally smaller than the pre-defined ones from Vaswani et al. 2017) (i.e., = (1/ (1/10000)2//)) The learned frequencies are close to the learned one since we use w; (1 (1/10000) 2i/D initialization.
As shown Fig. 4bj the patterns of dot products between positions for learnable frequencies are quite different to the predefined ones (denoted as 'default' in the figure); indeed, the former appears more predisposed deeming remote positions similar. Moreover, fine-tuned models for span pre- diction tasks (including SQuAD and SQuAD2) satisfy strict monotonicity in larger windows than for classification tasks. Observe also that the patterns in pre-training language models seem more similar to those in classification tasks than span prediction tasks.
B QUANTITATIVELY MEASURING THE PROPERTIES.
To quantitatively measure the primary properties treat in this paper, propose multiple criteria, described below.
Assume position-wise attention matrix A (denoted A since there is no risk for confusion), in which each element is the (softmax) activated attention value from the i-th query token to j-th key token (all elements are positive).
Average In-group Variance (AIV) for translation invariance Let be offset between two positions; we denote by r(I) the set of L-offset attention values {At,,j - l); for example, r(1) A2,12,A2,33 ) Translation invariance requires that all elements in each group T(1) should be identical, the smaller variance each r(I) has, it closer to translation invariance. The Average In-group Variance (AIV) is defined weighted average in in-group variances of all (-1 namely:
AIV(A) Σ1 1.+1 var (T(1) IT())
Σ 1+1 [T))
where | is the number of elements in the set. For normalization, this metric is further divided by the overall variance (i.e., var( (A)).
13

---

Published as a conference paper at ICLR 2021
Ordered Pair Ratio (OPR) for monotonicity For word in i-th position, based the increasing distance to the i-th position, there a forward attention sequence Si, {Ar,,, A1,441.... AA and backward attention sequence S denoted as = {, S1,- S2,+, - SL,+, St,-). The ideal (decreasing) monotonicity requires that each (an element in $) is totally ordered 80 Ordered Ratio of S by:
A..- .A.11:: This results in 2L sequences
8L-1- We define the
OPR(S) E Eej,ues,iis sign ((8, 8,)(i - 1))
-
O, and sign(x) 0 otherwise. Ideally, the OPR of totally ordered decreasing (increasing) sequence S should he zero (one). The expected OPR of randomly-ordered sequence (average OPR of the set of all such sequences) should 0.5.
We define sign(z)
1 if
Finally, we get a weighted sum of OPRs of all sequences in S.
OPR(A) Eses OPR (S)
Eses [S|
In the paper, we also consider version of monotonicity within offset of (e.g.. 'monotonicity (first 20 offsets)' in Tab. 2 which OPR is calculated in first k elements of each S E S.
Symmetrical Discrepancy for symmetry We define the Symmetrical Discrepancy (SD) by: SD(A) = Eij.es A Agal Lx (L- ((L--1) 1)/2
Direction Balance We define the Direction Balance (DB) as the ratio between the sum of the lower (left) triangle and the upper (right) triangle of A. Note that all elements are positive in A, DB(A) I-offset range is always positive.
Ei Ais
(B
Ei Arj
In Tab. 5 and 2 we report DB for a offset range of 20, see Fig. 5b for the performance correlations with other offset ranges.
C MEASURING CORRELATIONS BETWEEN PROPERTIES AND DOWNSTREAM TASKS.
In Tab. 5 monotonicity in 20 offsets and the correlations with performance in downstream tasks was reported; here we show how different ranges of the monotonicity correlate to performance in down- stream tasks. Among all tasks, choose all single sentence classification tasks (CoLA and SST-2), two biggest sentence pair classification tasks (MNLI and QQP tasks have more training samples than others), average performance in GLUE, and in SQuAd (Fl metrics nearly have identical trends with EM metrics).
As shown in Fig. 5a the monotonicity indicators in nearly 20-55 offsets are highest correlated the performance of span prediction tasks (with Pearson correlation larger than 90%). Note that some classification tasks (especially SST-2) also show opposite correlations comparing to span prediction tasks, probably due to the unshiftable [CLS] on which classification tasks rely for interference does not need monotonicity.
In Fig. 5b the performance correlates more to the direction balance indicators when considering neighboring tokens. For instance, the direction balance indicators within small offset has correla- tion bigger than 0.5, this tends to be smaller with increasing offsets.
D RELATIVE POSITION EMBEDDING WITH LONG OFFSETS
The dot product between two position embeddings are shown in Fig. Te) and (f). To analyze the behaviour, replace the dot product with the cosine similarity (as the latter normalized and
14

---

Published as a conference paper at ICLR 2021
COLA
351-2
MNU
DOP
02
QUUE
GLUE
SQUADIIEM SQUADZIEMI
0.0
SQUAD2(EM)
0.5
-0.6
0.4
-0.8
-1.0
20 40 60 100 120
300
(a) Pearson correlations between the performance of (b) Pearson correlations between performance downstream tasks (shown in Tables B and 4 and downstream tasks (shown in Tad and direc- monotonicity indicators different offset ranges. tion balance indicators in different ranges. This shows This shows violating monotonicity (especially in 15- the more attending preceding tokens than succeed- 60 offset range) is harmful for most tasks.
ing tokens (especially for neighboring tokens) usually leads to better performance for GLUE in SQuAD.
Figure 5: In which offset ranges the properties correlate with the performance in downstream tasks.
-00
-60
-60
-40
-00
-10
20
20
20
-00
40
-20
20
40
60
-40
-20
20
40
10
40
(a) fully-lcarnable RPE
(b) BERT-style APE fully-learnable RPE
(c) learnable sinusoidal APE + mully-lcamable RPE
Figure 6: Cosine similarities between any two relative position vectors. Cosine similarities bigger than 95% in blue.
thus easier to interpret). When the cosine similarity is one, the two vectors are perfectly colinear and share the same direction. For the purposes of this investigation, we arbitrarily pick 0.95 threshold for the cosine similarity, denote that two vectors are not significantly different.
From Fig. no significant difference between relative position vectors with longer than 20-25 offsets; (2) for- ward relative position vectors are slightly more similar to forward relative position vectors instead of backward relative position vectors, and vice versa (see the central lef-llowerrrihhtuupee white parts).
observe the following for all PE variants with fully-learnable RPE: (1) There is
DETAILED EXPERIMENTAL SETTING
We train BERT base and BERT medium with both masked language prediction and next sentence prediction tasks; most parameters listed in Tab. [6] with the remaining parameters in the original paper. Note that we share RPE in different heads and layers. Like (Shaw et al.] 2018) RPE truncated from -64 to 64.
15

---

Published as a conference paper at ICLR 2021
Table 6: Detailed Experimental Settings
Training
pre-training from scratch
max Length
spoch
learning rate
batch size
BERT-base 128 length
X
128
5o-5
64
BERT-base on 512 length
512
Se-5
512
BERT-mediue 128 length
128
5e.5
128
BERT-medium on 512 longth
512
2
5o-5
512
GLUE SQuAD
128
384
3e-5
32
We perform five runs for SQuAD and GLUE benchmark. The results GLUE are for the last checkpoint during fine-tuning while SQuAD takes the best one for every 1000 steps. Finally, calculate the average over 5 runs. All these settings are the same for all PEs. We use Mismatched MNLI. In GLUE (Wang et al., 2018), the train and dev somewhat adversarial: training samples (in train and dev) containing the same sentence usually have opposite labels. Models may get worse when overfits in the train set, resulting in unexpected results. Therefore, exclude WNLI to calculate average in the last column in Tab. The fine-tuning parameters are using default values Huggingface project Wolf al. (2019).
F GENERALIZATION TO LONGER SENTENCES IN DOWNSTREAM TASKS
To fairly compare models, we train medium setting (8-layer transformer) on 128-length input in the first 10 epochs and 512-length input in the last 2 epochs from scratch. Fig. 7 shows that before 512-length pre-trained (like the 10-th epoch 128-length pre-trained) learnable sinusoidal APEs and RPEs perform better than BERT-style (without sinusoidal parameterization) in both SQuADs. This happens because PEs with translation invariance (learnable sinusoidal APEs and RPEs) generalize into longer positions while position vectors between 128-512 positions are not trained in fully- learnable PEs and they are randomly initialized and finetuned in the downstream.
00-
40-
fully leammble APE leamable fully-leamabic RPE
lelymalaeee APe
leamable
full-llarmalle
30
10
(a) SQuAD V1.1
(b) SQuAD V2.0
Figure 7: Experimental results on SQuADs with BERT-medium. X-axis: epoch number (first trained on 128-length seq. with 10 epochs and then 512-length with epochs). Y-axis: score.
G THE EVOLUTION OF DOT PRODUCTS BETWEEN POSITION VECTORS
We exhibit dot products between position vectors during training a BERT-medium as shown in Fig. There is seemingly no pattern in the beginning. but as the number of training steps increase, regular pattern with translation invariance and local monotonicity emerges.
In practice, the document length of some tasks, like summarization, document-level translation, etc. may be much longer than the maximum length typical BERT models can deal with. i.c., 512. Then. learnable sinusoidal PEs or RPEs would be beneficial. Note that they also save parameters compared to typical BERT models, especially when document length very long like (Beltagy al. 2020
16

---

Published as a conference paper at ICLR 2021
Figure 8: Dot products between absolute position vectors evolving with training steps.
H DISCUSSIONS ON RELATED WORKS
Complementary effect between APE and RPE The complementary effect between APE and RPE was demonstrated to be effective in (Wang et al. 2019) for machine translation. In the pre- trained language model, Ke et al.] (2020) propose that combining APE and RPE could be beneficial for classification tasks (GLUE), which in this paper, this complementary effect not significant since most PE combinations (APE and RPE) do not outperform the BERT-style fully-Icarnable APE on classification. Instead, we empirically conclude that most PE combinations boost the perfor- mance in span prediction tasks. The benefit in classification tasks in (Ke et al.] (2020) may come from other modifications, for example, unties the [CLS] symbol from other positions. Moreover, in the paper, it adopts special relative position embedding like (Raffel et al. 2019) (as this paper also suggests to do so): a simplified form of PE that each "embedding" is simply a scalar bias added to the corresponding logits when computing the attention weights. The fundamental difference be- tween the "position bias' and position embedding is unknown from now.
Study on attention visualization. Many works focusing understanding attention patterns in individual heads. For example Vig (2019) introduced tool for visualizing attention in the Trans- former at multiple scales; Rogers al. (2020) suggest attention mechanisms like Vertical, Diagonal, Vertical + diagonal, Block, and Heterogeneous. Clark et al. (2019) found some attention mecha- nisms like attending broadly, to next, [CLS] or [SEP] ], attend to punctuation. While our paper focuses on the general attention introduced by PEs from an average point of view, without consider- ing any specific attention head.
Asymmetry in sequential labeling Yan et al. 2019) suggested asymmetry of position embedding in named-entity recognition task (without involving pre-trained language models) which is a kind of sequential labeling tasks like span prediction (SQuAD) in this paper. Their conclusion is generally compatible with ours, but we question its assumption that 'the property of distance-awares appears when query and key projection some distance-awareness by directly taking the average position-positionn correspondence in the first layer among many heads (i.e., Q. (WK.1) pT).
dis- we could slightly see
conducted', As shown in Fig. 9
Functional parameterization of PEs Xu et al. (2019) proposes various variants of sinusoidal positional encodings inspired by functional analysis. Wang al. (202)) proposed a sinusoid-li like
complex word embedding to encode word order. Both (Xu et al. 2019) and (Wang et al.) 2020)
assume that PEs should satisfy the translation invariance property, but they induce different types of sinusoidal PE parameterization either in real complex vector space. Moreover, Liu et al. (2020)
use a neural ODE component to parameterize position encoding as a continuous dynamical model, which could learn suitable PEs in neural networks. All of these PEs inspiring. Since selecting
17

---

Published as a conference paper at ICLR 2021
(a) hert-base-cane
(b)
(c) bert- lart-large-cased
(d) bert- bert-large-une large-uncased
Figure 9: Position-wist correlation matrix (PWQ) (WK.1) for first 128 positions in BERT pre-trained models
the suitable parameterization type not the main concern in this paper, we adopted the typical ones, namely, the fully-lcarnable, (learnable fixed) sinusoidal APEs/RPEs. The fundamental difference between these PE parameterizations needs further investigation. More recently, (Wang & Chen 2020) empirically study the behaviour of many position embeddings and their performance in Transformers for various NLP tasks.
I THE THREE PROPERTIES IN OTHER MODELS
By using the proposed identical probing test, we also check the properties of other trained Trans- former models with decoder components in Tab. and Fig, [0) The machine translation model [3 a typical encoder-decoder architecture using multiple-layer Transformers. GPT2 (Radford et al. 2019) adopts purely decoder architecture; 12-layer base setting is used in this work.
Monotonicity Compared to BERT and the machine translation, GPT2 satisfies monotonicity (es- pecially in the first 20 offsets) better than other models, showing capturing distance between neigh- boring tokens matters the language model.
Translation invariance As seen from the translation invariance indicators in Tab. GPT2 satis- fies translation invariance poorer than other models, since tokens in also additionally attend to a few beginning tokens matter how far the attended tokens are.
Symmetry GPT2 shows the biggest symmetrical discrepancy, since GPT2, which aims to predict the next word, adopts attention mask of succeeding tokens avoid information leakage. Plus, the machine translation encoder slightly attends more to the succeeding tokens while BERT attends more the preceding tokens than succeeding tokens.
Table 7: Quantitative measurement of the properties for models models.
of machine translation, language
PBs
PE type
model type
0.2461 0.0208 first odisets
officts
monotonicity
transiation invariance special tokens
symmetcy
direction balance
BERT GPT
fully leamable APE fells-leamabll
encoder encoder decoder
0,0143
0.0012
1.1980
0.1114 0.0214
Machine Tranelation
0.0902
WHITE BAND EFFECTS ALONG THE DIAGONAL
In order to analyze the white band effects along the diagonal, we show all results of identical word probing (average attention values in the first layer of identical word probing with respect to 100
13 An Feglish-to-Frendee machine translation model downloaded from httpp::/ /ttt:///gttuuu com/
pytorch fai /taar////aaaree/eeeeees/e/////// ratiiationn Vaswani et al. 2017 Oft et 2018
2019
18

---

Published as a conference paper at ICLR 2021
40
40
40
60
80-
80
100
120
100
(a) Encnder (w/ Decoder): Machine (h) Decoder
Generative language (c) Encnder Masked language
translation (Vaswani et al. 2017)
model (GPT2) (Radford et al. 2019) model (BERT) (Dvvinn al. 2018
Figure 10: Identical word probing with different types of trained models.
randomly-selected words). This effect is more clear for fully-learnable RPE, learnable sinusoidal RPE and any combination variants including them (see. Fig. [ (d.f.g.i...)).. To show the obvious differences between these PEs, in this paper, use average unnormalized attention weights matrix for probing, but all indicators are calculated using normalized attention values for better quantitative comparison.
19

---

Published as a conference paper at ICLR 2021
K THE REPLACEABLE PROPERTY ABOUT ABSOLUTE POSITIONS OF WORDS
For example (we do not consider subword tokenization for simplicity), we have two sentences for next sentence predictions (As BERT did)
sentencel: Deadlines are the No.1 productive forces sentence2 think therefore
By adding three special tokens, we will have example with 17 tokens as
[CLS] Deadlines are the No.1 productive forces therefore I am [SEP]
[SEP] think
with absolute positions in the bracket
[CLS] (1) Deadlines (2) are (3) the (4) No. (5) productive (6)
forces (7) (8) [SEP] (9) (10) think (11) (12) therefore (13) (14)
am (15) (16) [SEP] (17)
Assume that the expected maximum sequence length is 16 (actually 128 or 512 in BERT), need to randomly remove the first token of the first sentence (i.e., Deadl ines ) a
valid sample: I: [CLS] (1) are (2) the (3) No.1(4) productive (5). forces (6) (7) [SEP] (8) (9) think (10) (11) therefore I (13) am (14)
(15) [SEP] (16)
or last token of the second sentence (i.e.,
valid sample:
II: [CLS] (1) Deadlines (2) are (3) the (4) No. (5)
productive (6) forces (7) (8) [SEP] (9) (10)) think (11) (12)
therefore (13) (14) am (15) [SEP] (16)
Both the above two sentences are valid for training. If we replaced the first sentence with another shorter sentence (i.e., Publish/Launch or Perish ?), the sample would be
[CLS] (1) Publish (2) or (3) Perish (4) ? (5) [SEP] (6) (7) think (8) (9) therefore (11)) am (12) [SEP] (13) [PAD] (14)
valid sample: III:
[PAD] (15) [PAD] (16)
The three samples IIII.III are valid, but its absolute position indexes are not shiftable. Especially, the first sentence of the second sentence could be 9, 10, and 7, respectively, depending on the random seed for dropping and the length of the first sentence.
20

---

Published as a conference paper at ICLR 2021
0
20
20
03
60
60-
00
100
120
120
120
220
100
220
(a) hally-ecarnabee APE
(b) fixed sinusoidal APE
(c) learnable sin. APE
40
40
50
80
60
100
120
1.0
180
20
60
80
120
20
60
30
100
100
40
60
100
150
(d) Gealaalee RPE
(e) fixed sinusoidal RPE
(1) learnable sinusoidal RPE
40
40
50
50
60-
132
1.0
1.0
180
20
60
80
30B
130
20
60
30
300
120
40
60
100
350
(g) learnahle sinusoidal APE + fully- (h) learnahle sinusoidal APE + fixed (i) learnable sinusoidal APE + learn- learnable RPE
sinusoidal RPE
able sinusoidal RPE
40
40
50
80
60
100
1.00
1.00
120
20
40
40
320
120
20
60
30
300
100
60
100
150
(i) stRT-style APE learnable RPE
fully- (k) BERT-style APE fixed sinu- (1) BERT-style APE+ Iearnable sinu- soidal RPE
soidal RPE
Figure 11: Identical word probing (models with more PEs are shown here comparing to Fig. 2 Darker in the i-th row and j-th column means that the i-th words generally attend more on the 7-th words.
21


Reviews:
[{"review_id": "lQsimEZqETL", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a formal framework to compare position embeddings (PEs) and presents an empirical study comparing variants of absolute position embeddings (APEs) and relative position embeddings (RPEs) on three properties: 1) monotonicity, 2) translation invariance, and 3) symmetry and evaluates their performance on classification (GLUE) and span prediction tasks (SQUAD). The authors also report results on learnable sinusoidal APEs and learnable sinusoidal RPEs, PE variants which had not been previously proposed. \n\nThe first three properties seem well-motivated (monotonicity and translation invariance), but it is not obvious that symmetry should be a property of an ideal PE, or at least the paper is not convincing on this front. In a sentence (ABCD), doesn’t the word A typically have a different relationship to B than B does to A?\n\nThe identical word probing test was a clever way to disentangle the impact of the word from that of the PEs. \n\nWhile it does seem valuable to more rigorously compare PEs as they are critical components of SOTA language models, the experimental results were not particularly convincing (e.g. although it’s a very appealing story, it didn’t seem so clear from the tables that APEs did better at classification and that RPEs did better at span prediction.)\n\nThe writing quality was borderline and there were a number of small errors:\n- fully-learnable APEs nearly meet all properties even under no constrains” -> “constraints”\n- Under the equation at the beginning of Section 3.1, “word-word correspondence” is repeated four times, which I am sure was not the intention. \n- nit: “since relative distance with the same offset will be embedded as a same embedding.” -> “the same embedding”\n- “compared to far-way” -> “faraway”\n- “attends more on forwarding tokens than backward tokens” -> “forward tokens”?\n- nit: “In Transformer, where attention calculation does not…” -> “where the attention calculation”\n- “allows PEs o better perceive word order” -> “to”", "extra_scores": {}, "extra_text": {}}, {"review_id": "8ki42ZkgicZ", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Updates after author responses and revisions:\n\nI was positive about this paper previously and am glad to see that the authors have done a great deal to try to respond to our concerns and strengthen the paper. I am more positive about the paper now and have increased my score to an 8. I think this paper is going to be useful for the community and I know I will reference it later and direct others to it who are interested in learning more about position embeddings in transformers (whether or not it actually gets published). \n\n--------------\n\nThis paper studies position embeddings (PEs) in transformers, suggesting a few reasonable formal properties of PEs and determining whether these properties are captured by various choices for defining PEs. The paper considers both absolute and relative PEs, and both fully-learnable and sinusoidal. A new variation is to learn frequencies in the sinusoidal PEs. Experiments are conducted by training BERT with various choices for PEs and evaluating on GLUE tasks and SQuAD. Fully-learnable absolute PEs, the default in BERT, works quite well overall, but adding learnable sinusoidal relative PEs as well can improve performance on average. There are also visualizations of PE dot products for the various methods, showing that learning PEs in the context of BERT training can yield PEs that satisfy the formal properties (for the most part) laid out by the authors. \n\nI really enjoyed reading this paper. PEs in BERT have been the subject of a lot of informal discussion over the past couple of years, but I don't think I've read a paper that studies the topic with such breadth and depth. With some doable improvements and clarifications, I think the paper can become an excellent resource for others interested in representing position and distance in transformer-like models. \n\nI like the idea of learned sinusoidal embeddings and think that idea can be potentially useful for other researchers. The paper notes that with the fixed sinusoidal embeddings, distances beyond about 50 are not distinguished, but with learning they can be. The experiments show that learning frequencies in sinusoidal PEs works better than using fixed sinusoidal PEs. I'm curious what the learned frequencies look like and how similar they are to the original frequencies chosen by the transformer authors. \n\nSome other thoughts I had while reading: \n\nAt least for language tasks with a given window, could we just re-use the learned frequencies and use fixed sinusoidal PEs in the future with those same learned frequencies? I guess a similar choice can be made with fully learned PEs. Do we really need to learn PEs from scratch every time? What aspects of the data or task would influence this? Maybe given their experience, the authors could hand-design useful general-purpose PEs? \n\nBased on the discussion in Sec. 5.3, maybe a good recommendation for BERT-like models would be to use a single learnable PE only for [CLS] and something else (e.g., learned sinusoidal APEs or RPEs) for other positions? This seems easy enough to do in practice and may bring the best of both worlds. \n\n\nBelow are some questions and points of confusion I had:\n\nIn the fully-learnable APE experiments described in Sec 4.2, for the first 10 epochs, were the unseen PEs randomly initialized and finetuned in the downstream SQuAD experiments? \n\nI'm confused as to why some of the visualizations in Fig 3 show white bands along the diagonal (d, f, and g). I would have expected all to have dark bands along the diagonal (as in b, c, d, and h). \n\nI was super confused by the identical word probing parts of the paper. Sec. 3.1 includes a sentence beginning with \"This shows that the selection of words\". The sentence is awkward and confusing to me. I don't know what the \"This\" is referring to. It seems like there should have been a result reported there for the \"This\" to refer to. The following sentence, starting with \"Namely\", is also confusing to me. The description of identical word probing then points to Sec 5.2, and the Sec 5.2 section title is \"IDENTICAL WORD PROBING\", but it seems to be focused on describing Fig 3, which has as its caption \"Dot products between relative position vectors\". Where are the identical word probing results actually reported? \n\nIn the \"Pre-training\" paragraph of Section 4, it is mentioned that a pretrained BERT checkpoint is used, but also that other BERT models with other position embeddings were trained. How was the pretrained model actually used? Was it used to initialize the other models or were all models trained from scratch?\n\nMinor things: \n\nIn the first equation in Sec 3.2, there are a couple of instances of K/2 which I think should be D/2 instead. Also, the vectors start with \\omega_1 but the summation at the end of the equation starts with \\omega_0. \n\nI'm confused by the bolding in Table 2. The best number in each column is not always in boldface (QNLI, WNLI, etc.). Also, sometimes when there is a tie for the best result, multiple numbers are in bold, while other times only one result is in bold. \n\n\nTypos:\n\np. 1: \"constrains\" --> \"constraints\"\n\nfootnote 4: \"seeing\" --> \"See\"\n\nSec. 4.1: \"outperform notably\" --> \"notably outperform\"\n\nSec. 5.3: \"PEs o\" --> \"PEs to\"\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "7OEdSRfr94P", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the position embeddings of transform-based models, and proposed a unified position embedding evaluation method in three aspects, i.e., Monotonicity, Translation invariance and Symmetry, which can well summarise the properties of the existing position embedding methods.\n\n### Strengths of the paper\n\n1. It is good that the authors summarise three property for position embedding models, and discuss four related position embedding models under the three properties.\n2. The three properties proposed by this paper are suitable for most position embedding models.\n3. The experimental details are complete and easy to reproduce. Extensive experiment details are provided in the appendix.\n\n### Weaknesses of the paper\n\n1. The presentation and organization of this paper should be improved. Typos and language issues can be easily found, see the minor comments below. The contributions are not well highlighted in both the abstract and introduction section. \n\n2. The authors take many efforts to conduct experiments for the position embedding of BERTs, and provide an empirical study of the four existing position embedding models (fully learnable APEs (Gehring et al., 2017), (2) fixed sinusoidal APEs (Vaswani et al., 2017), (3) fully learnable RPEs (Shaw et al., 2018), and (4) fixed sinusoidal RPEs (Wei et al., 2019).). The author tested these models and their combinations over three benchmarking datasets, however, as an empirical study paper, the analysis is too weak.\n\n(1) For the qualitative result shown in Table 2 and Table 3, neither insight nor connection to three properties is provided for the result. In fact, the result shown in Section 4 is intuitive and not something new, since these are the basic motivations of the RPEs and learnable PEs. Moreover, the effectiveness of using both RPE and APE has already been validated in \"Self-Attention with Structural Position Representations, arXiv:1909.00383. 2019\".\n\n(2)  I would like to see, in the experiment part, some new experimental results and conclusions in terms of the four summarised properties, which are key contributions the authors claimed. However, what I see is just some position vector embedding similarities (i.e. Fig. 2 and Fig.3) in terms of the four position embedding models, where the results are also somehow expected and intuitive. The conclusions of the experimental results are too subjective. For example, from the analysis of Fig (2), the conclusion that : “Lastly, note that fully-learnable RPEs also do not significantly distinguish far-distant RPEs (from -64 to -20 and from 20 to 64), suggesting that truncating RPEs into a distance of 64, like (Shaw et al., 2018), is reasonable.”  is a bit farfetched. In this figure the white part is as narrow as (-5,+5),  a further quantitative evidence in other forms for this conclusion will be more preferable. The same issue also exists in the conclusion of Figure (3): “This may be an advantage of RPEs over APEs to perceive forward and backward words, especially in span prediction tasks where capturing this matters.”\n\n3. This paper lacks the discussion with the latest position embedding models such as,\n\n   [1] \"Learning to Encode Position for Transformer with Continuous Dynamical Model\". ICML. 2020\n\n   [2] \"Encoding Word Order in Complex Embeddings\",  ICLR. 2019, where the 'Translation' property is also discussed.\n\nOther minor comments:\n\n1. In the introduction section, \"distances in $\\mathbb{N}$ and $\\mathbb{R}^{D}$ \", $\\mathbb{N}$ and $\\mathbb{R}^{D}$ should be explained when they appear in the first place.\n\n2. The references should be formatted in a unified manner.\n\n3. Table 2, the bold indicators for the best performances are put on the wrong numbers, e.g. in QNLI, the bold should be 89.5,  in WNLI task it should be 51.3, and in STS-B it should be 87.5.\n\n   \nOverall, I think this paper indeed shows some interesting empirical results of position embedding models for BERT. But, the analysis is monotonous and too subjective, lacking the necessary mathematical quantitative indicators, which prevents it from being a general way to verify the conclusions in this paper.\n\n\n------\n### Comments after the discussion\n\nThank you for your detailed response. I think most of my concerns were addressed so I updated the score to 6.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "12pkZxA4ho5", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a systematic analysis of approaches used to encode position information in transformers and in particular BERT-based models. The paper investigates absolute and relative position embedding strategies that use either fixed/learnable sinusoidal or fully learnable position embeddings. These embeddings are characterized based on different properties that are either inherent from their formulation or observed empirically such as monotonicity, translation invariance, and symmetry. Interestingly these properties appear to emerge naturally when having learnable parameters in APEs and RPEs.\n\nDifferent PE strategies are empirically validated by pre-training BERT with different PEs (including a combination of absolute and relative position representations) and fine-tuning on GLUE and SQuAD (1.1 and 2.0). Visualizations of the dot products between position vectors for different PE strategies are presented as well that demonstrate the monotonicity (or local monotonicity), symmetry, and translation invariance.\n\nOverall, the paper is well written, motivated, and systematically studies an important design decision in transformers. The overall methodology is sound and should prove useful to the community when studying position embeddings in transformers.\n\nStrengths\n\nThe paper is well written, well-motivated, and is systematic in its claims, experiments, and methodology.\nIt takes a good step forward in characterizing desirable properties of position embeddings and studying if they emerge directly from their parameterization or via training.\nIt studies a variety of position embedding strategies and their conjunction and experiments with fairly realistic models and benchmarks.\n\nWeaknesses\n\nWhile the analyses are well carried out, they are still specific to BERT-like masked language modeling training and it isn’t clear if PEs behave similarly in encoder-decoder like models for summarization/translation or decoder only language models.\nIt does not control for the choice of (pre)training objective, for example, if trained from scratch, purely as a retrieval model or on supervised text classification, would different learned PE patterns emerge?\n\nIt provides certain properties one can hope PEs to exhibit but doesn’t talk too much about circumstances under which these would be good inductive biases to have.\n\nQuestions & Comments:\n\nUnder what circumstances do properties like monotonicity in learnable PEs arise? Would PEs for an autoregressive variant also display similar properties? (would be interesting to see what happens with say a transformer LM trained from scratch on wikitext-103)\n\nIt isn’t quite clear to me at first glance why the preservation of the order of distances is necessary for position embeddings when used with models that build nonlinear functions of these embeddings.\n\nWould absolute position embeddings that are randomly initialized or all initialized orthogonal to one another (something like 1-hot vectors of dimension L) *and fixed*  work? This would be interesting to see (especially the latter) because it does not satisfy monotonicity or translation invariance and violates the desiderata in eq (1)\n\nIs Figure 7 based on BERT fine-tuned for NER on the CONLL/Ontonotes dataset?\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: With the advent of non-recurrent sequence-processing models, it has become costumary to augment input tokens with positional embeddings providing implicit positional information. Despite their potentially crucial role in modern architectures, such positional embeddings are rarely addressed in analytical studies. The current paper provides a systematic investigation of positional embeddings, characterized in terms of properties such as monotonicity, translation invariance, and symmetry. These properties are studies for different positional embeddings using language models fine-tuned on two separated benchmarks, with an emphasis on visual analysis.

The authors provided an impressive rebuttal, adding many of the experiments required by the reviewers. The latter are still somewhat split about the paper. I lean towards the positive side. I find that some of the criticism, while valid, is not really granting a rejection, especially after the authors' clarifications. In particular, one reviewer assumed that the authors claim that symmetry should be a property of an ideal positional embedding, whereas the authors are rather studying whether it is an important property of them, in light of the previous literature. Some claims about the results being "interesting" or "surprising" enough might depend on what the reader is looking for in the paper. I think that many readers in the "black box NLP" community will find the methods and analyses presented in this paper interesting and useful.

Decision: Accept (Poster)