On Position Embeddings in BERT

1 INTRODUCTION . Position embeddings ( PEs ) are crucial in Transformer-based architectures for capturing word order ; without them , the representation is bag-of-words . Fully learnable absolute position embeddings ( APEs ) were first proposed by Gehring et al . ( 2017 ) to capture word position in Convolutional Seq2seq architectures . Sinusoidal functions were also used with Transformers to parameterize PEs in a fixed ad hoc way ( Vaswani et al. , 2017 ) . Recently , Shaw et al . ( 2018 ) used relative position embedding ( RPEs ) with Transformers for machine translation . More recently , in Transformer pretrained language models , BERT ( Devlin et al. , 2018 ; Liu et al. , 2019 ) and GPT ( Radford et al. , 2018 ) used fully learnable PEs . Yang et al . ( 2019 ) modified RPEs and used them in the XLNet pre-trained language model . To our knowledge , the fundamental differences between the various PEs have not been studied in a principled way . We posit that the aim of PEs is to capture the sequential nature of positions in vector space , or technically , to bridge the distances in N ( for positions ) and RD ( for position vectors ) . We therefore propose three expected properties for PEs : monotonicity , translation invariance , and symmetry 1 . Using these properties , we formally reinterpret existing PEs and show the limitations of sinusoidal 1Informally , as positions are originally positive integers , one may expect position vectors in vector space to have the following properties : 1 ) neighboring positions are embedded closer than faraway ones ; 2 ) distances of two arbitrary m-offset position vectors are identical ; 3 ) the metric ( distance ) itself is symmetric . PEs ( Vaswani et al. , 2017 ) : they can not adaptively meet the monotonicity property – thus we propose learnable sinusoidal PEs . We benchmark 13 PEs ( including APEs , RPEs , and their combinations ) in GLUE and SQuAD , in a total of 11 individual tasks . Several indicators are devised to quantitatively measure translation invariance , monotonicity , and symmetry , which can be further used to calculate their statistical correlations with empirical performance in downstream tasks . We empirically find that both text classification tasks ( in GLUE ) and span prediction tasks ( SQuAD V1.0 and V 2.0 ) can benefit from monotonicity ( in nearby offset ) and translation invariance ( in particular without considering special tokens like [ CLS ] ) , but symmetry decreases performance since it can not deal with directions between query vectors and key vectors when calculating attentions . Plus , models with unbalanced attention regarding directions ( generally attending more to preceding tokens than to succeeding tokens ) slightly correlate with better performance ( especially for span prediction tasks ) . Experiments also show that the fully-learnable APE performs better in classification , while RPEs perform better in span prediction tasks . This is explained by our proposed properties as follows : RPEs perform better in span prediction tasks since they meet better translation invariance , monotonicity , and asymmetry ; the fully-learnable APE which does not strictly have the translation invariance and monotonicity properties during parameterizations ( as it also performed worse in measuring translation invariance and local monotonicity than other APEs and all RPEs ) still performs well because it can flexibly deal with special tokens ( especially , unshiftable [ CLS ] ) . Regarding the newly-proposed learnable sinusoidal PEs , the learnable sinusoidal APE satisfies the three properties to a greater extent than other APE variants , and the learnable sinusoidal RPE exhibits better direction awareness than other PE variants . Experiments show that BERT with sinusoidal APEs slightly outperforms the fully-learnable APE in span prediction , but underperforms in classification tasks . Both for APEs and RPEs , learning frequencies in sinusoidal PEs appears to be beneficial . Lastly , sinusoidal PEs can be generalized to treat longer documents because they completely satisfy the translation invariance property , while the fully-learnable APE does not . The contributions of this paper are summarised below : 1 ) We propose three principled properties for PEs that are either formally examined or empirically evaluated by quantitative indicators in a novel Identical Word Probing test ; 2 ) We benchmark 13 PEs ( including APEs , RPEs and their combinations ) in GLUE , SQuAD V1.1 and SQuAD V2.0 , in a total of 11 individual tasks ; 3 ) we experimentally evaluate how the performance in individual tasks benefits from the above properties ; 4 ) We propose two new PEs to extend sinusoidal PEs to learnable versions for APEs/RPEs . 2 PROPERTIES OF POSITION EMBEDDINGS . Gehring et al . ( 2017 ) ; Vaswani et al . ( 2017 ) use absolute word positions as additional features in neural networks . Positions x ∈ N are distributively represented as an embedding of x as an element ~x ∈ RD in some Euclidean space . By standard methods in representation learning , similarity between embedded objects ~x and ~y is typically expressed by an inner product 〈~x , ~y〉 , for instance the dot product gives rise to the usual cosine similarity between ~x and ~y . Generally , if words appear close to each other in a text ( i.e. , their positions are nearby ) , they are more likely to determine the ( local ) semantics together , than if they occurred far apart . Hence , positional proximity of words x and y should result in proximity of their embedded representations ~x and ~y . One common way of formalizing this is that an embedding should preserve the order of distances among positions 2 . We denote φ ( · , · ) as a function to calculate closeness/proximity between embedded positions , and any inner product can be a special case of φ ( · , · ) with good properties . We can express preservation of the order of distances as : For every x , y , z ∈ N , |x− y| > |x− z| =⇒ φ ( ~x , ~y ) < φ ( ~x , ~z ) ( 1 ) Note that on the underlying space , the property in Eq . ( 1 ) has been studied for almost 60 years ( Shepard , 1962 ) , in both algorithmics ( Bilu & Linial , 2005 ; Badoiu et al. , 2008 ; Maehara , 2013 ) , and machine learning ( Terada & Luxburg , 2014 ; Jain et al. , 2016 ) under the name ordinal embedding . As we are interested in the simple case of positions from N , Eq . ( 1 ) reduces to the following property : 2Theoretical evidence for this is nontrivial unless we assume more about the particular non-linear functions . We empirically find that all learned PEs can preserve the order of distance Property 1 . Monotonicity : The proximity of embedded positions decreases when positions are further apart : ∀x , m , n ∈ N : m > n⇐⇒ φ ( ~x , −−−−→x+m ) < φ ( ~x , −−−→x+ n ) ( 2 ) A priori , a position embedding might treat every element N individually . However , considering pairs of positions based on their relative proximity ( rather than the absolute value of the positions ) , can lead to simplified and efficient position embeddings ( Wang et al. , 2020 ) . Such embeddings satisfy translation invariance : Property 2 . Translation invariance : The proximity of embedded positions are translation invariant : ∀x1 , . . . , xn , m ∈ N : φ ( ~x1 , −−−−→ x1 +m ) = φ ( ~x2 , −−−−→ x2 +m ) = · · · = φ ( ~xn , −−−−−→ xn +m ) ( 3 ) Finally , since the inner product is symmetric , we also consider whether φ ( · , · ) is symmetric : Property 3 . Symmetry : The proximity of embedded positions is symmetric , ∀x , y ∈ N : φ ( ~x , ~y ) = φ ( ~y , ~x ) ( 4 ) There is no generally accepted standard set of properties for position embeddings ; based on prior work as described above , we posit that the above properties are important , and now examine several existing PEs in relation to these properties , either formally ( in Sec . 3 ) or empirically ( in Sec . 4 ) . 3 UNDERSTANDING PES VIA THE PROPERTIES . PEs come in two variants : absolute PEs ( APEs ) where single positions are mapped to elements of the representation space , and relative PEs ( RPEs ) where the difference between positions ( i.e. , x−y for x , y ∈ N ) is mapped to elements of the embedding space . For Transformer-based architectures , the difference between APEs and RPEs manifests itself in the attention mechanism , in particular how the matrices of query , key , and value weights WQ , WK , and WV are used to calculate attention in each attention head . Consider two positions x , y ∈ N , let WEx be the word embedding of the word at position x , and let Px and Px−y be the embeddings of the position x and relative position x− y , respectively . The query-key-value vector for the word at position x is typically calculated as below for APEs and RPEs3 respectively : APE : QxKx Vx = ( WEx + Px ) WQWK WV ; RPE : QxKx Vx = WEx WQWK WV + 0Px−y Px−y ( 5 ) Observe that while the APEs calculation is linear in ( WQ , WK , WV ) with the word and position embeddings merged into the coefficient , the RPEs calculation is affine , with the relative position embedding Px−y acting as an offset independent of the word embedding WEx . In Transformers , the resulting representation is a sum of value vectors with weights depending on A = QKT , that is , Attention ( Q , K , V ) = softmax ( QKT / √ dk ) V . In the rest of the paper , we examine PEs in the above architecture with respect to the properties introduced in Section 2 . In particular , we study four well-known variants of PEs : ( 1 ) the fully learnable APE ( Gehring et al. , 2017 ) , ( 2 ) the fixed sinusoidal APE ( Vaswani et al. , 2017 ) , ( 3 ) the fully learnable RPE ( Shaw et al. , 2018 ) , and ( 4 ) the fixed sinusoidal RPE ( Wei et al. , 2019 ) . 3.1 UNDERSTANDING SINUSOIDAL PES . With a sinusoidal parameterization in PEs , we may use a specific proximity , i.e. , an efficient inner product like a dot product , to check if the sinusoidal form of PEs meets the above properties . The dot product between any two position vectors is Ax , y = 〈~x , ~y〉 = sum sin ( ω1x ) cos ( ω1x ) · · · sin ( ωD 2 x ) cos ( ωD 2 x ) sin ( ω1y ) cos ( ω1y ) · · · sin ( ωD 2 y ) cos ( ωD 2 y ) = sum sin ( ω1x ) sin ( ω1y ) cos ( ω1x ) cos ( ω1y ) · · · sin ( ωD 2 x ) sin ( ωD 2 y ) cos ( ωD 2 x ) cos ( ωD 2 y ) = D 2∑ i=0 cos ( ωi ( x− y ) ) ( 6 ) 3There are many variants of RPEs ( e.g. , ( Dai et al. , 2019 ) ) . As selecting RPEs is not the main concern in this paper , we give the original ( and typical ) RPEs only . One can easily extend this work to other RPE variants . Note that sinusoidal PEs satisfy both Property 2 ( translation invariance ) because the inner product is only associated with its position difference x − y , and Property 3 ( symmetry ) , because the dot product itself is symmetric : 〈~x , ~y〉 = 〈~y , ~x〉 . Note also that checking Property 1 is equivalent to checking monotonicity of the map ψ ( m ) = ∑D/2 i=1 cos ( ωim ) . ψ ( m ) is monotone on intervals where its first order derivative ψ′ ( m ) = ∑D/2 i=1 −ωi sin ( ωim ) does not change sign , and these intervals depend on the choice of ωi . With fixed frequencies ωi = ( 1/10000 ) 2i/D , it is monotonous when m is roughly between 0 and 50 , indicating that it can only strictly perceive a maximum distance of 50 and it is insensitive to faraway distances ( e.g . longer than 50 ) . Although sinusoidal PEs with fixed frequencies ( i.e. , ωi = ( 1/10000 ) 2i/D ) are common in APEs and RPEs , we argue that learning these frequencies is useful because it can adaptively adjust intervals of monotonicity ( they do not have to be 0-50 as in the fixed sinusoidal APE ) 4 . With trainable frequencies , we can adaptively allocate a number of frequencies in a data-driven way . App . A.2 explains the expressive power of sinusoidal PEs with trainable frequencies from the perspective of the Fourier series . Extending existing fixed sinusoidal PEs to a learnable version with learnable frequencies gives two variants : a learnable sinusoidal APE and a learnable sinusoidal RPE .

Summary: The paper presents a systematic analysis of approaches used to encode position information in transformers and in particular BERT-based models. The paper investigates absolute and relative position embedding strategies that use either fixed/learnable sinusoidal or fully learnable position embeddings. These embeddings are characterized based on different properties that are either inherent from their formulation or observed empirically such as monotonicity, translation invariance, and symmetry. Interestingly these properties appear to emerge naturally when having learnable parameters in APEs and RPEs.