UNDERSTANDING THE ROLE OF POSITIONAL ENCODINGS IN SENTENCE REPRESENTATIONS
Authors: Lihu Chen
Gael Varoquaux
Fabian M. Suchanek
Venue: Submitted to ICLR 2023 (2023)
Keywords: Positional Encodings
Sentence Representations
Pre-trained Language Models

Abstract: Positional encodings are used to inject word-order information into transformer-based language models. While they can significantly enhance the quality of sentence representations, their specific contribution to language models are not fully understood, especially given recent findings that building natural-language understanding from language models with positional encodings is insensitive to word order. In this work, we investigate the role of positional encodings systematically. (1) We uncover the core function of existing positional encodings is to symmetrically combine local units by identifying two common properties, Locality, and Symmetry. (2) We reveal that positional and contextual encodings play a distinct role in understanding sentences. (3) Based on these findings, we propose a simplified new method to inject positional information into such models. Empirical studies demonstrate that this method can improve the performance of the BERT-based model on 10 downstream tasks. We hope these new probing results and findings can shed light on how to design and inject positional encodings into language models.


Under review as a conference paper at ICLR 2023
UNDERSTANDING THE ROLE OF POSITIONAL ENCOD- INGS IN SENTENCE REPRESENTATIONS
Anonymous authors Paper under double-blind review
ABSTRACT
Positional Encodings (PEs) used inject word-order information into transformer-based language models. While they can significantly enhance the quality of sentence representations, their specific contribution to language models are not fully understood. especially given recent findings that building natural- language understanding from language models with positional encodings are in- sensitive to word order. In this work, we conduct more in in-depth and systematic studies of positional encodings, thus complementing existing work in four aspects: (I) We uncover the core function of PEs by identifying two common properties, Locality and Symmetry; (2) We first point out a potential weakness of current PEs by introducing two new probing tasks of word swap; (3) We first investigate the linguistic capability of (4s;(4) Based on these findings, we propose a simpli- fied method to inject positional information into such models. Empirical studies demonstrate that this method can improve the performance of the BERTT-aase model on 10 downstream datasets. We hope these new probing results and find- ings can shed light on how to design and inject positional encodings into language models.
INTRODUCTION
Transformer- based language models with Positional Encodings (PEs) can improve performance con- siderably across wide range of natural language understanding tasks. Existing work resort to either fixed (Vaswani et al. 2017; Su et al. 2021 Press et al.] (02Ta) or learned (Shaw et al., [018 De-
vlin et al.) 2019 Wang et al. 2019) PEs to infuse order information into attention attention-based models. To understand how PEs capture word order, prior studies apply visualized (Wang & Chen (2020) and quantitative analyses (Wang et al., 2020) various PEs, and their findings conclude that all encodings, both human-designes and learned, exhibit a consistent behavior: First, the position wiie weight matrices show that non-zero values gather on local adjacent positions. Second, the matrices are highly symmetrical, as shown in Figure These are intriguing phenomena, with reasons not well understood.
To bridge this gap, we strive to uncover the core properties of PEs by introducing two quantitative metrics, Locality and Symmetry. Our empirical studies demonstrate that these two properties are correlated with sentence representation capability. This explains why fixed encodings are designed to satisfy them and learned encodings are favorable be local and symmetrical. Moreover, we show that if BERT is initialized with PEs that already share good locality and symmetry, it can obtain better inductive bias and significant improvements across downstream tasks.
Although PEs with locality and symmetry can achieve promising results on natural language un- derstanding tasks (such as GL.UE Wang et al. (2018), ) the symmetry property itself has an obvious weakness, which is not revealed by previous work. Existing studies shuffled text to probe the sensitivity of PEs to word orders (Yang et al. 2019a Pham et al., [021]; Sinha et al. 2021; Gupta al. 2021 Abdou et al. 2022). and they all assume that the meaning of sentences with random swaps remains unchanged. However, the random shuffling of words may change the semantics of the original sentence and thus cause the change of labels. For example, the sentence pair below from SNLI (Bowman et al. (015)) satisfies the entailment relation:
A man playing an electric guitar stage
b. A man playing guitar on stage

---

Under review as a conference paper at ICLR 2023
If we change the word order of the premise sentence that it becomes "an electric guitar playing man on stage", but fine-tuned BERT still finds that the premise entails the hypothesis. Starting from this point, we design two new probing tasks of word swap: Constituency Shuffling and Semantic Role Shuffling. The former attempt to preserve the original semantics of the sentence by swapping words inside constituents (local structure) while the latter intentionally changes the semantics by swapping the semantic roles in a sentence (global structure), c.g., the agent and patient, Our probing results show that existing language models with various PEs are robust against local swaps but extremely fragile against global swaps.
Moreover, we investigate the linguistic roles of positional encodings, which have not yet been stud- ied by prior work. Our empirical results show that there is clear distinct role between the positional and contextual encodings sentence comprehension: positional encodings play more of role at the syntactic level while contextual encodings serve more at the semantic level (if the semantic task does not require word order information), and the combination of the two can consistently yield better performances these probing tasks. As for the dependency relations, positional weights capture more short-distance dependencies while contextual weights capture more long-distance ones. Fi- nally, based features, which a simple yet effective way to inject positional encodings into language models. Experimental results show that our proposed method can bring improvements across 10 sentence- level downstream tasks.
new findings, we propose new method to combine positional and contextual
The key contributions of our work are:
We introduce two quantitative metrics, locality and symmetry, systematically uncover the main functions of positional encodings.
We design new probing tasks of word swaps, which show weakness of existing positional encodings, namely the insensitivity against the swap of semantic roles.
We first probe the linguistic roles of positional encodings, which reveals contextual and positional encodings play distinct roles at the syntactic level.
Based our findings, propose novel way combine positional and contextual en- codings, which can bring performance improvement without introducing complexity.
2
PRELIMINARIES
The central building block of transformer architectures is the self-attention mechanism (Vaswani et al. 2017). Given an input sentence: X = X2,,.2,..., x,) words and d is the dimension of word embeddings, then the attention computes the output of the i-th token in this way:
Raxd, where n is the number of
8;
exp(as.) Z
X, W
where 0s, (x: (xx, W
2 Σ exp(o
(I)
Vd
Self- seff-attention heads do not intrinsically capture the word orders in sequence. Therefore, specific methods are used to infuse positional information into self-attention Dufter et al. (2022)
Absolute Positional Encoding (APE) computes positional encoding for each token and add to the input content embedding to inject position information in the original sequence. The as in Equation I are then written:
(x. + p: ((x) WTJT
Va
where P. Id is position embedding of the ith token, obtained by fixed (Vaswani et al. 2017
Dehghani et al. 2018, Takase & Okazakil 2019 Shiv & Quirk] 2019 Su al. (021) or learned
encodings (Gehring el al. Devin et al. 2019 Wang et al. 2019 Press et al. (021b). lb). Further,
TUPE model simpifies Equation by removing two redundant items (see details in Section A of the appendix):
W I(, " + (p. (p,
(3)
Va
2

---

Under review as a conference paper at ICLR 2023
Relative Positional Encoding (RPE) produces a vector Fi,j scalar value 3r, that depends on the relative distance of tokens. Specifically, these methods apply such vector or bias to the attention head so that the corresponding attentional weight can be updated based on the relative distance of two tokens (Shaw et al. 2018 Raffel et al. 2019):
x. W (x, W 04
(x. W. [xx, W +
(4)
va
vd
where the left mode uses vector Fú, while the right uses scalar value Bry, for infusing relative distance into attentional weight.
Recent research of RPEs has been remarkably vibrant, with the emergence of diverse novel and promising variants (Dai et al., 2019; He et al.) 2020 Press et al. 2021a).
Unified Positional Encoding. Inspired by TUPE <e et al. 2021 and relative positional encodings in a unified way:
rewrite all above absolute
pos ctioonaal
(5)
where, the left half of the numerator, Wi captures contextual correlations (or weights), i.e., the semantic relations between token x; and Ty.. In this case, it is Yt. (x, (x) W 6, the right half, captures positional correlations, i.c., the positional relations between tokens and zy. For example. TUPE's positional correlation can be represented as 6,,5 = (p. W (p,, > while the relative encoding in Shaw et al. (2018) 8 can be represented as 0G,j = x, W (.j) ). Thus, existing positional encodings all add contextual and positional correlations together in every attention head.
K
POSITIONAL ENCODINGS ENFORCE LOCALITY AND SYMMETRY
3.1 THE PROPERTIES OF LOCALITY AND SYMMETRY
Existing work analyze positional encodings with the help of visualizations Wang & Chen (220): Wang et al. (2020); (bdou et alla (2022). and their analyses of either fixed or learned encodings led to similar visualized results, as shown in Figure I These position-wise weight matrices are computed by using the Identical Word Probing proposed by |Wang al. (2020): many repeated identical words are fed to the pre-trained language model, so that the atten- tion values (ax.)) in Equation 5) are unaffected by contextual weights (See details of visualiza- tions in Section B.T). Each matrix in this figure is a positional weight map. where each row is vector for the i-th position and the element (i, j) indicates the attention weight between the i-th position and the j-th position. We can first observe these attention matrices are all diagonal heavy, which means various positional encodings highly attend to local positions. Second, all matrices are nearly symmetrical, We call these two phenomena the Locality and Symmetry of positional encod- ings, and we provide linguistic explanation for the two properties in Appendix [B.3] The symmetry property has been discovered and quantified already Wang et al. (2020). Here, provide more in-depth analysis symmetry. We will also point out the potential flaw of symmetry itself, which is not considered by prior work. To better understand how encodings capture word order, we introduce two quantitative metrics to depict the Locality and Symmetry for an attentional weight vector 4 where the element Ei,j can be denoted as:
Figure Visualizations of different pre-trained language models by using Identical Word Prob- ing (Wang et al. 2020). The attention weights are averaged across different layers.
explas.))
where 6S. 2 O and
=1==1
(6)
exp(os.)
Locality is a metric that depicts the degree of the gathering of weights in local positions for an attentional weight vector. Given weight vector for the i-th position & {66,1 6,2; G,n), ), define locality as:
Locality (c) E [0, I]
(7)
28
3T
3

---

Under review as a conference paper at ICLR 2023
C10
171
110
C110
114
Relamner
0.30
0:34
food
1.0
5.3
6.3
Symmetty
Symmety
Symmetty
ElEanntttasseetton Syoneenry
Figure 2: Empirical studies the properties of locality and symmetry. The accuracy tested on the
MR dataset (Pang & Lee 2005) The yellow line shows the locality or symmetry for the pre-trained BERT. Results on SUBJ dataset Pang & Lee 2004) are shown in Figure 7
Here, a value of means the vector perfectly satisfies the locality property. For example, given a sequence whose length is 5 and weight vector for the first position [L, 0, 0, O). the locality is I, which means perfectly matches the locality. In contrast, the locality is 1/16 if the weight only attends the last position [0, 0, O, 0, 1]. For measuring the locality of matrix, average the locality values of all vectors in the matrix.
Symmetry is metric that describes how symmetrical the weights scatter around the current position for an attentional weight vector. We adapt the Symmetrical Discrepancy from Wang al. (2020) for this goal:
[n/2] Norm
Symmetr((s) E [0,
-6.n-++1l [n/2]
(8)
Here, value of I means that the vector is completely symmetrical. We modify the original formula in two points: First, we apply a min-max normalization to each position to obtain more uniform distributions, because the values of the original one extremely cluster around O Second, reverse the value so that 1 means a perfect symmetry instead of O. Likewise, the average value of all vectors matrix is used as the matrix-level symmetry.
3.2 ARE LOCALITY AND SYMMETRY LEARNED?
The manually designed encodings Sinusoidal Vaswani et al. 2017) and Roformer (Su et al. 2021 both satisfy the symmetry and locality properties. However, it is not clear why they were designed this way. More surprisingly, learned encodings all display locality and symmetry. Therefore, one may ask whether the two properties are Iearned after pre-training and what effect they have.
To answer this question, we use our two proposed metrics to quantify the positional weight matrix (the averaged weight across layers) before and after gre-training. Specifically, three language mod- els, BERT (Devlin et al. 2019), XLNet (Yang et al. 2019bb and DeBERTa (He al.) (020) tested in this experiment. As shown in the left in Figure 2 the three language models all become much more local and symmetrical after pre-training, which proves that the two properties are indeed leamedd
To further explain why positional encodings have a preference for learning these two properties, we probe the correlations between the two properties and the representation ability in downstream tasks. To avoid pre-training all language models from scratch, we use static word embeddings from GloVe (Pennington et al. 2014) and an encoder that is fully based on our handcrafted positional encodings for sentence classification task. The benefit that we can adjust the hyper-parameter in the handcrafted encodings to obtain encodings with different degrees of locality and symmetry, so that we can evaluate the correlations precisely. Specifically, we obtain around 100 encoders whose locality (or symmetry) varies from 0.01 to 1.0 and test their accuracy the MR sentiment analysis task. We will describe our handcrafted encodings in Section 3.3 The details of the encoder used in this experiment is described in Appendix B.4
The middle figure in Figure shows the results for different locality values. In this experiment, the symmetry value is 1.0 for all encoders. We observe that the accuracy constantly increases as the locality of encodings strengthens, which means a higher locality induces better sentence represen- tation. The yellow line is the locality value for BERT (around 0.2), and BERT actually does not
4

---

Under review as a conference paper at ICLR 2023
Model
Size
Semtiment Analais
Testual Entailment
Paraphrase ldemification MRPC
Textual Similartt
MR
SUBI
S51-2
ONLI
RIE
MNLI
OOP (755k)
SIS-8
SICK-R
Avg
(22K)
(20K)
(68.8)
(110K)
(5,5K)
(4138)
(5.4K)
(8.4K)
(9.4K)
BERT
110M
2..5
91.4227
864
85841.0
59221.2
78.2
7.5
$8..0.0
77.8
649
77.8
BEREA"
113M
79442.8
98740.6
88010.7
$6311.1
59449.7
28.
88740.4
76.31 8
81.6
BERTA"
138M
78.23..6
93.00..9
+1.1
57.00..5
61041.4
78.9
20.9
89240.3
$43
76.044.7
Table 1: Evaluations of handcrafted encodings across 10 downstream tasks. We report the average score (Spearman correlation for textual similarity and accuracy for others) of five runs using dif- ferent learning rates. means the encodings are learnable and s means that positional encodings are shared within the attention headers of layers.
have an extreme locality. Experimental results on another dataset (Figure 7 show that the accu- racy growth slows down particular locality value (0.3), which means that a perfect locality is unnecessary. The right figure in Figure shows the results for different symmetry values. In this experiment, we vary the symmetry while keeping the locality in the interval [0.15, 0.3). which is close to the value of BERT. Because the change of symmetry will impact the value of locality, we can only observe this type of partial correlation. We find that symmetry affects performance only after a certain value (0.65), and a better symmetry leads to better accuracy. Also, the encodings of the pre-trained BERT are highly symmetrical. Besides, experiments on SUBJ dataset (Pang & Lee 2004) obtain similar conclusions, as shown in Figure 7
We conclude that positional encodings with more suitable locality and symmetry can yield better performance downstream tasks, which may explain why fixed encodings are designed to meet the two properties and why learned encodings all exhibit this behavior. However, encodings are not perfectly local, which might be due to the network architectures and the specific target tasks.
3.3 CAN LOCALITY AND SYMMETRY YIELD BETTER INDUCTIVE BIAS?
Given that locality and symmetry stand out as important learned features of existing positional en- codings, it begs the question that what happens if a language model is initialized with positional encodings with good locality and symmetry.
For this purpose, we replace the positional correlations 8 in Equation 5 with handcrafted Posi- tional Encodings to probe. There are various dessgned positional encodings. e.g.. sinusoidal encodings (Vaswani et al.] (2017), rotary encodings (Su et al. (022)) and ALiBi (Press et al. 2021a), but the locality and symmetry cannot be modified easily for these encodings. To address this issue, propose the Attenuated Encoding, which use a Gaussian kernel (Guo et al. 2019):
exp(os.)) 6., ((l,)) ET exp(az.)
8 w (3, $5$ 1 {
where a..
(9)
where 4, is the relative distance, s is a scalar parameter that controls the symmetry value. Note that there are two key differences between our encodings and other manually designed ones such as the T5 bias (Raffel et al.) 2019) and ALiBi (Press et al., 2021a). First, the output generated by our method is an attentional vector (or a discrete probability distribution) that can be regarded as type of attention mechanism. Thus, we can estimate the locality and symmetry individually. ALiBi biases, in contrast, cannot be measured by our proposed metries directly. Second, we can adjust the hyper-parameters in our method for obtaining encodings with different localities and symmetry, which ALiBi does not allow.
O is a scalar parameter that controls the locality value, and
In this experiment, we adjust the parameter w and for obtaining weight vector 5 that share similar locality and symmetry with pre-trained BERT (Locality=0..1 17 and Symmetry=1.0) After, we pre- train BERT base initialized with 8 and compare them to learned encodings on downstream natural language understanding tasks. Three variants are compared with the original BERT: 1) BERT-A"- uses learnable and shared 8, but the weights are shared inside a particular layer; 2) BERT-A" uses learnable but not shared 8, which means à is different in each attentional head. More details of the datasets and pre-training are shown in the Appendix Section B.6 and Section B.5 respectively. The empirical results are shown in Table I We observe that both BERT-A"- -s and BERT-A can signif- icantly outperform the original BERT, which demonstrates positional encodings with initialization of suitable locality and symmetry can have better inductive bias in sentence representation. Besides,
5

---

Under review as a conference paper at ICLR 2023
Model
Symmetry
Locality
Original
Shuflle (A)
Shufile- (A)
Shufile-S ((A)
Sbuille-6(A)
Random (A)
Original
Shutlle SR (A)
BERT
57.9
16.2
89.8
-0.4
-0.6
-0.3
-0.5
-2.7
89.8
-63.9
ALBERT
82.0
70.3
91.8
-0.5
-1.1
-1.3
-1.7
-6.0
92.0
-66.8
DcRERTa
85.0
-0.5
-0.7
-1.3
-1.1
-5.1
-58.9
XLNN
72.7
17.5
91.5
-0.2
-0.3
-0.7
-1.2
-3.4
91.3
-57.8
StickERT
96.3
7.5
90.9
-0.5
-0.9
-1.3
-1.1
-44
90.8
-44.5
Table 2: Results of Constituency Shuffling and Semantic Role Shuffling, measured by accuracy. Shuffle-z means phrases with length are shuffled. Shuffle-SR means the semantic roles of agent and patient are swapped.
the visualizations of attentional heads in BERT are shown in Figure 8 and the visualizations of 8 are In fact, there is great diversity of behaviors within different attentional heads.
shown in Figure g
3.4
WHAT IS THE DRAWBACK OF SYMMETRY?
Although positional encodings with good symmetry perform well on a series of downstream tasks, the symmetry property has an obvious flaw in sentence representations, which is ignored by prior studies. Existing probes study the sensitivity of language models to word order by shuffling the words in sentence, and they can be roughly divided into three categories: random swap (Pham itt al. 2021 I Gupta et al. 2021 Abdou et al. 22022). n-gram swap (Sinha et al. (022)) and subword- level phrase swap (Clouatre et al.] 2002). All these works assume that the labels of the randomly shuffled sentences are unchanged. However, this is obviously not the case. In particular, the shuffled sentence may have another label (think of the textual entailment example from the introduction).
To address the issue, we propose two new probing tasks of word swaps: Constituency Shuffling and Semantic Role Shuffling. Constituency Shuffling aims to disrupt the inside order of constituents, which is able to change the word order while preserving the maximum degree of original semantics. A constituent parsing case is shown in Figure 3 and we can shuffle the word order inside some phrases, e.g., the noun phrase "an electric guitar" while the seman- tic will not be changed (the grammar structure may be destroyed). We construct different shuffled datasets by phrase length, e.g., "an electric guitar" is phrase of length 3 and we can obtain tri-gram shuffled sets. Datasets constructed by constituency shuffling are re- ferred to as Shuffle-z and æ means the length of phrase. On the other hand, Semantic Role Shuffling intentionally changes the semantics by swapping the order of the agent and patient of sentences and thus results in a new sen- tence with different meanings. In Figure 3 "a man" as the entity that performs the action, technically known as the agent, and "an electric guitar" as the entity that is in- volved in or affected by the action, which is called the patient. We refer to this dataset Shuffle-SR because it swap the semantic roles in sentence. Some shuffled examples are shown in Table
electric gullar
man
playing an electri i guitar on stage
man
plyyng
on electric quiaar
stage
electric
guitor
Figure 3: Illustration of constituent
parsing for one sentence in SNLI "a man playing an electric guitar on stage". The result is generated by Berkeley Neural Parser.
The distinction of our proposed two probing tasks is that one preserves the semanties while another changes the semantics. Then, we can probe the capability of language models to correctly recognize the new sentence's meaning. Specifically, the Stanford Natural Language Inference (SNLI) (Bow- man et al., 2015) dataset is used in this experiment and it provided constituent structure for each sentence. To probe the sensitivity of language models to the two types of shuffling, we fine-tune 5 pre-trained language models with good symmetry on SNLI training set and evaluate them on the newly constructed Shuffle-x and Shuffle-SR datasets (see details in section B.2 The overall results of word swap probing are shown in Table We first observe performances of all language models across Shuffle-z sets basically do not degenerate, which confirms the benefits of the locality and symmetry properties. Second, most models fail on the Shuffle-SR dataset, which demonstrates local symmetry does not capture global position changes well, which explain the reason that BERT fails on the example: "an electric guitar playing by positional encodings can performs well on a series of language understanding tasks, the symme- try itself has obvious flaws. The better performance of StrucBERT the Shuffle-SR suggests that
stage". Although the local symmetry learned

---

Under review as a conference paper at ICLR 2023
introducing additional order-sensitive training tasks may improve this problem. More details of the probing tasks are described in Appendix B.2
4 LINGUISTIC ROLES OF POSITIONAL ENCODINGS
The syntactic structure is crucial to our understanding of sentences. Many studies have shown that a substantial amount of linguistic knowledge can be found in contextual word representations, c.g.. subject-verb agreement [Goldberg] 2019) and dependency tree (Hewitt & Manning 2019; Wu al.)
2020). uollow-up work further systematically analyzed the syntactic and semantic capabilities of the BERT model (Clark et al., 2019; Jawahar et al.) [0199] Lin et al. 2019: Rogers et al., 2020). Nonetheless, the linguistic roles of Positional Encodings are still under-explored. In this section, we discuss what linguistic knowledge PEs have Iearned.
4.1 LINGUISTIC PROBING TASKS
We first conduct an ablation study to check the importance of positional and contextual encodings. The experimental details are shown in Appendix (Section C.2 and we find that the removal of any encodings degrade the model performances (Table[7). We therefore hypothesize that both encodings play a role in sentence comprehension and have different responsibilities.
Positional and contextual weights are usually entangled in every attentional head, and therefore the behavior of positional encodings cannot be observed independently (as shown in Equation (5) To address this, we hide the contextual correlation Yi in Equation (0 pre-trained BERT (instead of removing the contextual encodings completely) and thus the attentional weight 04 only depends on positional correlation. Note that this operation does not alter the structure of the original network because a softmax layer is applied to the vector aj and the output is still an attentional weight vector that can be regarded sentence representation is somewhat decoupled from contextual encoding. We refer to this adapted model as BERT-p. For comparison, we remove the positional correlations 0z to obtain BERT-c We do the same for a pre-trained TUPE model, to obtain TUPE-p and TUPE-c.
kind of discrete probability distribution. Therefore, the output
In this linguistic probing, we adopt widely used 10 probing tasks (Conneau al., 2018) with stan- dard evaluation toolkit (Conneau & Kiela 2018). These tasks are a series of classification tasks that covers three categories: SentLen (Surface), WC (Surface), BShift (Syntactic), TreeDepth (Syntac- tic), TopConst (Syntactic), Tense (Semantic), SubjNum (Semantic), ObjNum (Semantic), SOMO (Semantic), CoordInv (Semantic). The details of this experiment are described in Section
Bertian
TrentAuth
Coprarad
30
Butto,
Buth
Taft
Suldium
Ctthinn
SOMO
Coundiyy
Figure 4: Results of linguistic probing tasks across different layers. BERT-based models are shown in triangle while TUPE-based models are shown in circle. The red, blue and yellow lines represent the use of positional weights, contextual weights and a combination of both, respectively.
7

---

Under review as a conference paper at ICLR 2023
Figure 4 gives the results. We first observe that the combination of contextual and positional encod- ings can have better performances across all probing tasks (yellow lines). Secondly, compared to contextual encodings, positional encodings perform better on syntactic tasks (TreeDepth, TopConst, BShift), which require more information of word orders. On semantic tasks, contextual encodings outperform contextual encodings on Tense and ObjNum while performs poorly when the seman- tic probing tasks require order information (CoordInvv. Thirdly, a hierarchical structure exists here when we check the peak of probing tasks for each model, as observed by Jawahar al. (2019). For surface tasks, the surface knowledge is stored more in bottom layer, syntactic knowledge is in middle layer and semantic knowledge is in middle and top layer. Therefore, we conclude that po- sitional encodings play more of a role at the syntactie level tasks. On semantic tasks, especially position-independeen ones, contextual encodings are more important.
4.2 DEPENDENCY SYNTAX OF POSITIONAL ENCODINGS
Clark et al. 2019) showed that particular heads of BERT indeed correspond remarkably well to particular long-distance dependencies, like ccomp and mark. To test the dependency knowledge stored in positional weights, we follow the syntactic probing test by Clark al. (2019). Specifically, each head in PLMs is regarded as simple predictor of dependency relations. Given the attention weight vector of an input word, we output the word with the highest values and think the pair of words hold some type of dependency relation. While no single head can perform well on all relations, the hest-performed head is selected as the final ability of model for each particular relation. In this experiment, we adopt the original TUPE that uses absolute positional encodings as our base model. The ability of contextual and positional weights evaluated by removing the unrelated encodings in Eq c.g., the first term is removed when checking the syntactic ability of PEs. The two variants are referred as to Contextual Attention (CA) and Positional Attention (PA).
100
Performance (%)
Encoding contextual positional both
We extract attention maps from BERT on the MRPC
75
(Dolan & Brockett 2005) annotated by dependency parser of spaCy We report the results on top-20 de-
50-
pendency relations. Figure5Jshows performance on dif-
ferent relations function of word distance in these rela- tions (Table [ gives relation-specifi results). First, we observe that positional attention significantly more im- portant than contextual attention short-distance de- pendency relations (distance from to 4). Second, contextual attention performs takes the lead on long- distance relations (after 6). Again, the combination of the two features can yield the best performance. The "outlier" in the lower left corner is the Root dependency. Because this relation is self-reflexive edge, contextual (or self) attentions can performs well on it while learned PEs do not attend to the current word itself, e.g., visualizations of BERT and DeBERTa in Figure
25
Distance (in words)
Figure 5: Accuracy of top-20 dependency relations. Detailed results in Table g Ap- pendix.
5 How TO COMBINE CONTEXTUAL AND POSITIONAL FEATURES
As discussed before, the core function of positional encodings is to symmetrically combine local units. As for the linguistic role, the contextual weights are good at long-distance syntactic relations while positional weights are suitable for short dependencies. The combination of the two weights can lead to better sentence representations. The add operation of contextual and positional corre- lation (7.j and 8.,; in Equation (5) is widely used in many PLMs (Devlin et al. 2019. Liu et al.) 2019 Yang al. 2019b Ke et al. 2021). shown in Eq 5 However, to comprehend a sentence, the semantic meaning is composed bottom-up: morpheme - word + phrase + clause - sentence. For example, the two sub- words are composed a particular meaning ("context", ""conte"""""H"ua"""" "#ttua"") - "contextual" and the two words are composed to a phrase ["take"."""" "off""" - "take off" When we reach word take and off, the two words should be composed of phrase first and then its meaning is understood by using the long-distance context. At least, it is not necessary to consider contextual in-
https://spacy.io// io/ htapppppyy....//e///eeeeenneeeer
8

---

Under review as a conference paper at ICLR 2023
Model
Size
Sentiment Analysis
Texuual Entailment
Panphrase Idenificatioo MRFC
Tetuual Similanity
MR (22K)
SLBJ
SS1-2
ONLI
KIb
MNLI
OUP (7553
SIS-a
SICK-R
Aug
(208)
(68.8)
(110K)
(5.5)
(413K)
(5.4K
(8.4K)
(9.4K)
A"
138
9..0
47.....5
6.041.4
789.0.0
4.943.9
3.2....5
84.3.2.5
4..44..7
81.8
A" Seg
138M
78.74-2.4
93.3+0.7
8877-0.3
86340.44
641122.6
78.1+0.4
811+2.11
89.540.3
85.+1.9
793+1.1
825
110M
78.04..1 1
9..1-2.
8.6-1.8
$4440.3
636+3.9
74.+0.3
26.+2.0
0..240.5
4..2
2,3
79.3
ALaBi-Sey
110M
2..
1.221.
$4.400.4
62.02..
76.711.3
76.4110
88.6
$2.1
25.9
80.4
Table 3: Evaluations of our sequence combinations of positional and contextual encodings across 10 downstream tasks. means the encodings are learnable. We report the average score (Spear- man correlation for textual similarity and accuracy for others) of five runs using different learning rates.
formation until we compose basic semantic units like words ( ("contextual") and short phrase ("take of Therefore, key question arises: how to combine the contextual and positional weights?
To combine the contextual and positional correlations better and according to our analysis, we pro- pose a new strategy: Sequence Combination. This method uses two correlations in a particular order. First, the positional correlation ōú, in Equation 5 used to combine nearby tokens for com- posing basic semantic units, c.g., {take, off) take off. Afterward, contextual correlation Y. is applied to these composing units and thus they can be better understood under the contexts. We refer to the separate usage of the two correlations as Contextual Attention and Positional Attention, respectively, and this process can be written
X CA(PA(X))
(10)
The computation of CA is obtained by removing the positional item òú from Equation 5 and PA can be computed through the weighted sum of the input sequence by using 8z,j. A code example of the implementation is shown in Listing 3
In this experiment, we use two positional encodings as baselines: Learned Attenuated Encodings and Fixed ALiBi (Press et al. [02 la). After, the sequence combination is applied to the two baselines for comparison. Note that to make ALiBi suit for our experiment, we apply softmax layer the linear biases to obtain an attentional vector, and thus it can be used individually for positional attention module. The experimental results are shown in Table 3 We observe our proposed sequence com- bination outperforms the original add operation without introducing additional parameters, which shows that prioritizing local semantic unit composition is beneficial for sentence representation. Besides, the sequence combination can achieve lower training and validation loss than the original BERT using the same steps (Figure (6). A conclusion is that we can subtly adjust the sequence of using positional encoding without introducing additional parameters obtain certain degree of improvement in the language understanding tasks.
6 CONCLUSION
We have proposed series of probing analyses for understanding the role of positional encodings in sentence representations. We find two main properties of existing encodings, Locality and Sym- metry, which is correlated with the performance of downstream tasks. Meanwhile, we point out an obvious flaw of the symmetry property. We further complement existing research by distinguish- ing the linguistic roles of positional and contextual encodings, and by proposing combine them sequentially rather than additively.
The limitations of this work are two-fold. First, our analysis is limited to the natural language understanding of the English language. Different languages display different word order properties. For instance, English subject-verb-objeee order (SVO) while Japanese is subject-ohject-ver order (SOV), and natural language generation models are not included in this work. Besides, a recent work finds that the autoregressive models without any explicit positional encoding are still competitive with standard models (Haviy et al.) 2022), which shows the generative model might not be a perfect target for researching order information. Second, although our handcrafted positional encodings satisfy the Symmetry property, it merely replicates the limitations of current positional encoding, albeit in a simplified form. Further architecture development should address the problem of the "an electric guitar playing a man stage." mentioned in the introduction.

---

Under review as a conference paper at ICLR 2023
7 REPRODUCIBILITY STATEMENT
The complete experimental settings, as well as the implementation details, are given in Section B Besides, we have submitted our source code and will make it publicly available.
REFERENCES
Mostafa Abdou, Vinit Ravishankar, Artur Kulmizev, and Anders Søgaard. Word order does mat- ter and shuffled language models know it. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 6907-6919, 2022.
Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg, Fine-grained analysis of sentence embeddings using auxiliary prediction tasks. In International Conference on Learning Representations, 2017. URL https:
Luisa Bentivogli, Ido Kalman Dagan, Dang Hoa, Danilo Giampiccolo, and Bernardo Magnini. The fifth pascal recognizing textual entailment challenge. In TAC 2009 Workshop. no publisher, 2009.
Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. A large anno- tated corpus for learning natural language inference. EMNLP, 2015.
Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. Semeval-2017 task I: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the I Ithh International Workshop on Semantic Evaluation (SemEval-20117 20)7), pp. 1-14, 2017.
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning. What does bert look at? an analysis of bert's attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 7766-26,,200 2019.
Louis Clouatre, Prasanna Parthasarathi, Amal Zouaq, and Sarath Chandar. Local structure matters most: Perturbation study in nlu. In Findings of the Association for Computational Linguistics: ACL 2022, pp. 3712-3731, 2022.
Alexis Conneau and Douwe Kiela. Senteval: An evaluation toolkit for universal sentence repre- sentations. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), 2018.
Alexis Conneau, German Kruszewski, Guillaume Lample, Loic Barrault, and Marco Baroni. What you can cram into a single vector: Probing sentence embeddings for linguistic properties. In ACL 2018-56th Annual Meeting of the Association for Computational Linguistics, volume 1, pp- 2126-2136. Association for Computational Linguistics, 2018.
Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textual entailment challenge. In Machine learning challenges workshop, pp. 177-190. Springer, 2005.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V Le. and Ruslan Salakhutdi- nov. Transformer-xl: Attentive language models beyond fixed-length context. arXiv preprint arXiv:1901.02860, 2019.
Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser, Universal transformers. In International Conference on Learning Representations, 2018.
Jacob Devlin, Ming- Hiig-Wei oe Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp.
2019.
Bill Dolan and Chris Brockett. Automatically constructing corpus of sentential paraphrases. In Third International Workshop on Paraphrasing (/WP2005), 2005.
Philipp Dufter, Martin Schmitt, and Hinrich Schütze. Position information in transformers: An
overview. Computational Linguistics, 48(3):733-763, 2022.
10

---

Under review as a conference paper at ICLR 2023
William Edward Dyer. Minimizing integration cost: A general theory of constituent order. PhD thesis, University of California, Davis, 2017.
Fernanda Ferreira, Karl GD Bailey, and Vittoria Ferraro. Good-enough representations in language comprehension. Current directions in psychological science, 11(1):11-15, 2002.
Richard Futrell, Roger P Levy, and Edward Gibson. Dependency locality as an explanatory principle for word order. Language, 96(2)3371-412, 2020.
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann Dauphin. Convolutional sequence to sequence learning. In International Conference Machine Learning, pp- 1243- 1252. PMLR, 2017.
Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and William B Dolan. The third pascal rec- ognizing textual entailment challenge. In Proceedings of the ACL-PASCAL workshop on textual entailment and paraphrasing, pp. 1-9, 2007.
Edward Gibson, Leon Bergen, and Steven Piantadosi. Rational integration of noisy evidence and prior semantic expectations in sentence interpretation. Proceedings of the National Academy of Sciences, 110(20):8051 8056, 2013.
Yoav Goldberg. Assessing bert's syntactic abilities. arXiv preprint arXiv:1901.05287, 2019.
Maosheng Guo, Yu Zhang, and Ting Liu. Gaussian transformer: lightweight approach for natural language inference. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 6489-6496, 2019.
Ashim Gupta, Giorgi Kvernadze, and Vivek Srikumar. Bert & family eat word salad: Experiments with text understanding. In Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 35, pp. 12946-12954, 2021.
Bar Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. The second pascal recognising textual entailment challenge. In Proceedings of the Second PASCAL Challenges Workshop on Recognising Textual Entailment, volume 7, 2006.
Adi Haviv, Ori Ram, Ofir Press, Peter Izsak, and Omer Levy. Transformer language models without positional encodings still learn positional information. arXiv preprint arXiv:2203. 2022.
Pengcheng He, Xiaodong I Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhanced bert with disentangled attention. In International Conference on Learning Representations, 2020.
John Hewitt and Christopher D Manning. A structural probe for finding syntax word representa- tions. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Pa- pers), pp. 4129 4138, 2019.
Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. What does bert learn about the structure of language? In ACI. 2019-57th Annual Meeting of the Association for Computational Linguisties, 2019.
Guolin Ke, Di He. and Tie-Yan Liu. Rethinking positional encoding in language pre-training. In International Conference on Learning Representations, 2021. URL h/tp:::///ppeeeview. net./forn dimmmmiii=0-900 528y
Urvashi Khandelwal, He He, Peng Qi, and Dan Jurafsky. Sharp nearby, fuzzy far away: How neural language models use context. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 284-294, 2018.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 14Xiv:1412.66 19800 2014.
Dan Klein and Christopher D Manning. Accurate unlexicalized parsing. In Proceedings of the 4 annual meeting of the association for computational linguistics, pp. 423-430 2003.

---

Under review as a conference paper at ICLR 2023
Roger Levy. A noisy-channel model of human sentence comprehension under uncertain input. In Proceedings of the 2008 conference on empirical methods in natural language processing, pp- 234-243 2008.
Yongjie Lin, Yi Chem Tan, and Robert Frank. Open sesame: Getting inside bert's linguistic knowl- edge. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neu- ral Networks for NLP, pp. 241-253, 2019.
Haitao Liu, Chunshan Xu, and Junying Liang. Dependency distance: A new perspective syntactic patterns in natural languages. Physics of life reviews, 21:171-193, 2017.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1907. J 1692, 2019.
Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli. A sick cure for the evaluation of compositional distributional semantic models. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC' 14), pp. 216-223, 2014.
Francis Mollica, Matthew Siegelman, Evgeniia Diachek, Steven T Piantadosi, Zachary Mineroff, Richard Futrell, Hope Kean, Peng Qian, and Evelina Fedorenko. Composition the core driver of the language-selective network. Neurobiology of Language, 1(1):104-33, 2020.
Bo Pang and Lillian Lee. A sentimental education: Sentiment analysis using subjectivity summa- rization based on minimum cuts. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACI-04), pp. 271-278, 2004.
Bo Pang and Lillian Lee. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL (CC,, pp. 115-124, 2005.
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc-Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset: Word prediction requiring a broad discourse context. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1525-1534, 2016.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532- 1 1543, 2014.
Thang Pham, Trung Bui, Long Mai, and Anh Nguyen. Out of order: How important the sequen- tial order of words in a sentence in natural language understanding tasks? In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 1145 1160, 2021.
Ofir Press, Noah Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. In International Conference on Learning Representations, 2021a.
Ofir Press, Noah A Smith, and Mike Lewis. Shortformer: Better language modeling using shorter inputs. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the Ith International Joint Conference on Natural Language Processing ( (Volume I: Long Papers), pp. 5493-5505 2021b
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910. 100.:1910..0083. 2019.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrew, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2383-2392, 2016.
12

---

Under review as a conference paper at ICLR 2023
Nils Reimers and Iryna Gurevych. Sentence- bert: Sentence embeddings using siamese bert- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-
IJCNLP), pp. 3982-3992, 2019.
Anna Rogers, Olga Kovaleva, and Anna Rumshisky. A primer in bertology: What we know about how bert works. Transactions of the Association for Computational Linguistics, 8:842-866, 2020.
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representa- tions. arXiv preprint arXiv:1803.00 155, 2018.
Xing Shi, Inkit Padhi, and Kevin Knight. Does string- neural mt learn source syntax? In Proceedings of the 2016 conference on empirical methods in natural language processing, pp- 1526-1534, 2016.
Vighnesh Shiv and Chris Quirk. Novel positional encodings to enable tree-based transformers. Advances in neural information processing systems, 32, 2019.
Koustuv Sinha, Robin Jia, Dieuwke Hupkes, Joelle Pineau, Adina Williams, and Douwe Kiela. Masked language modeling and the distributional hypothesis: Order word matters pre-training for little. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 1888-29133 2021.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language pro- cessing, pp. 1631-1642, 2013.
Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. arXiv preprini 1xXiv:2104.0986: 2021.
Sho Takase and Naoaki Okazaki. Positional encoding to control output sequence length. arXiv preprint arXiv: 1w0.i::904.0744 07410, 2019.
David Temperley and Daniel Gildea. Minimizing syntactic dependency lengths: Typological/cogni- tive universal? Annual Review of Linguistics, 4:67-80, 2018.
Matthew J Traxler. Trends in syntactic parsing: Anticipation, bayesian estimation, and good-enough parsing. Trends in cognitive sciences, 18(1 1):005-611, 2014.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 353-355, 2018
Benyou Wang, Donghao Zhao, Christina Lioma, Qiuchi Li, Peng Zhang, and Jakob Grue Simon- sen. Encoding word order in complex embeddings. In International Conference on Learning Representations, 2019.
Benyou Wang. Lifeng Shang, Christina Lioma, Xin Jiang, Hao Yang, Qun Liu, and Jakob Grue Simonsen. On position embeddings in bert. In International Conference on Learning Represen- tations, 2020.
Yu-An Wang and Yun-Nung Chen. What do position embeddings learn? an empirical study of pre- trained language model positional encoding. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6840-6849, 2020.
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad coverace challenge corpus for sen- tence understanding through inference. In Proceedings of the 2018 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long Papers), pp. 1112-1122, 2018.
13

---

Under review as a conference paper at ICLR 2023
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, et al. Transformers: State-of-the-ar art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp- 3 38-45, 2020..
Zhiyong Wu, Yun Chen, Ben Kao, and Qun Liu. Perturbed masking: Parameter-free probing for analyzing and interpreting bert. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4166-41766 2020.
Baosong Yang. Longyue Wang, Derek F Wong, Lidia S Chao, and Zhaopeng Tu. Assessing the ability of self-attention networks to learn word order. In Proceedings of the 57th Annual Meeting of the Association for Compulational Linguistics, pp. 3635-3644, 2019a.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, 32, 2019b.
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. Proceedings of the IEEE international conference on computer vision, pp. 19-27, 2015.
A DETAILS OF TUPE MODEL.
In absolute positional encoding, the positional encoding is added together with the contextual en- coding:
x; Pa W ((x, + +Pj) WK)T
Oijj
(11)
Vd
wehre Ps Id is position embedding of the i-th token. Further, the above equation can be
expanded as:
Ogj- (x. (x, (p,
(x, +
(D. (p,
Vd
Vd
Vd
(12)
There are four terms in this expression: context-to-comtett. context-to- to-positio, position- to-comeex,, and position-to- teantttt--oosst to-position. While the first and the fourth term are intuitive, the token encodings and positional encodings do not have strong correlations with each other, and the context- pooiixtiooiiionn corre- lations may even induce unnecessary noise. Based on this analysis, Ke et al. (202I) propose TUPE (Transformer with Untied Positional Encoding) that removes the second and third redundant terms and introduces different parameters for the position encoding:
(x, (xxx)) W + (p. (p,
Oij
(13)
Vd
Here, Uo and UK are weights that need learned, capturing positional queries and keys, re- spectively. Their empirical results confirm that the removal of the two caneext-to-positin terms consistently improves the model performance on series of tasks.
B DETAILS OF EXPERIMENTS
B. VISUALIZATIONS OF POSITIONAL ENCODINGS
To understand what positional encodings learn after pre-training, we visualize the positional weights in attentional heads. The Identical Word Probing adopted in this experiment (Wang et al.) 2020). The used pre-trained language models are shown in Table 4 and the repeated words are randomly selected from the corresponding vocabulary. Note that sub-tokens like single characters and non- physical words removed. For visualization, adopt the Identical Word Probing proposed
14

---

Under review as a conference paper at ICLR 2023
Model
Size
Version
Language
BERT
110M
base--mcaedd
English
DeBERTa
100M
microsofVäebertt base
English
XLNet
110M
sleet-base-cased
English
Table 4; Details of pre-trained language models used in visualizations.
Original
Shuffled
Shuffled-3
An old
with a package
poses front
of on advertisement
An mos old
with packuge a
poses fronz
advertizement on
Shufiled-4
A land rover is being
driven across a river
A land rover being
driven river actoss
Shuffled-5
A mon recals the paper in hor with green lighting
mon reads the paper in with groen lighting bar
Shuffled-6
A litll boy in gnay and white striped swester and tast is play-
limll boy
striped swenter and white and tan play-
piece of playground equipment
ing
playground of equipmeni
Shuffled-SR
several playing volleyhall
volleyhall
are playing
severaa
Shuffled-SB man and wocann are sharing
a boodog
a hoodog
are sharing
a man and woman
Table Some cases of the shuffled SNLI datasets in our word swap probing. Texts in the same color mean the corresponding phrases.
by Wang et al. (2020), which feeds many repeated identical words to pre-trained language models and thus the attention values (as,; in Equation 3 are disentangled with contextual weights. More specifically, we randomly select 100 words from the corresponding vocabulary (filtering out single characters and sub-words such as "# #nd"). We repeat each word to compose a sentence of length 128. These 100 sentences are fed into language model and the attention weights across different layers are averaged as the positional weight matrix of particular language model.
B.2 WORD SWAP PROBING
To valid language models with positional encodings are sensitive to the local and global word swaps, we construct Shuffle-r and Shuffle-SR SNLI datasets. Shuffle-r means the word orders of phrases with length x are disrupted, c.g. "an electric guitar" is a 3-gram phrase, and it might be "guitar electric" in Shuffle-3 SNLI. Through this way, new sentence with the same meaning can be obtained and therefore the initial label of the sample will not be changed. To construct such shuffled datasets, the premise sentences in SNLI test set are shuffled and we keep the hypothesis sen- tences intact. Here, we let x E [3, 6) and select a subset from SNLI to make sure that every premise sentence has at least one phrase with length from 6. We select five types of target phrases for shuffling: Noun Phrase, Verb Phrase, Prepositional Phrase, Adverb Phrase, and Adjective Phrase. Finally, a Shuffle-r SNLI obtained by disrupting the order inside phrase with length æ and the size for each shuffle-r is around 5000. The first fourth rows in Table 5 shows some samples.
As for the Shuffle-SR SNLI dataset, the semantic roles of agent and patient are swapped in sen- tence. We use the Algorithm [ to collect a subset from SNLI test set. This algorithm is applied successively to the premise and hypothesis sentence for a sample whose label is entailment, and if the result of either of them is not null, we consider it a valid shuffled sample, which means we only shuffle the premise or hypothesis. After, we can obtain a new sample and the pair of sentences contradicted with each other. In total, there are 1329 samples. To ensure that all sentences semantically correct, we manually selected 300 pairs from them. The last two rows in Table5]shov shows two examples in Shuffle-SR dataset.
To probe the capabilities of language models on our newly constructed datasets, we adopt five dif- ferent pre-trained language models (as shown in Table and use Hugging Face for implemen- tation (Wolf et al. 2020). These models are fine-tuned on the training set of SNLI, and the model with the best score validation set is stored for the follow experiments. Note that there are off-the-
15

---

Under review as a conference paper at ICLR 2023
Algorithm 1: Construction of ShuffleSR Sentences
Input:
s: premise or hypothesis sentence in SNLI.
A: Auxiliary verb list
M: Semantic Role Labeling Model,
D: Subject and Object Case Mapping // e.g., I < me
Output: A sentence s* with shufiled agent and patient or None
R Predict the semantic roles of words in sentence by using the model M
V + Take the verb list from R
foreach verb in V do
if'v appears in A then
continue
if R. does not contain an agent or patient then continue
a. p
Take the agent and patient from R
S" Swap the o, in sentence 8
10
Transform the subject and object case in s* if a or p D
11
return s*
12 return None
shell ALBERT and XLNet for natural language inference, we therefore use them directly without fine-tuning. During fine-tuning stage, the maximum length of the tokenized input sentence pair is 128, and the optimizer is Adam (Kingma & Ba 2014) with learning rate of 2e-5. The batch size is 32 and the epoch is 3. After fine-tuning, the best model evaluated on shuffle SNLI test set, an we record their performances when faced with local and global word swaps.
B.3 LINGUISTIC DISCUSSIONS OF LOCALITY AND SYMMETRY
Locality means that the positional weights favor the combination of units in sentence to their ad- jacent units when creating higher-level representations. For example, sub-tokens can be composed into lexical meanings (c.g., {"context", "#Hual") } + "contextual") or words can be composed into phrase-level meaning (e.g.. ("take" "oakke""""f"") - "take of and clause-level and sentence-level mean- ing can be obtained through an iterative combination of low-level meanings, which is consistent with the multi-layer structure in pre-trained language models. From a linguistic perspective, words linked in syntactic dependency should be close in linear order, which forms what can be called a depen- dency locality (Futrell et al., 2020j. Dependency locality provides potential explanation for formal features of natural language word order. Consider the two sentences "John throws the trash" " and "John throws the trash out". Both are grammatically correct. There a dependency re- lationship between "throws" " and "out" and the verb is modified by the adverb. However, language users prefer the expression with the first sentence because it has a shorter total dependency length (Dyer 2017 Liu al. 20177] Temperley Gildea 2018). Based on the visualizations and depen- dency locality, we, therefore, speculate that one main function that positional encodings have learned during pre-training is local composition, which exists naturally in our understanding of sentences. Empirical studies also demonstrate that performances of shuffled language models are correlated with the violation of local structure (Khandelwal et 2018 Clouatre et al. 2022).
The symmetry (also observed by Wang & Chen (2020); Wang et al. (2020)) of the positional matrices implies that the contributions of forward and backward sequences are equal when combining adja- cent units under the locality constraint. This is contrary to our intuition, as the forward and backward tokens play different roles the grammar, as we have seen in the examples of "a man playing an electric guitar stage" and "an electric guitar playing man stage". However, this symmetry less disruptive at the local level inside sentences. Recent work in psycholinguistics has shown that sentence processing mechanisms
well designed for coping with word swaps (Ferreira al.) 2002 Levy 2008 Gibson et al. 2013) Traxler 2014). Further, MMolica et al. (2020) hypothesizes that the composition process is robust to local word violations. Consider the following example:
16

---

Under review as a conference paper at ICLR 2023
their last day they were overwhelmed by farewell messages and gifts
b. on their last day they were overwhelmed by farewell and messages gifts farewelll C. on their last they day were overwhelmed farewell messages by and gifts
The local word swaps (colored underlined words) are introduced in the latter two sentences, leading to less syntactically well-formed structure. However, experimental results show that the neural re- sponse (fMRI blood oxygen level-dependent) in the language region does not decrease when dealing with word order degradation (Mollica al., (020),, suggesting that human sentence understanding is robust to local word swaps. Likewise, symmetry can be understood in this way: when reader processes word in sentence, the forward and backward nearby words are the most combinable, and the comprehension of this composition is robust to its inside order. On the other hand, symmetry not an ideal property for sentence representations (consider the case of "an electricity guitar"), and we show the flaws of symmetry in the word swap probing task in Section 3.4
B.4 DETAILS OF POSITIONAL ENCODERS
To probe the correlations between the two properties and downstream tasks. We introduce fully position-based based encoder, which is adapted from the self-attention encoder. The key difference is that the attentional weights in the encoder are our handcrafted attenuated encodings, therefore, the locality and symmetry can be adjusted easily and we can observe the correlations caused by the changes of the two properties. An implementation example of this positional encoder shown in Listing 2
In this experiment, two sentence-level datasets, MR (Pang & Lee 2005) and SUBJ (Pang Leej 2004), are used for evaluation. we use single-layer and single-head positional attention as the encoder and the handcrafted encodings are fixed during training. We use the 840B-300d Glo Pen- mington al. (2014) vectors word embeddings. For training, we use Adam optimizer with an initial learning rate 0.002, and we use a decaying strategy to decrease the learning rate. We adopt dropout method after the encoder layer, and train models to minimize the cross-entropy with dropout rate of 0.5. We train 5 epochs for each model and select the best model on dev sets to evaluate on test set. We repeat this procedure 5 times and use the average score report.
B.5 DETAILS OF PRE-TRAINING
We use the configuration of the original BERT base Devlin et al. 2019) with 11M parameters for pre-training. Our model is implemented with PyToorch using the pytorchic-bert too The number of layers, attention heads, and the projection dimension are 12. 12, and 768 respectively. We use the original vocabulary with a size of 30522. The training corpus the English Wikipedia (20200101 dumps), which totals 13G after preprocessing by WikiExtractor. We pre-train with sequences of at most T 512 tokens, and set the batch size as 64 to pre-training 600K steps. The optimizer is Adam with learning rate of 5e-4, ,31 = 0.9, f22 = 0.999, 1.2 weight decay of 0.01, and warmup rate 0.1. The dropout probability is always set as 0.1.
We use the original BERT base as our backbone and vary the positional encodings to pre-training different variants for comparison.
Listing [ shows code example about how to inject handcrafted positional encodings to the BERT backbone. Each variant is fine-tuned on the training dataset with different learning rates (among 9c-5, 7e-5, 5c-5, 3e-5, Ic-5). After, we evaluate the fine-tuned model the Dev set and report the average score of five learning rates. Apart from BERT, we introduce the TUPE model as another baseline. Specifically, we pre-train the following variants:
BERT is the original one and we use it as baseline.
BERT-A* and BERT-/" are varaints of the former two, but the encodings are learnable during pre-training.
BERT-A* -s shares learnable positional encodings within a layer.
17

---

Under review as a conference paper at ICLR 2023
Model
Size
Version
Fine-tuned by us
BERT
110M
ALBERT
223M
penolulbeettxxxarrg v2 snli mnnii mnnlifeeerr anti R1 J3nii
x
DeBERTa
100M
microsoffilleeern--uusee
XLNet
340M
milehlhhee-arrg--cooe--iiii.. nonti feeer aeell RI R2 R3-nii
X
StrucBERT
340M
beyartoggkrrccberttoo large
Table 6: Details of pre-trained language models used in word swap probing.
BERT-only-c is for ablation study, and the positional encodings Pt in Equation 2 are re- moved.
BERT-only-p is for ablation study, and the contextual encodings x; in Equation 2 are moved.
BERT-A -Seq combines the two features in a sequential way, and the positional attentions are first used and then contextual attentions.
ALiBi adds linear biases to contextual weights proposed by Press et al. (202 and we apply a softmax layer to the original biases for obtaining attention weight vector.
ALiBi-Seq uses the same biases with ALiBi but combines the two features in sequential way.
Suppose that the hidden dimension is 768, the layer number is 12, the head number is 12, and the maximum length 512 for BERTbase model, we can calculate the size for each variant. The number of parameters of handcrafted positional encoding for each head is 262K (512 512). If positional heads are different across all layers, the total cost 37.7M (512 x 512 x 12 x 12). If the positional encodings are shared across heads, the total cost is 3.1M (512 x 512 X 12).
class RoItLHeadeddee1chc (ac. Hodule): ane Headed init (aelf, config):
Peudumt Allent fun ...
supor (1. init__n wilf.n_heeds config haheade
selc.dros Dn.prot teanfig, p_dros_atta)
selt.9rei_ Linear isonfig. coafiq.dim)
pelf.rrj.k an.hinear toun fag fag.diii din, config-din aelt.grolv on. Linein foonfig.din, confiq. din)
dar forward/self, & mask, pa]:
glquesy), (lay), vivaluet (B(bstol lem))_ (idin) nask (Bibatel sen)) per posit ponitional weights 18 (Btbeschusisel sise),, R(leed Sssee len), 8 Sene Jent)
split brdin) into [8 (eadd)) Wisidth of head)) D. H. ean
(B, S. D) proj & 18, 5, DI split à 18, 8, H, trans 18, H, 8, 60 q. ha v-- sull.proj_g(t)
seif.pcoj__Ma self.pse (split last teel feeliinn heade, -113 -1t).trarapose 2)
in (a, vi)
(B, B, 8, k.transpose -11 np.egrt (kaaize(-11)
[B, H, B1 (B, B, 8, 81 -aofcnax->> (8, 8,
scores q
injact pesitional weights contextual veights Ch, H. 5, 8) (8, H, 5, 18, HI, 8, scores scores
if
mask mank None, None, flos scores - 10000.0 + C..3 nask)
se1f.drqp (F.aaaapp..fffaaaaaaaaa. aufunax CB, H. 5, 5) [8, H. 5, XI 18, H. S, -trane-> 18, 8, H. 81
dine 1)1
(scores E vt.. fh, an.enniiuuuus
-mezge-> (B, S,
nerge_
return
Listing A code example of how to inject handcrafted positional encodings into self attentoten
18

---

Under review as a conference paper at ICLR 2023
To inject handcrafted positional encodings, pre-compute the positional weights and add them to the contextual weights directly, as shown in Listing [ These weights can be either frozen or learnable during pre-training. The code of the sequence combination is shown in Listing
B.6 DETAILS DOWNSTREAM DATASETS
SentEval is based on a set of existing text classification tasks involving one or two sentences as input. However, most tasks in SentEval are closely related sentiment analysis and thus diverse enough. GLUE benchmark introduces series of difficult natural language understanding tasks while some particular tasks only contain one dataset, e.g., sentiment analysis and textual similarity. Moreover, the size of WNLI in GLUE is rather small and the GLUE webpage notes that there are issues with the construction of this dataset To better evaluate the capability of models for sentence representation, we, therefore, select 10 datasets from SentEval and GLUE, covering four types of sentence-level tasks:
Sentiment Analysis is also known as opinion mining, which aims to classify the polarity of given text, whether the expressed opinion is positive, negative, neutral. We use MR (Pang & Lee 2005). SUBJ (Pang & Lee 2004) and SST (Socher et al. 2013) for this task.
Textual Entailment describes the inference relation between a pair of sentences, whether the premise sentence entails the hypothesis sentence. Actually, this is a classification task with three labels: entailment, contradiction, and neutral. Here, we use QNLI (Rajpurkar
let al. 2016), RTE (Dagan et al. 2005 Haim et al. [2006] Giampiccolo et al. 2007 Ben- tivogli al. 2009) and MNLI (Williams al. 2018) for evaluation. Note that we report
the average score for the two test sets of MNLI.
Paraphrase Identification to determine whether a pair of sentences have the same mean- ing. We use MRPC (Dolan & Brockett 2005) and QQP for evaluation.
Textual Similarity deals with determining how similar two pieces of texts are. We use STS-B (Cer et al. 2017) and SICK-R (Marelli al. 2014) for evaluation.
class Molei HeadPosi tonelArtent Modules e
whe Readed Soaled Produmt AlLent e.e
det init (selt, contigl:
super (1... init beede config a_heada
selt.drop Dropoot roontig.p. p.rpppppppppppppee aten:
forward (self,
Xs giquery), kikey), vivaiuet 1 (Bibstch_size), Stoeq_Lon), Dieim) mask (Bibatch sixel x Sieeg lenil
poxit weights (Ribesch_xise) (RiR_nesde), Wixidth seadr)
number)), $(seg_len), 2ent)
eppit Disin)
(B, S, D) proj 18, 5, at split > 18, 5, Ht KI irann > 18, H, 8, R) (split_last ieeatt ttraspppees
"a. k,
(2, H, 8, N) 18, H. N. S1 +> (B, B. S, -softnax-> (B, 8,
it not HEnD: scores.nasced 1iLL mask, 0.1
IB, (scores vt.cronepose cl.contigueus(
-trone->
-mezac-> (B, 5, DJ
return merge_lasttif
Listing 2: A code example of the Positional Attention.
httrass/qquubenchnakkkcoooooo
19

---

Under review as a conference paper at ICLR 2023
C ADDITIONAL EXPERIMENTS
C.I LOSS CURVES OF PRE-TRAINING
Apart from the performances downstream tasks, the loss variants. For this goal, the training loss and validation loss are stored after certain steps. We use hold-set as the validation set. As shown in Figure our proposed BERT-A" and BERT-A* "Seq have smaller loss than the original BERT. This can be observed again on the validation set.
also checked for different
C.2 ABLATION STUDY OF POSITIONAL AND CONTEXTUAL ENCODINGS
To check the importance of positional and contextual Encodings, we conduct an ablation study. For this goal, the contextual encodings X; or positional encodings p, in Equation 2 are removed, respectively, during pre-training and the two new models are evaluated on 10 sentence-level datasets. As shown in Table the BERT-only-c and BERT-only-p both lag behind the original BERT models, which means the combination the two features is beneficial for sentence representations. On the other hand, positional encodings more important for sentiment analysis, and the cross-ates coss-attentions from contextual embeddings matter in sentence-pair tasks.
Midel
Scntiment Analysis SURI
Texmal Fatailment
Paraphmse Idemiiication MRPC (54K)
Texnal Similarity
QNLI (110K)
MNLI (413K)
SIS-B
SICK-R
(25K)
(20K)
(68.8K)
(5.5K)
(755k)
(8.4K)
(9:4K)
2..5
91.022.7
1.D8
59.21.2
7822
1
$8.7.2..6
77.84.1
64.-6..0
BERT-onl--
73.014.6
38.9 2..5
82910.4
82.01..1
627 14.3
70.8
74.10033 a
86.9 20.5
76.2 78.5
61.5
BEKT-ooly
73.814.0
90811.4
84.010.7
7.
30.91.4
68.311.0
13.9
85.8.0.6
47.1118.0
31.229.2
70.6
Table 7: Ablation study across 10 sentence-level tasks. We report the average score of five runs using different learning rates.
Sequence Module) Sequence Block
daf init (aelf, config): init_ pud.ee contig.poo__nit
self. sos_soosaeearbbeee p..fee.po__laazniie.
solt.self.n sol__acctionnn ptl..oooooiiooallttteeeee
ioneliiontttttttiooo toenfig, lessnablewaell pos_leessabble)
der tocwerdiscis, x, mask):
pos mositlnnal attention
self.positional, aatiienaaaatttttooooe nask)
cont uall attent
aa = sulf wulf.swif_ne aattffatttnnniooee mask)
return sa
Listing 3: A code example of the Sequence combination of positional and contextual features.
C.3 RESULTS OF LINGUISTIC PROBING
To understand the linguistic role of positional encodings, we adopt the probing tasks proposed by Conneau et al. (2018), which feeds sentence representations obtained by neural models series of linguistic classification tasks. The sentences for all our tasks are extracted from the Toronto Book Corpus Zhuu al. 2015), pre-processed by Paaperno al. (2016) and parsed by using the pre-trained PCFG model Kleinn& & Manning [003). We use the standard SentEval (Conneau & Kiela 2018) 8) to test every model. For obtaining sentence representations, we apply the mean pooling strategy to the output of each layer of frozen language model, which is suggested by Reimers & Gurevych (2019). We follow the standard setup and run a grid search for selecting the layer with the best score the dev set, and report the best score across layers for each probing task (Table 9 which represents the linguistic ability of each model. The following are the details each probing task, including three categories:
SentLen (Surface) aims to predict the length of sentences terms of the number of words, and the dataset is constructed following Adi al. (2017).
20

---

Under review as a conference paper at ICLR 2023
Relation
Distance
conrermal
postiional
both
Root
0.0
99.3
86.5
1.1
44.6
91.1
92.9
surgoudd
70.6
aux
1.3
25.2
79.1
nummod
1.3
26.8
78.9
75.5
amod
1.4
19.7
69.3
66.9
det
$1.6
edvnod
21
58.7
pobj
2.3
9.0
33.9
46.3
neub:
2.3
134
58.2
52.6
poss
dobj
2.5
20.0
41.6
prep
26
23.1
62.8
63.4
apadrmod
3.1
43.8
mark
41
64.4
coc.
4.9
25.1
31.2
33.6
7.3
25.9
30.3
32.7
ccomp
8.1
29.0
33.2
long
54
24.7
21.0
36.8
Macro Avg
27.5
46.0
55.4
Table 8: Evaluations of predictions of dependency relations on MRPC dataset. The top-20 common relations are shown. The distinction of "short" " and "long" is whether the average length of the relation greater than
WC (Surface) means word content, which checks whether it possible to recover informa- tion about the original word from the embedding of the sentence.
BShift (Syntactic) means bigram shift. In this task, two random adjacent words in a sen- tence are swapped and the goal is to detect if model sensitive legal word orders.
TreeDepth (Syntactic) tests whether a model can infer the depth of the syntactic tree of sentences.
TopConst (Syntactic) tests whether a model can recognize the top constituents of the sen- tence, e.g., "/Then] [very dark gray letters on black screen] [appeared] [-]" has top constituent sequence: "ADVP NP VP This dataset is first introduced by Shi et al. (016))
Tense (Semantic) asks for the tense of the main clause verb.
SubjNum (Semantic) focuses on the number the subject the main clause
Obj (Semantic) tests for the number of the direct object of the main clause.
SOMO (Semantic) checks the sensitivity of model random replacement of noun or verb.
CoordIny (Semantic) tests whether model can recognize the order of clauses is inverted.
C.4 DEPENDENCY ANALYSIS OF POSITIONAL ENCODINGS
The detailed scores of each relation are shown Table find that positional attentional heads outperform contextual heads short-distance relations, e.g., auxpass and compound. Contex- tual attention can capture better long-distance relations than positional attention while contextual attention itself has a certain degree of ability to detect some long-distance relations such conj and punct. Note that there exists a head in contextual attention maps attending the token itself, therefore, the score on the Root relation is the best.
C.5 VISUALIZATIONS
In Figure I we visualize the averaged positional weights of various pre-trained language models and identify they have similar visualized results. However, We find that the behavior of positional en- codings is very diverse across attention heads. Note that there are 144 attentional heads (12 layers
21

---

Under review as a conference paper at ICLR 2023
Surface
Syutactic TopConst
Semantic ObjNum
Model
SentLen
WC
TreeDepth
BShift
Touse
SubjNum
SOMO
Coordlay
BERT
92.55
23.54
38.75
77.21
77.67
85.30
84.50
83,94
52.43
61.22
BERT-c
32.86
BERT-p
89.64
38.27
33.78
63.15
53.68
67.17
71.17
66.73
51.19
57.42
TUPE
90.08
85.76
39.14
78.38
79.44
87.13
86.50
86.36
53.78
62.60
TUPE-c TUPE-p
91.85
62.42
34,21
58.63
50.86
83.43
80.63
81.96
51.75
50.74
37.74
75.32
82.13
80.40
51.96
Table Linguistic role probing of positional encodings. c and p means only using contextual and positional weights, respectively.
12 heads) for the BERT hase model. For example, the visualizations BERT (Figure validate this phenomenon. Besides, We observe that BERT exhibits hierarchical structure: positional weights of lower layers are nearly uniform (Layer-4), middle layers attend more to local units (Layer-7) and higher layers demonstrate the asymmetric property (Layer-12). We also visualize all the positional heads in BERT-A (Figure 9 and BERT-A" (Figure 10).
Visualization Analysis of BERT-A". BERT-A" outperforms BERT by 3.1 percentage points on average across 10 downstream tasks. The main difference between BERT-A and BERT is the learnable handcrafted positional encodings. For visualizations, we take the positional weight 6,, in Equation instead of Identical Word Probing.. Figure shows Figure[shows that most positional heads perfectly satisfy the properties of locality and symmetry, which can bring better inductive bias for sentence representations. Another observation is that the diagonal bandwidths are diverse across positional heads after learning, which means proximity units can combined at different distances. We conclude that, Compared to randomly initialized positional encodings, the encodings initialized with locality and symmetry properties lead to better sentence representation models.
Visualization Analysis of BERT-A -Seq. Compared to BERT-A", BERT-A* can further im- prove the scores on downstream tasks by combining positional and contextual features in a sequen- tial way. Figure T0 visualize all positional heads in BERT-A" -Seq, we observe these matrices still perfectly localized and symmetrical, which is similar to the behaviors of BERT-A*. We con- clude that the Sequence combination can yield better sentence representations than the traditional Add combination.
Training
Validation
BERT
BERT-A"
11-
BERT BERTTA
1
LO3
5-
50
100
150
200
300
10
15
20
(eveny (overy 100 steps)
Sups (every 600 siepo)
Figure 6: Loss curves of different variants adapted from BERT base model.
22

---

Under review as a conference paper at ICLR 2023
0.832
0.982
0.930
0.930
0.929
0.828
Accuracy
0.928
0.326
0.924
0.922
0.0
0.920
0.918
0.0
0.0
0.4
0.8
Locality
Symmetry
(a) Correlations between Locality and Accuracy
(b) Correlations between Symmetry and Accuracy
Figure 7: Correlations between the two properties (Locality and Symmetry) and accuracy on SUBJ dataset (Pang & Lee 2004). The yellow line shows the locality or symmetry of the pre- trained BERT.
23

---

Under review as a conference paper at ICLR 2023
Pourt
Lawel Foael
L
Lowel Hoser2
Lappert
Figure& 8: Visualizations of positional weights of BERT across all layers. The weights are computed by Identical Word Probing. Red color means lower values and blue color means higher values.
24

---

Under review as a conference paper at ICLR 2023
Lowerh
Hear
SAWEY
Madt?
Scan
Lowi
Heald
Lown
Lowel
Lewel Hear
Laard
Lear H6477
LAar Mel1
Lowil Heals
Lowe'
Lowe
Lewel Hear'2
Lewrt Heart
Leard Heard
Hears
Heal
Heal
Law RcHol'd
Suerl1
19W 1941
Lean Hamr?
Low Heal
Loan
11 Heald
Taye 11
Howe
Law Hoad
Loan Head
Taw ElHoot
Figure 9: Visualizations of positional weights of BERT-A" across all layers. Red color means lower values and blue color means higher values.
25

---

Under review as a conference paper at ICLR 2023
Linel
Lowel
Years
Lewrl
Heure
How
Lave
Hoir
Figure 10: Visualizations of positional weights of BERT-A* -Seg across all layers. Red color means lower values and blue color means higher values.
26


Reviews:
[{"review_id": "kKDukJISkUd", "reviewer": "Reviewer_eCeS", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a deep study on positional encoding for transformer models. They visualize the positional attentions for various models and point out that the current learned positional encodings have two important properties: locality and symmetry. Based on the two properties, they design handcrafted positional encodings and show their advantages by conducting experiments.", "summary_of_the_review": "Something unclear to be clarified and similar conclusions have already analyzed by previous work.", "strength_and_weaknesses": "Strength\n- Comprehensive studies.\n\nWeaknesses\n- There are too many important contents putting in the appendix. For example, the equation of the proposed encodings should be put in the main context.\n- Maybe I misunderstood. But why the symmetry is inconsistent with the examples “a man playing an electric guitar on stage” is totally different from “an electric guitar playing a man on stage”? I feel like the symmetry just implies that the attention of the i-th word to the j-th word is almost equally important to the attention of the j-th word and the i-th word. From my point of view, the locality does capture the meaning change in this case.\n- In the original Transformer paper, they also design the positional encodings based on sine and cosine. It's satisfied the locality and symmetry properties as well and is reported to have similar performance to the learned positional encodings. So I am not super excited about the experimental results.\n\nQuestion\n- It's a bit interesting that handcrafted + learnable is better than handcrafted and learnable separately. Is it because handcrafted provides better initialization? I suggest to explore this direction more.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Fkn3FqP36zE", "reviewer": "Reviewer_WDLJ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the role that positional embedding play in large language models through a series of probing tasks. In particular, they first use the previously proposed task of *Identical Word Probing (*where the same word is repeated and passed to a model and the attention weights in the attention-mechanism are visualized) and find that, similarly to previous work, these embeddings seem to pass two properties to the attention weights: symmetry and locality. \n\nThey then propose a new probing task, *word swap probing,* where words inside constituents are swapped and the predictions of the model are compared. For example, they observe that swapping word inside noun/verb/etc... phrases as little effect on the predictions, as expected, but that swapping agent and patient also don’t change prediction when it should. They also propose a set of hand-crafted positional encodings that satisfy the locality and symmetry properties and show that they perform comparably to learned kernels. \n\nThe authors explore how positional and contextual encodings encode different linguistic features through a series of linguistic probing tasks, finding that positional encodings encode knowledge for lower-level, syntactic probing tasks while on semantic tasks contextual embeddings/encodings perform better. The authors conclude with a new proposed positional encoding mechanism, where-as an attention mechism is applied only to the positional encodings and after another mechanism is applied to the contextual encodings (ie embeddings), finding that it performs comparably or better than a BERT models trained with traditional positional encodings.", "summary_of_the_review": "This paper provides an extensive analysis of the role of positional embeddings in language encoders. While the analysis is throughout and covers many aspects, and there are some novel contributions in terms of diagnosis tasks and positional encoding methods, the quality of the narrative and writting is sub-par to the point that it's hard to understand the contributions of the paper. \nOverall I still think the contributions are novel enough to warrant this conference, and would be willing to increase my score if writing was improved.\n\nUPDATE: The updated manuscript improved the quality of writing massively. There are still some concerns that other reviewers highlighted (such as if these findings generalize to models other than BERT). Overall, raising my score to 6.", "strength_and_weaknesses": "In terms of strenghs\n- The paper has a very throughout analysis of different models and positional encodings\n- It proposes several contributions, including a new probing task and several positional encoding methods.\n\nMy main problem with this paper at the moment is that it could be much better written: The overall narrative of the paper is unpolished and it’s hard at first to understand the contributions of the paper. I had to go multiple times back-and-forward in this paper to understand what was new in it. The introduction of two different positional encoding methods in different sections is confusing (even if one only serves the purpose of understanding the role of positional encodings). A lot of attention/space is dedicated to locality and symmetry properties of positional encodings, which from my understanding, isn’t very novel and has been explored in previous work.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "4qODQ5-U5G", "reviewer": "Reviewer_gutD", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper investigates the role of position encodings in contrast to contextual encodings in BERT-based models. They illustrate two biases of these encodings: locality (a tendency to focus on nearby tokens) and symmetry (a tendency to focus at a particular distance from the target to the same degree whether the distance is measured backwards or forwards). They decompose attention into the sum of position and contextual components, and ablate each component to measure the role of the remaining representation. They then propose several modifications to the encoding scheme in order to test their intuitions about the biases of the default scheme (one of these modifications, sequence combination, appears to lead to a performance improvement).", "summary_of_the_review": "The paper has two potentially significant contributions: a discussion of the drawbacks of symmetry bias in position encoding and a new (to my understanding) method of \"sequence combination\" that presents an alternative to summation of position and contextual encodings.\n\nHowever, the other experimental results and methods proposed are unconvincing to me as evidence of their claims of the importance of locality and symmetry. Furthermore, they are very similar to existing work in the analysis and architecture literature, most of which is uncited here.", "strength_and_weaknesses": "\nStrengths:\n- I find the question they are attempting to ask about the role of position encodings to be an interesting one.\n- I have never seen a clear discussion of the symmetry property, although it is evident in attention heat maps in every paper on transformers. \n- More importantly, I have never seen a discussion of the potential drawbacks of a symmetry bias. It’s not something that I had considered before, and it makes me curious about human cognition: are we better able to handle symmetric word swaps than random word swaps, for example? I expect not. This is a clear example of cognitively implausible behavior at the position encoding level, although to be fair nobody is claiming any kind of cognitive plausibility in position encoding. I would like to see the focus on symmetry expanded.\n- I was impressed by the incorporation of research in psycholinguistics to discuss word swaps. I would actually have liked to see a longer discussion of the implications of the different studies, in terms of local and more distant word swaps.\n- In future work or versions of this paper, I would be extremely interested to see the symmetry and locality biases treated separately.\n- The new method of Sequence Combination as an alternative to adding together position and contextual encodings appears to work well.\n\nWeaknesses:\n- My main problem with this paper is that I do not find the experiments to be particularly convincing evidence of the properties that they discuss. In particular, they use a new encoding scheme to artificially impose strict symmetry and locality, and find that performance is somewhat damaged but not too much. I don’t know what result they would consider to be evidence against the importance of symmetry and locality, perhaps some very extreme blow to performance? In general, I don’t think that the performance of these simplistic position encodings give any particular evidence as to whether their biases are *also* the reason why the original encoding scheme worked.\n- I would have liked to see more variety in the position encoding schemes being used, rather than using only BERT’s default position encoding.\n- When you set the contextual or position encoding to zero, you create a significant distribution shift in representations. Perhaps taking an average would be better? In general, I’m skeptical about these kinds of causal interventions on a fully trained model. You are changing the expected norm and representational geometry, and that might change to different degrees depending on which component you are removing. \n- They seem to be arguing that symmetry in the attention weights means that you can swap the position of different tokens without changing the meaning of the sentence, but that’s not clear to me? I feel like I need an explanation that either gives the mathematical process or diagrams the reason why this would be the effect.\n- I would like to see for learned position encodings whether symmetry is a learned property.\n- The experiments on simplistic position encodings bear a significant resemblance to Alibi, and I’m not sure what they contribute over that scheme in terms of understanding https://arxiv.org/pdf/2108.12409.pdf\n- There are nonlinear interactions between the position encodings and the context. Currently they consider these interactions to be part of the position encoding, but I’m not convinced that that’s appropriate. It’s possible that some of the bias they are talking about is due to the isolated role of position $\\frac{(p_iW^Q)(p_jW^K)^T}{\\sqrt{d}}$ so I think that experiments should separately consider what happens when interactions are included or removed.\n    - When you measure the effect of removal, do you just straightforwardly remove it or do you take an average value or do you retrain the network to work without the encoding in question? There is a big difference between these strategies because you are creating a very large domain shift by removing a component of the representation, but that doesn’t necessarily mean that the information of structure within that component is as important as it seems based on changes effected by the wholesale removal of a component.\n    - Maybe you should be considering something like shapley values.\n- There should be an explicit comparison between their word swap probe and other existing methods for behaviorally testing the role of word order. Currently I’m not clear on what the new contributions are of this particular experiment.\n- Do you have a metric for measuring symmetry? That would be better than all of the figures. Even something simple like the matrix norm of the difference between the original attention matrix and its transpose.\n- There is a lack of detail about the implementation of sequence combination, although it seems like it should be one of these strongest points in the paper.\n\nMinor/references:\n- This paper needs extensive editing for typos (“encoings”, mismatched subject-object number, etc).\n- Probably should add a note as to why autoregressive models aren’t relevant to position encodings research http://arxiv.org/abs/2203.16634\n- Sharp Nearby, Fuzzy Far Away: How Neural Language Models Use Context https://arxiv.org/pdf/1805.04623.pdf\n- When you describe the symmetry property, make sure to clarify the axis of symmetry. Even with the example given, which initially just confused me, it took me a while to understand that this was referring to equidistant tokens having the same weight rather than the same weight applying both to both key-query and query-key directions.\n- “BEET-A∗-s” I don’t understand what this string means.\n- “we adopt widely used 10 probing tasks” — make sure that you actually cite each of the tasks separately, rather than only the benchmark paper that aggregates them.\n- “For surface tasks, the surface knowledge is stored more in bottom layer …” I find this a confusing sentence\n- Don’t start a sentence with a variable name if you can help it, as in the paragraph before equation 10 (https://jmlr.csail.mit.edu/reviewing-papers/knuth_mathematical_writing.pdf)\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "3nHZVo4UOK", "reviewer": "Reviewer_LpaP", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper investigates the role of position encoding of pre-trained LM (BERT base) for sentence-level downstream tasks (NLI, STS):\n- By visualizing the attention heat map of PLM given a sequence of the same repeating token, this paper finds that LOCALITY and SYMMETRY are two key features of the position encoding of PLM.\n- By shuffling spans in different lengths of SNLI data and testing with these shuffled texts, this paper finds PLMs are less sensitive to local word order.\n\nThis paper also proposed two handcrafted position encodings, by adding positional bias to the context attention scores. \n- By training BERT and BERT with the proposed encoding methods, this paper finds the proposed encoding methods can outperform BERT a lot.", "summary_of_the_review": "This paper should be improved before it can be published because:\n\nNo discussion between Alibi[1] and the proposed method;\nMissing details to understand the results;\nOnly experimented with the BERT base model.\n\n\n\n\n[1] Press, Ofir, Noah Smith, and Mike Lewis. \"Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation.\" International Conference on Learning Representations. 2021.", "strength_and_weaknesses": "Strength:\n- Some probing experiments are novel and the findings are interesting\n- The proposed methods get good results\n\nWeaknesses:\n- Many details are missed, which makes parts of this paper hard to be understood. 1) what are the learnable form of equations 12 and 13? A random initialized matrix? 2) What is PA(X)? Multiply X with the PA matrix? 3) The results of STS and SICK-R seem much better than RoBERTa-large. What is the metric of STS and SICK-R? Spearman or Pearson? STS has 12-16 and STS-B, which one is used in this paper? 4):\n- The proposed method is similar to AliBI, in terms of adding a bias mask to the attention scores. Is there any discussion of the difference or which one is better?\n- The models are all BERT-base trained with 600K steps. Experimenting with another model can help to support the claims of this paper. For example, show the BERT-large/RoBERTa can also be improved.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject