The Locality and Symmetry of Positional Encodings
Authors: Lihu Chen
Gael Varoquaux
Fabian M. Suchanek
Venue: EMNLP 2023 Findings (2023)
Keywords: Positional Encodings
Sentence Representations
Pre-trained Language Models
TL;DR: Positional Encodings in Setence Representations

Abstract: Positional Encodings (PEs) are used to inject word-order information into
transformer-based language models. While they can significantly enhance
the quality of sentence representations, their specific contribution to
language models is not fully understood, 
especially given recent findings that various positional encodings are insensitive to word order.
In this work, we conduct a systematic study of positional encodings in \textbf{Bidirectional Masked Language Models} (BERT-style)
, which complements existing work in three aspects: (1) We uncover the core function of PEs by identifying two common properties, Locality and Symmetry; 
(2) We show that the two properties are closely correlated with the performances of downstream tasks;
(3) We quantify the weakness of current PEs by introducing two new probing tasks, on which current PEs perform poorly. 
We believe that these results are the basis for developing better PEs for transformer-based language models.

The Locality and Symmetry of Positional Encodings
Lihu Chen', Gaël Varoquaux', Fabian M. Suchanek2
Inria, Soda, Saclay, France
2 LTCI, Télécom Paris, Institut Polytechnique de Paris, France
(lihu. gael jeinquau
jumecaak fr
Abstract
Positional Encodings (PEs) are used inject word-order information into transformer-base language models. While they can significantly enhance the quality of sentence representations, their specific contribution to language models is not fully understood, especially given findings that various positional encodings are insensitive to word order. In this work, we conduct systematic study of positional codings in Bidirectional Masked Language Models (BERT-style). which complements ex- isting work in three aspects: (1) We uncover the core function of PEs by identifying two com- mon properties, Locality and Symmetry; (2) We show that the two properties are closely cor- related with the performances of downstream tasks: (3) We quantify the weakness of current PEs by introducing two new probing tasks, on which current PEs perform poorly. We believe that these results the basis for developing better PEs for transformer-hased language mod- els. code available at Ω https://githul com/ wttptttircenn//oooall. symmety
Figure I: Visualizations of positional weight matrices by using Identical Word Probing (Wang et al., 2021). matrices highly attend local positions (Locality) and are nearly symmetrical (Symmetry).
are intriguing phenomena, with reasons not well understood.
In this work, we focus on uncovering the core properties of PEs in Bidirectional Masked Lan- guage Models (BERT-style (Devlin al., 2019)). We do not include analyses of GPT (Brown et al., 2020) and LLaMA (Touvron et al., 2023) since there an obvious distinction between them and BERT-style models. Decoder-only models pre- diet the next word based on previous words the sentence (Left-to-Right mode), and therefore im- plicitly introduce directional information, which is different from bidirectional models. Further- more, recent studies have shown that decoder- only transformers without positional encodings able achieve competitive or even better perfor- mance than other explicit positional encoding meth- ods (Haviv et al., 2022; Kazemnejad et al., 2023). Hence, we focus on the BERT-style models in this work leaving the Decode-only models as a future study.
1 Introduction
Transformer-based language models with Posi- tional Encodings (PEs) can improve performance considerably across a wide range of natural lan- guage understanding tasks. Existing work resort to either fixed (Vaswani et al., 2017; Su et al., 2021; Press al., 2022) or learned (Shaw al., 2018; Devlin et al., 2019; Wang et al., 2020) PÉs infuse models. To understand how PEs capture word order, prior studies apply visualized (Wang and Chen, 2020) and quantitative analyses (Wang et al., 2021) to var- ious PEs, and their findings conclude that all encod- ings, both human-designed and learned, exhibit consistent behavior: First, the position-wise weight matrices show that non-zero values gather on lo- cal adjacent positions. Second, the matrices are highly symmetrical, as shown in Figure 1. These
order information into attention-
We study various positional encodings by in- troducing two quantitative metrics, Locality and Symmetry. Our empirical studies demonstrate that the two properties correlated with sentence rep- resentation capability. This explains why fixed encodings are designed to satisfy them and learned encodings are favorable
local and symmet- rical. Moreover, we show that attention- based models are initialized with PEs that already share

---

good locality and symmetry, they can obtain better inductive bias and significant improvements across 10 downstream tasks. Our findings of the local preference may explain why Sliding Window Atten- tions (MistralAI, 2023) and Attention Sinks (Xiao et al., 2023) can work effectively.
We design two new probing tasks of word swaps, which show a weakness of existing positional encodings, namely the insensitivity against the swap of semantic roles.
2 Preliminaries
Although PÉs with locality and symmetry can achieve promising results on natural language un- derstanding tasks (such GLUE (Wang et al., 2019)), the symmetry property itself has an ob- vious weakness, which is not revealed by previous work. Existing studies use shuffled text to probe the sensitivity of PEs to word orders (Yang et al., 2019a; Pham et al., 2021; Sinha et al., 2021; Gupta et al., 2021; Abdou et al., 2022), and they all as- sume that the meaning of sentences with random swaps remains unchanged.
The central building block of transformer archi- tectures is the self-attention mechanism (Vaswani et al., 2017). Given an input sentence: X {x1 X2, .... xn E Rx where is the number of words and d the dimension of word embeddings, the attention computes the output of the i-th token as:
exp(ait)
X
Xj
W
(1)
Z
However, the random shuffling of words may change the semanties of the original sentence and thus cause the change of labels. For example, the sentence pair below from SNLI (Bowman et al., 2015) satisfies the entailment relation:
(x; (x:WQ) (x, WK)T
where Cijj
Vd
2 exp(ait)
a. A man playing electric guitar stage
Self-attention heads do not intrinsically capture the word order in a sequence because there is no posi- tional constraint in Equation 1. Therefore, specific methods are used to infuse positional information into self-attention (Dufter et al., 2022).
b. A man playing guitar on stage
If we change the word order of the premise sen- tence so that it becomes "an electric guitar playing a man on stage", a fine-tuned BERT still finds (in- correctly!) that the premise entails the hypothesis. Starting from this point, we design two new prob- ing tasks of word swap: Constituency Shuffling and Semantic Role Shuflingg The former preserves the original semantics of the sentence by swapping words inside constituents (local structure) while the latter intentionally changes the semanties by swapping the semantic roles in a sentence (global structure), i.c., the agent and patient. Our results show that existing language models with various PEs are robust against local swaps, but extremely fragile against global swaps. The key contributions of our work are:
A Absolute Positional Encoding (APE) computes positional encoding for each token and adds it to the input content embedding to inject position information into the original sequence. The og in Equation are then written:
aiyy (xi + PP: + P.j) W
(2)
Vd
Here, P; E IRd is position embedding for the ith token, obtained by fixed (Vaswani et al., 2017;
Dehghani et al., 2019; Takase and Okazaki, 2019; Shiv and Quirk, 2019; Su et al., 2021) or learned encodings (Gehring et al., 7017; Devlin et al., 2019; Wang et al., 2020; Press et al., 2021; Ke et al., 2021).
We reveal the function of PEs by identi- fying two common properties, Locality and Symmetry, and introduce two quantitative metrics to study them.
Relative Positional Encoding (RPE) produces a vector r,j or scalar value B. that depends on the relative distance of tokens. Specifically, these methods apply such vector bias to the attention head so that the corresponding attentional weight can be updated based on the relative distance of two tokens (Shaw et al., 2018; Raffel et al., 2020):
We discover that suitable symmetry and lo- cality lead to better inductive bias, which explains why all positional encodings (both learned or human-designed) exhibit these two properties.

---

contextual weights (We elaborate more on visual- izations in Section A.I) The obtained matrices are usually diagonal-heavy, which means that the po- sitional encodings highly attend to local positions. Second, the matrices are usually nearly symmetri- cal.
x, x x3 W + +
aij-
Vd
(x (, W W + Bis
oij
Vd
We call these two phenomena the Locality and Symmetry of positional encodings. A prior study shows that the local structure is crucial for under- standing the semantics of sentences (Clouatre al., 2022). The symmetry property has been discov- ered and quantified already by Wang et al. (2021). Here, we provide a more in-depth analysis the locality and symmetry. We analyze the linguistic role of locality and point out the potential flaw of symmetry, which is not considered by prior work. To better understand how encodings capture word order, 1et us consider an attentional weight vector E, and the element Ei, can be denoted as:
Here, the first mode uses a vector ra,j while the second uses a scalar value Bigj for infusing relative distance into attentional weight. Recent research on RPEs has been remarkably vibrant, with the emergence of diverse novel and promising vari- ants (Dai et al., 2019; He et al., 2021; Press et al., 2022).
Unified Positional Encoding. Inspired by TUPE (Transformer with Untied Positional Encoding) (Ke et al., 2021) rewrite all of the above absolute and relative positional encodings in unified way as follows:
exp(a C = Σド= exp((as)
positional
contentual
(4)
dd
where EL 2 0 and Σ =1=11
(5)
Here, the left half of the numerator, 7i.j captures contextual correlations (or weights), i.e., the se- mantic relations between token 2; and zy. Hence, the contextual correlation can be denoted as "N = (x; ()) (x, W The right half captures po- sitional correlations, i.c., the positional relations between tokens x; and 2y. For example, the relative encoding in (Shaw et al., 2018) can be represented o x; w TT Thus, existing positional encodings all add contextual and positional corre- lations together in every attention head.
We then define the two metries: Locality and Symmetry. Locality is metric that measures how much the weights of an attentional weight vector are gathered in local positions. Given a weight vector for the i-th position e, {(,,,, 6 c,n we define locality as:
n
Loc(e) c [0, = 2
(6)
Here, a value of 1 means the vector perfectly sat- isfies the locality property. For example, given a sequence whose length is 5 and weight vector for the first position [1, 0, O, 0, O]. the locality is 1, which means it perfectly matches the locality, In contrast, the locality is 1/16 if the weight at- tends only the last position, as (0,0, 0, 0, 1]. For measuring the locality of a matrix, we average the locality values of all vectors in the matrix.
3 Positional Encodings Enforce Locality and Symmetry
3.1 The Properties of Locality and Symmetry Existing work analyzes positional encodings with the help of matrix visualizations (Wang and Chen, 2020; Wang et al., 2021; Abdou et al., 2022). Such a matrix is a positional weight map, where each row is a vector for the i-th position of the sentence and the element at (i, j) indicates the attention weight between the i-th position and the j-th po- sition. The matrices are computed by using the Identical Word Probing proposed by Wang et al. (2021): many repeated identical words are fed to the pre-trained language model, so that the atten- tion values (as.j in Equation 4) are unaffected by
Symmetry is metric that describes how symmetrical the weights scatter around the current position for attentional weight vector. To measure the symmetry, we first truncate a weight vector to obtain a new one with the same length to the left and right of the current position i: c = [Er,eets Et,efftt,, Ej ..., Eiright-1 Girrght }. where left and right are the left start and the

---

lyzed in this experiment. We measure the two prop- erties of the positional weight matrix (the averaged weight across layers) before and after pre-training. Figure 2 shows the visualization results. We find that the the three language models all be- come much more local and symmetrical after pre- training, which proves that the two properties are indeed learned. The manually designed positional encodings, too, are already well localized and sym- metrical, which demonstrates that the two prop- erties are important for positional encodings, al- though the reason is not clear.
DEATT
Sinwaldd
Localty
0.19
Rs Termer
0.27
Intialzedd Pre trained Foodd
086
be08RTa
Symmetry
Figure 2: Locality and symmetry values of positional en- codings. The green points are fixed and human-designed positional encodings while the orange points are posi- tional encodings after pre-training.
3.3 Can Locality and Symmetry Yield Better Inductive Bias?
Since locality and symmetry are common features of existing positional encodings, we investigate what happens if a language model is initialized with positional encodings that exhibit good. We conduct empirical studies in both non-pre-trained and pre- pre-trained settings.
right end position, respectively. The length of the left segment is left = i left and the length of the right segment is tight - right - i. The two lengths are the same: len!eft = right = min(i 1, i). Then, detect whether the left and right sequences are symmetric with respect the center position
Hand-crafted Positional Encodings There are various human-designed positional encodings, but the locality and symmetry cannot be modified eas- ily for these encodings. To study the effect of vary- ing locality and symmetry. we design an Attenuated Encoding that can be parameterized along these di- mensions. Our encoding is defined as follows:
Sym(E)) E [0
lenteft
Norm(cs
hii c d lenteft
1 Σ
exp(ais)
81, = =((,)) = ΣE exp(ass)
First, we apply a min- max crepancy of corresponding position pairs to obtain more uniform distributions. Otherwise, the symme- try values will extremely cluster around 0. Second, we reverse the value so that means a perfect sym- metry instead of 0. For example, given sequence whose length is 5 and weight vector for the third position 10.1, 0.2, 0.4, 0.2, 0.1], the symmetry is 1, which means that the vector completely symmet- rical.
min-max normalization to the dis-
ssw w i
where Oi.j
(8)
w ij
>
Here, 4.jj the relative distance, w > is scalar parameter that controls the locality value, and s a scalar parameter that controls the symmetry value.
Note that the key difference of our method is that the two properties can be adjusted while other manually designed ones such the T5 bias (Raffel et al., 2020) and ALiBi (Press et al., 2022) do not allow this.
3.2 Are Locality and Symmetry Learned?
To answer this question, we use our two proposed metrics to quantify the locality and symmetry of both manually designed (fixed) and learned en- codings. Specifically, three fixed encodings (Sinu- soidal (Vaswani et al., 2017). Roformer (or RoPE) (Su et al., 2021), and ALiBi (Press et al., 2022)) and three language models with learned encodings (BERT (Devlin et al., 2019), XLNet (Yang et al., 2019b) and DeBERTa (He et al., 2021)) are ana-
3.3.1 Non-pre-training Setting
It is impractical to train large language models with different values of locality and symmetry from scratch. Therefore, we use static word embeddings from GloVe (Pennington al., 2014) and an en- coder that is fully based on our handcrafted posi- tional encodings for our experiment. The position- based encoding is adapted from the aelf-attention

---

Model
Size
Sentiment Analysis SUBI
Textual Entailment
Puraphrase Identification MRPC
Textual Similarity STS-B
SST-2
QNLI
RTE (5.5K)
MNLI (413K)
QQP (755k)
SICK-R (9.4K)
Avg
(22K)
(20K)
(68.8)
(L10K)
(5.4K)
(8.4K)
BERT
110M
2..
91.
864
85.8
59.211
78.2...
23.5
$8.7.0.0
77.84.1
64.20.0 77.8
BERT-A"
113M
79442.9
93.740.0
88.0=0.7
86.
594+2.
78 88.4
81,5+22
48.7+0.4
83642.0
76331..1
81.6
BERTA"
138M
78.2
93.020.8
18.1
87.0
61.0g1.4
4
78.9009
80.9
89.220.3
84.3
76.0
B1.7
Table I: Evaluations of handcrafted encodings across 10 downstream tasks. We report the average score (Spearman correlation for textual similarity and accuracy for others) of five runs using different learning rates. means the encodings are learnable and means that positional encodings are shared within the attention headers of layers.
encoder, which means we only keep the 6Gg in Equation 4. We use our handcrafted attenuated en- codings to compute the d.g. Therefore, the locality and symmetry can be adjusted easily and we can observe the correlations caused by the changes of the two properties. An implementation example of this positional encoder is shown in Listing 2 of the Appendix A.
XM0
We use two sentence-level text classifica- tion datasets, MR (Pang and Lee, 2005) and SUBJ (Pang and Lec, 2004), for evaluation. As for the encoder, single-layer and single-head posi- tional attention is used and the handerafted encod- ings are fixed during training. We use the 840B- 300d GloVe (Pennington al., 2014) vectors as word embeddings. For training, we use an Adam optimizer with an initial learning rate 0.002, and introduce a decaying strategy to decrease the learn- ing rate. We adopt dropout method after the encoder layer, and train models to minimize the cross-entropy with dropout rate of 0.5. Each model is trained 5 epochs and we select the best model validation sets evaluate on the test set. We repeat this procedure times and use the average score to report.
Localy
Symmway
(a) Locality
(h) Symmetry
Figure 3: Empirical studies of the properties cality and symmetry the MR sentiment analysis dataset (Pang and Lee, 2005).
try value of BERT. Since the change of symmetry will impact the value of locality, we can only ob- serve this type of partial correlation. We find that symmetry affects performance only after certain value (0.65), and larger symmetry leads better accuracy. However, this does not mean that symme- try is a good property of sentence representations because many natural language understanding tasks such as sentiment analysis do not require strict word order information. In section 3.5, we will discuss a potential flaw caused by symmetry. Experiments the SUBJ dataset (Pang and Lec, 2004) lead to similar conclusions, and are shown in Figure A the Appendix A.
Figure 3 (a) shows the impact of the locality on the performance of the MR dataset. In this experiment, the symmetry value is 1.0 for all en- coders. We observe that the accuracy constantly increases as the locality of encodings strengthens, which means a higher locality induces better sen- tence representation. Experimental results on the SUBJ dataset (Figure A1) show that the accuracy growth slows down at a particular locality value (0.3), which means that completely perfect local- ity unnecessary. The locality value for BERT is around 0.2, and BERT actually does not have an extreme locality. Figure 3 (b) shows the results for different symmetry values. In this experiment, we vary the symmetry while keeping the locality in the interval [0. 10.15.00 0.3]. which is close to the symme-
3.3.2 Pre-training Setting
In this experiment, we adjust the parameters w and in Equation 8 to obtain a weight vector 8 that shares the locality and symmetry of the pre-trained BERT (Locality=0.17 and Symmetry=1.0), We pre-train BERT initialized with 5 and compare them to learned encodings downstream natu- ral language understanding tasks. Two variants are compared with the original BERT: 1) BERT- A uses learnable and shared 8, but the weights are shared inside particular layer, 2) BERT-A* uses learnable but not shared 6, which means 8

---

100
playing electric guitar stage
Performance (%)
Encoding
VP
75
contextual
man
playing guitar stage
positional
50
both
DT
NN
VEG
NP
PP
man
playing
electric guitar
stage
25
DT
1)
NN
IN
an
electric
guitar
on
stage
0
2
8
Distance (in words)
Figure 4: Illustration of constituent parsing for one
sentence SNLI "a man playing an electric guitar on stage", generated by Berkeley Neural Parser (Kitaev et al., 2019).
Figure 5: Accuracy of top-20 dependency relations.
Detailed results Table A3, Appendix.
is different in each attentional head. More details of the datasets and pre-training are shown in Ap- pendix A.4 and Section A.2, respectively. The empirical results shown in Table 1. We ob- serve that both BERT-A"- and BERT-A" can sig- nificantly outperform the original BERT, which demonstrates positional encodings with initializa- tion of suitable locality and symmetry can have a better inductive bias in sentence representation. The visualizations of attentional heads in BERT are shown in Figure A3 and the visualizations of 8 in BERT-A" are shown in Figure A4, both in the AppendixA. In fact, there great diversity of be- haviors within different attentional heads, mainly for the diagonal bandwidths of attentional maps (locality), which signifies proximity units can be combined at different distances.
phrase-level meaning (e.g.. ("take", + "take off"), and clause-level and sentence-level mean- ing can be obtained through an iterative combi- nation of low-level meanings, which is consistent with the multi-layer structure in pre-trained lan- guage models. From linguistic perspective, words linked in syntactic dependency should be close in linear order, which forms what can be called dependency locality (Futrell et al., 2020). De- pendency locality provides a potential explanation for the formal features of natural language word order. Consider the two sentences "John throws out the trash" and "John throws the trash out". Both grammatically correct. There is a de- pendency relationship between "throws" and "our" and the verb is modified by the adverb. However, language users prefer the expression with the first sentence because it has a shorter total dependency length (Dyer, 2017: Liu et al., 2017; Temperley and Gildea, 2018). Based on the visualizations and dependency locality, we, therefore, speculate that one main function that positional encodings have learned during pre-training is local composi- tion, which exists naturally in our understanding of sentences. Empirical studies also demonstrate that performances of shuffled language models are cor- related with the violation of local structure (Khan- delwal et al., 2018; Clouatre et al., 2022).
We conclude that positional encodings with more suitable locality and symmetry can yield bet- ter performance on downstream tasks, which may explain why fixed encodings are designed to meet the two properties and why learned encodings all exhibit this behavior. However, learned encodings are not perfectly local, but they still have the ability to represent sentences very well, which might be due to the network architectures and the specific target tasks. Moreover, increasing the diversity of positional correlations in different attentional heads, e.g., the diversity of locality, also benefi- cial for representation capabilities.
To verify that the main function of locality is to compose adjacent units with short-distance de- pendency relations, we examine the dependency knowledge stored in positional weights. Specifi- cally, follow the syntactic probing test by Clark et al. (2019), and then each head in PLMs garded as simple predictor of dependency rela- tions. Given the attention weight vector of an input word, we output the word with the highest values and think the pair of words holds some type of
3.4 Why the Locality Matters?
Locality means that the positional weights favor the combination of units in a sentence their ad- jacent units when creating higher-level representa- tions. For example, sub-tokens can be composed into lexical meanings (c.g., { ""context", "#Hual") "contextual") or words can be composed into

---

Original
Shuffled
Shuffled-3
mans
package poses front of aos auhretiionnn
vid
vild paclage
proses de fromt adverttionnn
Shuffled-4
ban nover is being driven acroes river
kand nover is being
driveen river accruse
Shuffled-S
man reods paper with green lighting
man reads paper with green lightingg bar
Shuffled-6
buy
gray sriped swcteer bun purts is play-
buy in strppel a owenter while gray and puets is play- ing piece groumnd equigmen
piece playground squipment
ShuffeedSR
several
are playing
valleyboll
volleyball playing several women.
Shufled-SR 3 man and woman are sharing
holdog
hoodog are sharing a man and womas
Table 2: Some cases of the shuffled SNLI datasets in our word swap probing. Shared color indicates corresponding phrases.
Model
Symmetry
Locality
Original
Shuffle (A)
Shuffle (A)
Shuille (A)
Ramdom (A)
Original
Shuflle SR (2)
BERT 87.9
16.2
89.8
-0.4
-0.6
-0.3
-2.7
89.8
-63.9
ALBERT
82.0
20.3
91.8
-1.1
-1.3
-6.0
92.0
466.8
DeBERTa
85.0
17.8
91.6
-0.5
-0.7
-1.3
-5.1
91.6
-58.9
XL.Net
72.7
17.5
91.5
-0.2
-0.3
-0.7
-5.4
91.3
-57.8
Stu:BERT
90.9
Table 3: Results of Constituency Shuffling and Semantic Role Shuffling, measured by accuracy. Shuffle-ar means phrases with length are shuffled. Shuffle-SR means the semantic roles of the agent and patient are swapped.
dependency relation. While single head can per- form well all relations, the best-performed head is selected the final ability of model for each particular relation. In this experiment, we adopt the original TUPE (Ke et al., 2021) that uses absolute positional encodings as our base model. The ability of contextual and positional weights evaluated by setting the unrelated correlations to zero (in Eq 4), e.g., the first term is set to dumb when checking the ability of positional weights. The two variants referred as to contextual and positional attention, respectively.
it while learned PEs do not attend to the current word itself, e.g., visualizations of BERT and De- BERTa in Figure 1. Moreover, our empirical results show that there is clear distinct role between the positional and contextual encodings in sentence comprehension: positional encodings play more of role at the syntactic level while contextual encodings serve more at the semantic level (see Section A. in the Appendix A).
We summarize that the locality property guides positional encodings to capture more short-distance dependencies while contextual weights capture more long-distance ones.
We extract attention maps from BERT on the MRPC (Dolan and Brockett, 2005) annotated by the dependency parser of spaCy We report the results on top-20 dependency relations.
3.5 What Is the Drawback of Symmetry?
Although positional encodings with good symme- try perform well on series of downstream tasks, the symmetry property has obvious flaw in sen- tence representations, which is ignored by prior studies.
Figure 5 shows performance on relations with different distances. (Table A3 gives relation- specific results). First, we observe that positional attention significantly more important than con- textual attention in short-distance dependency re- lations (distance from to 4). Second, contextual attention takes the lead on long-distance relations (after 6). Again, the combination of the two fea- tures yield the best performance. The "outlier" in the lower left corner is the Root dependency. Because this relation is a self-reflexive edge, con- textual (or self) attentions can performs well on
The symmetry (also observed by Wang and Chen (2020); Wang et al. (2021)) of the positional matri- ces implies that the contributions of forward and backward sequences are equal when combining ad- jacent units under the locality constraint. This is contrary to our intuition, the forward and back- ward tokens play different roles in the grammar, we have seen in the examples of "a man playing an electric guitar on stage" and "an electric guitar playing a man on stage". However, this symmetry
Phttps://sppace http::///sppy..iooppp//eeeeeeeee

---

is less disruptive at the local level inside sentences. Recent work in psycholinguisties has shown that sentence processing mechanisms are well designed for coping with word swaps (Ferreira et al., 2002; Levy, 2008; Gibson et al., 2013; Traxler, 2014). Further, Mollica et al. (2020) hypothesizes that the composition process is robust to local word viola- tions. Consider the following example:
As an example, consider the constituent parsing in Figure 4. We can easily shuffle the word order inside "an electric guitar" (say, to, "guitar an elec- tric"), which will lead to a grammatically incorrect, but still comprehensible sentence because humans are able to understand sentences with local word swaps (Ferreira et al., 2002: Levy, 2008). To con- struct such shuffled datasets, the premise sentences in the SNLI (Bowman al., 2015) test are shuf- fled and we keep the hypothesis sentences intact, Here, we let x E [3, 5] and select a subset from SNLI to make sure that every premise sentence has at least one phrase with a length from to 5. We select five types of target phrases for shuffling: Noun Phrase, Verb Phrase, Prepositional Phrase, Adverb Phrase, and Adjective Phrase. Finally, a Shuffle-z SNLI obtained by disrupting the order inside a phrase with length x and the size for each shuffle-w around 5000.
a. on their last day they were overwhelmed by farewell messages and gifts
b. on their last day they were overwhelmed by farewell and messages gifts
c. on their last they day were overwhelmed farewell messages by and gifts
The local word swaps (colored underlined words) are introduced in the latter two sentences, lead- ing to a less syntactically well-formed structure. However, experimental results show that the neural response (fMRI blood oxygen level-dependent) in the language region does not decrease when dealing with word order degradation (Mollica al., 2020), suggesting that human sentence understanding is robust to local word swaps. Likewise, symmetry can be understood in this way: when a reader pro- cesses a word in a sentence, the forward and back- ward nearby words are the most combinable, and the comprehension of this composition is robust to its inside order. On the other hand, symmetry is not an ideal property for sentence representations (consider the case of "an electric guitar"). Next, we use two new probing tasks of word swap to illustrate the flaws of symmetry.
Our other task, Semantic Role Shuffling, inten- tionally changes the semantics of the sentences by swapping the order of the agent and patient of sen- tences. For example, in Figure 4, "a man" is the entity that performs the action, technically known as the agent, and "an electric guitar" as the entity that is involved in or affected by the action, which is called the patient. Our dataset Shuffle-SB swaps these semantic roles. Some examples are shown in Table 2.
To probe the sensitivity of language models to the two types of shuffling, we fine-tune 5 pre-trained language models with good symme- try on the SNLI training set and evaluate them on the newly constructed Shuffle-r and Shuffle-SR datasets (see details in Appendix A.3). The overall results of the word swap probing are shown in Ta- ble 3. We first observe that the performances of all language models across Shuffle-rr datasets (local swaps) basically do not degenerate, which confirms the benefits of the locality and symmetry proper- ties. Second, most models fail on the Shuffle-SR dataset (global swaps), which demonstrates local symmetry does not capture global swaps well. This explains the reason that BERT-style models fail on the example: "an electric guitar playing a man stage". Although the local symmetry learned by positional encodings can perform well on a se- ries of language understanding tasks, the symmetry itself has obvious flaws. The better performance of StrucBERT on the Shuffle-SB suggests that in- troducing additional order-sensitive training tasks
Existing probes study the sensitivity of lan- guage models to word order by shuffling the words in a sentence, and they can be roughly divided into three categories: random swap (Pham et al., 2021; Gupta al., 2021; Abdou et al., 2022), gram swap (Sinha et al., 2021). and subword-level swap (Clouatre et al., 2022). All these studies as- sume that the labels of the randomly shuffled sen- tences are unchanged. However, this is obviously not the case. In particular, the shuffled sentence may have another label (think of the textual entail- ment example from the introduction).
To address the issue, propose two new prob- ing tasks of word swaps: Constituency Shuflling and Semantic Role Shuffling. Constituency Shuf- fling aims to disrupt the inside order of constituents, which is able to change the word order while pre- serving the maximum degree of original semantics.

---

may improve this problem. More details of the probing tasks are described in Appendix A.3.
Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. 2017. Fine- Fine-grained grained analysis of sentence embeddings using auxiliary prediction tasks. In Proc. of ICLR.
4 Conclusion
Luisa Bentivogli, Ido Kalman Dagan, Dang Hoa, Danilo Giampiccolo, and Bernardo Magnini. 2009. The fifth pascal recognizing textual entailment challenge. In TAC 2009 Workshop. publisher.
We have proposed a series of probing analyses for understanding the role of positional encodings in sentence representations. We find two main proper- ties of existing encodings, Locality and Symmetry, which are correlated with the performance of down- stream tasks. We first investigate the linguistic role of positional encodings in sentence representation. Mcanwhile, we point out an obvious flaw of the symmetry property. We hope that these findings will inspire future work to better design positional encodings.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large anmo- tated corpus for learning natural language inference. In Proc. of EMNLP.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, 1lya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems 33: Annual Conference on Neural Information Process- ing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
Limitations
The limitations of this work are three-fold. First, our study focuses on positional encoding in Masked Language Models (bidirectional), but this work does not involve decoding-oni language mod- els (GPT-style). Furthermore, recent studies have shown that decoder-only transformers without posi- tional encodings is able to outperform other explicit positional encoding methods (Haviv et al., 2022; Kazemnejad et al., 2023), which deserves further research. Second, our analysis is limited to the natural language understanding of the English lan- guage. Different languages display different word order properties. For instance, English is subject- verb-object order (SVO) while Japanese subject- object-verb order (SOV), and natural language gen- eration tasks are not included in this work. Third, although our handcrafted positional encodings sat- isfy the symmetry property, they merely replicate the limitations of current positional encoding, al- beit in simplified form. Further architecture de- velopment should address the problem of the "an electric guitar playing a man on stage." " mentioned in the introduction.
Daniel Cer, Mona Diab, Eneko Agirre, lñigo Lopez- Gazpio, and Lucia Specia. 2017. SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 1 1thh International Workshop on Semantic Evalu- ation (SemEval-200 2017)
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019. What does BERT look at? an analysis of BERT's attention. In Pro- ceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP.
Louis Clouatre, Prasanna Parthasarathi, Amal Zouaq. and Sarath Chandar. 2022. Local structure matters most: Perturbation study in NLU. In Findings of the Association for Computational Linguistics: ACL 2022.
Alexis Conneau and Douwe Kiela. 2018. SentEval: An evaluation toolkit for universal sentence representa- tions. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC
A Akknowledgements
This work was partially funded by projects NoRDF -0-00-00HH and LearnI (ANR-20- CHIA-0026).
Alexis Conneau, German Kruszewski, Guillaume Lam- ple, Loic Barrault, and Marco Baroni. 2018. What you cram into single S&!#* vector: Probing sentence embeddings for linguistic properties. In Proc. of ACL.
References
Mostafa Abdou, Vinit Ravishankar, Artur Kulmizev, and Anders Søgaard. 2022. Word order does matter and shuffled language models know it. In Proc. of ACI.
Ido Dagan, Oren Glickman, and Bemardo Magnini. 2005. The pascal recognising textual entailment chal- lenge. In Machine learning challenges workshop. Springer.

---

Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Car- bonell, Quoe Le, and Ruslan Salakhutdinov. 2019. X Attentive language models beyond a fixed-length context. In Proc. of ACL.
2006. The second pascal recognising textual entail- ment challenge. In Proceedings of the Second PAS- CAL Challenges Workshop Recognising Textual Entailment.
Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser, 2019. Universal transformers. In Proc. of ICLR.
Adi Haviv, Ori Ram, Ofir Press, Peter Izsak, and Omer Levy. 2022. Transformer language models without positional encodings still learn positional information. In Findings of the Association for Computational Linguistics: EMNLP 2022.
Jacob Devlin, Ming-Woy Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proc. of NAACI-HLT.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. Deberta: decoding-enhancee bert with disentangled attention. In Proc. of ICLR.
William B. Dolan and Chris Brockett. 2005. Automati- cally constructing corpus of sentential paraphrases. Proceedings of the Third International Workshop on Paraphrasing (IWP2005).
Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. 2019. What does BERT learn about the structure of language? In Proc. of ACI.
Amirhossein
Inkit Padhi, Karthikeyan Natesan Ramamurthy, Payel Das, and Siva Reddy. 2023. The impact of positional encoding on length generalization in transformers. ArXiv preprint.
Kazemnejad,
Philipp Dufter, Martin Schmitt, and Hinrich Schütze. 2022. Position information in transformers: An overview. Computational Linguistics, (3).
William Edward Dyer. 2017. Minimizing integration cost: A general theory of constituent order. Ph.D. thesis, University of California, Davis.
Guolin Ke, Di He, and Tie- Yan Liu. 2021. Rethinking positional encoding in language pre-training. In Proc. of ICLR.
Fernanda Ferreira, Karl GD Bailey, and Vittoria Fer- raro. 2002. Good-enough representations in language comprehension. Current directions in psychological science, (I).
Urvashi Khandelwal, He He, Peng Qi, and Dan Jurafsky. 2018. Sharp nearby, fuzzy far away: How neural language models use context. In Proc. of ACL.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proc. of ICLR.
Richard Futrell. Roger P Levy, and Edward Gibson. 2020. Dependency locality as explanatory princi- ple for word order. Language, (2).
Nikita Kitaev, Steven Cao, and Dan Klein. 2019. Multi- lingual constituency parsing with self- oeff-attention and pre- pre-training. In Proc. of ACL.
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. 2017. Convolutional sequence to sequence learning. In Proc. of ICML, Proceedings of Machine Learning Research.
Roger Levy. 2008. A noisy-channel model of human sentence comprehension under uncertain input. In Proc. of EMNLP.
Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. 2007. The third PASCAL recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing.
Haitao Liu, Chunshan Xu, and Junying Liang. 2017 7. Dependency distance: A new perspective on syntactic patterns in natural languages. Physics of life reviews.
Marco Marelli, Stefano Mcnini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zam- parelli. 2014. A SICK cure for the evaluation of compositional distributional semantic models. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC' 14).
Edward Gibson, Leon Bergen, and Steven T Piantadosi. 2013. Rational integration of noisy evidence and prior semantic expectations in sentence interpretation. Proceedings of the National Academy of Sciences, (20).
MistralAI. 2023. Mistral 7b. ArXiv preprint.
Ashim Gupta, Giorgi Kvernadze, and Vivek Srikumar. 2021. BERT & family cat word salad: Experiments with text understanding. In Thirty-Fifh AAAI Con- ference on Artificial Intelligence, AAAI 2021, Thirty- Third Conference on Innovative Applications of Arti- ficial Intelligence, IAAI 2021, The Eleventh Sympo- sium on Educational Advances in Artificial Intelli- gence, FAAI 2021, Virtual Event, February 2-9, 2021.
Francis Mollica, Matthew Siegelman, Evgeniia Diachek, Steven T Piantadosi, Zachary Mineroff, Richard Futrell, Hope Kean, Peng Qian, and Evelina Fe- dorenko. 2020. Composition is the core driver of the language-selective network. Neurobiology of Lan- guage, (1).
Bo Pang and Lillian Lee. 2004. A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. In Proc. of ACL.
R Bar Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor.

---

Bo Pang and Lillian Lee. 2005. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In Proc. of ACI..
David Temperley and Daniel Gildea. 2018. Minimizing syntactic dependency lengths: Lypoloiica//oogniivee universal? Annual Review of Linguistics.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proc. of EMNLP.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Ilama: Open and efficient foun- dation language models. ArXiv preprint.
Thang Pham, Trung Bui, Long Mai, and Anh Nguyen. 2021. Out of order: How important is the sequential order of words in a sentence in natural language un- derstanding tasks? In Findings of the Association for Computational Linguisties: ACL-ICCNLP 2021.
Matthew J Traxler. 2014. Trends syntactic parsing: Anticipation, bayesian estimation, and good-enough parsing. Trendy in cognitive sciences, (11).
Ofrr Press, Noah Smith, and Mike Lewis. 2021. Shortformer: Better language modeling using shorter inputs. In Proc. of ACI..
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez. Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems 30: Anmual Conference on Neural Information Processing Systems 2017, December 2017, Long Beach, CA, USA.
Ofir Press, Noah A. Smith, and Mike Lewis. 2022. Train short, test long: Attention with linear biases enables input length extrapolation. In Proc. of ICLR.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. J. Mach. Learn, Res.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. GLUE: A multi-task benchmark and analysis plat- form for natural language understanding. In Proc. of ICLR.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proc. of EMNLP.
Benyou Wang, Lifeng Shang, Christina Lioma, Xin Jiang, Hao Yang. Qun Liu, and Jakob Grue Simonsen. 2021. On position embeddings in BERT. In Proc. of ICLR.
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018. Self-attention with relative position representations. In Proc. of NAACI-III.T.
Benyou Wang. Donghao Zhao, Christina Lioma, Qiuchi Li, Peng Zhang, and Jakob Grue Simonsen. 2020. Encoding word order in complex embeddings. In Proc. of ICLR.
Xing Shi, Inkit Padhi, and Kevin Knight. 2016. Does string-based neural MT learn source syntax? In Proc. of EMNLP.
Yu-An Wang and Yun-Nung Chen. 2020. What do
position embeddings learn? an empirical study of pre-traie tre-trained language model positional encoding. In Proc. of EMNLP.
Vighnesh Leonardo Shiv and Chris Quirk. 2019. Novel positional encodings to enable based transform- ers. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Informa- tion Processing Systems 2019, NeurlPS 2019, De- cember 8-14, 2019, Vancouver, BC, Canada.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sen- tence understanding through inference. In Proc. of NAACL-HLT.
Koustuv Sinha, Robin Jia, Dieuwke Hupkes, Joelle Pineau, Adina Williams, and Douwe Kicla. 2021. Masked language modeling and the distributional hy- pothesis: Order word matters pre-train pre-tranning for little. In Proc. of EMNLP.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-ar natural language processing. In Proc. of EMNLP.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng. and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proc. of EMNLP.
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2023, Efficient streaming lan- guage models with attention sinks. ArXiv preprint.
Jianlin Su, Yu Lu, Shengfeng Pan, Wen, and Yunfeng Liu. 2021. Roformer: Enhanced transformer with rotary position embedding. ArXiv preprint.
Baosong Yang, Longyue Wang, Derek F. Wong, Lidia S. Chao, and Zhaopeng Tu. 2019a. Assessing the ability of self-attention networks learn word order. Proc. of ACI.
Sho Takase and Naoaki Okazaki. 2019. Positional en- coding to control output sequence length. In Proc. of NAACL-HLL.

---

Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Car- bonell, Ruslan Salakbutdinov, and Quoe Le. 2019b. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in Neural In- formation Processing Systems 32: Annual Confer- ence on Neural Information Processing Systems 2019, NeurlPS 2019, December 8 8-14, 2019, Vancouver, BC. Canada.
We use the original BERT our backbone and vary the positional encodings to pre-training different variants for comparison. Listing 1 shows a code example about how to inject handcrafted po- sitional encodings into the BERT backbone. Fach variant is fine-tuned on the training dataset with different learning rates (among 9e-5, 7e-5, 5e-5, 3e- 5, le-5), After, we evaluate the fine-tuned model the Dev set and report the average score of five learning rates. Apart from BERT, we introduce the TUPE model as another baseline. Specifically, we pre- pre-train train the following variants:
A Appendix
A.1 Visualizations of Positional Encodings
To understand what positional encodings learn after pre-training, we visualize the positional weights in attentional heads. The Identical Word Prob- ing is adopted in this experiment (Wang et al., 2021). The used pre-trained language models are shown in Table AI, and the repeated words are randomly selected from the corresponding vocab- ulary. Note that sub-tokens like single characters and non-physical words are removed. For visualiza- tion, adopt the Identical Word Probing proposed by Wang et al. (2021), which feeds many repeated identical words to pre-trained language models and thus the attention values are disentangled with con- textual weights. More specifically, we randomly select 100 words from the corresponding vocabu- lary (filtering out single characters and sub-words such as "##nd""). We repeat each word to compose a sentence of length 128. These 100 sentences are fed into language model and the attention weights across different layers are averaged as the positional weight matrix of particular language model.
class Melddeedde FAstent Markula): Multi-Reesde
Dot (s config):
Attention
apper....nit...... 1nitt.....
selr-n_heads con config.n_hhan headss
dropp on (eopf
(oonfiggun sudroppatto
selr. projar en. in. Loneartoonfts (config din, config, diu)
wil proj.k k. mn inea (omnfig din, config. dis)
proj.y Ln.Linar
contig.div)
forsard(self, x, mack, pe):
a(query). k(key), e(valus) (B(tatch_size)
5(sea Jen), D(d(t))
nask (3(batch (iize) S(ane_len))
posittoral weights (S(bstch_sizze) (((kennnaeer)).
((wen_le)) solit o(din) into (lKnnheedds),
head))
* (8, D) -proj-> (8, 5, spplit>> (8, S, -trans-> (0, H. W)
k, v salr se1r. pro__er)))))) () self, proj..po seot.prod_
(aplit_laet(e, n_twwin
-:33.transpase(), for [q.
(8, H,
9) 5.
-softanc-> (b, Bi,
scores 0 transpoosee-...... -1) ne.sert s12e(-11)
t sndect positional weights into contextual weights (8, H, 53 (8. 5, 5) (8,
mask is not nase maak[: Rone, None, (].flaxt)) scores 18890.8 (1. noss)
scores selt seit.croptt wtw.orooooooooooosssoooooooooo soforax dimm-13) (8, H, 5) 6 (8. (8, 5, tr.araaspeeet,, 2).contiquos())
(B, H. -trans->
A.2 Details of Pre-training
-megee-> (B. 5, eergo_laat(h,
We the configuration of the original BERT base (Devlin et al., 2019) with 110M parameters for pre-training. Our model is implemented with Py- Torch using the pytorchic-bert tool?. The number layers, attention heads, and the projection di- mension are 12, 12, and 768 respectively. We use the original vocabulary with size of 30522. The training corpus is the English Wikipedia (20200101 dumps), which totals 13G after preprocessing by WikiExtractor. We pre-train with sequences of at most T = 512 tokens and the batch size 64 to pre-training 600K steps. The optimizer Adam with learning rate of 5e-4, B1 0.9, f22 0.999, L2 weight decay of 0.01, and a warmup rate of O.I. The dropout probability is always set as 0.1.
return
Listing 1: how inject handcrafted positional encodings into self-attentions.
code example of
BERT is the original one and we use it as a baseline.
BERT-A" is a variant of the former, but the encodings are learnable during pre-training.
BERT-A"-s shares learnable positional encod- ings within layer.
Suppose that the hidden dimension 768, the layer number 12, the head number is 12, and the max- imum length is 512 for BERT base model, we can

---

Model
Size
Version
Language
BERT
110M
bert
English
DeBERTa
100M
microsotbdebettt buse
English
XLNet
110M
siact bare bare-cased
English
Table AI: Details of pre-trained language models used in visualizations.
Model
Size
Version
Fine-tuned by
BERT
110M
bert-buus-wweeeseed
ALBERT
223M
juivlalberr-ssargg---.o... fover fover
R3mil
DeBERTa
100M
microsalideberttebuus
XLNet
340M
yulefstnettlorge--ooedd.. muli jevier A1_22 R3-ali
StrueBERT
340M
beyerrtogg/rmncbbee-ooreee
Table A2: Details of pre-trained language models used in word swap probing.
calculate the size for each variant. The number of parameters of handcrafted positional encoding for each head is 262K (512 x 512). If positional heads are different across all layers, the total cost is 37.7M (512 x 512 x 12 x 12). If the positional encodings are shared across all attentional heads, the total cost is 3. (512 x 512 x 12).
subset from the SNLI test set. This algorithm is applied successively to the premise and hypothesis sentence for a sample whose label is entailment, and if the result of either of them is not null, we consider it valid shuffled sample. which means only shuffle the premise hypothesis. After, we can obtain new sample and the pair of sen- tences are contradicted with each other. In total, there are 1329 samples. To ensure that all sentences are semantically correct, we manually selected 300 pairs from them. The last two rows in Table 2 show two examples in the Shuffle-SR dataset.
A.3 Word Swap Probing
To validate if language models with positional encodings are sensitive to the local and global word swaps, we construct Shuffle-z and Shuffle-SR SNLI datasets. Shuffle-r means the word orders of phrases with length x are disrupted, e.g. "an electric guitar" is 3-gram phrase, and it might be "guitar electric" in Shuffle-3 SNLI. In this way, new sentence with the same meaning can be obtained and therefore the initial label of the sam- ple will not be changed. To construct such shuffled datasets, the premise sentences in the SNLI test set are shuffled and we keep the hypothesis sentences intact. Here, we let x E [3, 5] and select subset from SNLI to make sure that every premise sen- tence has at least one phrase with length from 2 to 5, We select five types of target phrases for shuffling: Noun Phrase, Verb Phrase, Prepositional Phrase, Adverb Phrase, and Adjective Phrase. Finally, Shuffle-z SNLI is obtained by disrupting the order inside phrase with length x and the size for each shuffle-w is around 5000. The first fourth rows in Table 2 shows some samples.
To probe the capabilities of language models on our newly constructed datasets, we adopt five dif- ferent pre-trained language models (as shown Table A2) and we use Hugging Face for implemen- tation (Wolf et al., 2020). These models are fine- tuned on the training set of SNLI, and the model with the best score on the validation set is stored for the following experiments. Note that there off- the-shell ALBERT and XLNet for natural language inference, we therefore use them directly without fine-tuning. During the fine-tuning stage, the max- imum length of the tokenized input sentence pair is 128, and the optimizer is Adam (Kingma and Ba, 2015) with learning rate of 2e-5. The batch size is 32 and the epoch is 3. After fine-tuning, the best model is evaluated our shuffle SNLI test set, and record their performances when faced with local and global word swaps.
A.4 Details of Downstream Datasets
As for the Shuffle-SR SNLI dataset, the seman- tie roles of the agent and patient are swapped in a sentence. We use the Algorithm to collect a
SentEval is based on a set of existing text clas-

---

0832
0.00
0.22
0420
0520
0.24
0:24
0322
0:222
0422
0.19
(e) Correlations between Locality and Accuracy
(b) Correlations between Symmetry and Accuracy
Figure A1: Correlations between the two properties (Locality and Symmetry) and accuracy SUBJ dataset (Pang and Lee, 2004). The yellow line shows the locality or symmetry of the pre-trained BERT.
Algorithm 1: Construction of Shuffle-SR
some particular tasks only contain one dataset, e.g.. sentiment analysis and textual similarity, Moreover, the size of WNLI in GLUE is rather small and the GLUE webpage notes that there are issues with the construction of this dataset 3. To better evaluate the capability of models for sentence representation, we, therefore, select 10 datasets from SentEval and GLUE, covering four types of sentence-level tasks:
Sentences
Input:
premise or hypothesis sentence in
SNLI,
A: Auxiliary verb list
M: Semantic Role Labeling Model,
D: Subject and Object Case Mapping
// e.g., I + me Output: A sentence s* with shuffled agent and patient or None
Sentiment Analysis is also known as opinion mining, which aims to classify the polarity of a given text, whether the expressed opin- ion is positive, negative, neutral. We use MR (Pang and Lee, 2005), SUBJ (Pang and Lee, 2004), and SST (Socher al., 2013) for this task.
R Predict the semantic roles of words in sentence s by using the model M
2 V+ Take the verb list from R foreach verb v in V do
if c appears in A then
continue
Textual Entailment describes the inference relation between a pair of sentences, whether the premise sentence entails the hypothesis sentence. Actually, this a classification task with three labels: entailment, contradiction, and neutral. Here, we use QNLI (Rajpurkar et al., 2016), RTE (Dagan et al., 2005; Haim et al., 2006; Giampiccolo al., 2007: Ben- tivogli al., 2009) and MNLI (Williams al., 2018) for evaluation. Note that report the average score for the two test sets of MNLI.
R does not contain an agent or patient then
continue
< Take the agent and patient from
R
Swap the a, p in sentence Transform the subject and object
10
case in s* if or p return
11
return None
Paraphrase Identification is to determinc whether a pair of sentences have the same meaning. We use MRPC (Dolan and Brockett, 2005) and QQP data.quura.com fee---oorr--tttttt--eeee---eeeee qusllaass--usstioo-PPaar] for evaluation.
sification tasks involving one sentences as input. However, most tasks in SentEval closely related sentiment analysis and thus not diverse enough. GLUE benchmark introduces series of difficult natural language understanding tasks while
hnttes//gllueeennharrk.co/////

---

Textual Similarity deals with determining how similar two pieces of text are. We use STS-B (Cer et al., 2017) and SICK-R (Marelli et al., 2014) for evaluation.
class hultilleshssttiom/lttetttoo Module):
Nulti-tadedd Sealed Doc Product Attention
def _neit____nlf f, config):
super Intt welf nuhwads config rcheads self, ww.Dropo.tccnnfic.o (udeoennnngg..do...ttt
def Forsaro(self, x, rask, pe):
3, a(qurrs). k(key), v(valus) : (B(batel_s12))
A.5 Details of TUPE Model
S[see, Zen), ((dte))
mask (S(hateh_sia)) siize) S(see_Jen))
positional (B(batch, stibt)
(uhea))r Senn,_len), s(xen_ee)) o(din)
absolute positional encoding, the positional en- coding is added together with the contextual encod- ing:
(linnheecs)) Miwodth heao))
# (B, 5, D) -prof-> (B, S, -split-> (B. 5, -trans-> (8. B. ()
k, v "split_lastte. (seIT.n_heass,
trpp.transpose(),
N)
(x: + +
(Q
Pj) W
-softree>> (8, H,
Oij
Vd
(A.1)
Scores = pe if rask is not Kone: scores, rasked r111_ (>w2sk, 0.)
S. (scores v). transposee), 2).contiqvous))
where P: C Rdd is position embedding of the i-th token. Further, the above equation can be expanded
mergers (8, S. h rerge_ return
Listing 2: A code example of the Positional Attention.
(x, (x, (x, (p,,, w Vd (p,w)) WK)T Vd
ag
Vd
A.6 Linguistic Roles of Positional Ecnodings
(p. (x, WHIT +
Positional and contextual weights are usually en- tangled in every attentional head, and therefore the behavior of positional encodings cannot be ob- served independently (as shown in Equation 4),
Vd
(A.2)
To address this, we the contextual correlation Yi (in Equation 4) to zero (instead of removing the contextual encodings completely) and thus the attentional weight Ci,j only depends on positional correlation. Note that this operation does not alter the strueture of the original network because a soft- max layer is applied to the vector aq, and the output is still attentional weight vector that be garded kind of discrete probability distribution. Therefore, the output sentence representation is decoupled from contextual encoding. We refer to this adapted model as BERT-p. For comparison, we remove the positional correlations ō.,j to ob- tain BERT-c. We do the same for a pre-trained TUPE model, to obtain TUPE-p and TUPE-c, and already described TUPE in Section A.5.
There are four terms in this expression: context- to-context, context-to-position position-to-contex,, and cisation-to-posttion While the first and the fourth term are intuitive, the token encodings and positional encodings do not have strong correla- tions with each other, and the context-position correlations may even induce unnecessary noise. Based on this analysis, Ke et al. (2021) propose TUPE (Transformer with Untied Positional Encod- ing) that removes the second and third redundant terms and introduces different parameters for the position encoding:
(x, (x) WKYT + (p:U9)p, UK)T
aij
Vd
(A.3) Here, Uº and K are weights that need to be learned, capturing positional queries and keys, re- spectively. Their empirical results confirm that the removal of the two context-to-positin terms consistently improves the model performance on a series of tasks.
A.6.1 Linguistic Probing Tasks
In this linguistic probing, adopt widely used 10 probing tasks (Conneau et al., 2018) with a stan- dard evaluation toolkit (Conneau and Kiela, 2018). The following are the details of each probing task, including three categories:

---

SentLen (Surface) aims predict the length of sentences in terms of the number of words, and the dataset constructed following Adi et al. (2017).
Relation
Distunce
conteciual
Booh
Rodt
0.0
99.3
3.8
86.5
auxpasa
1.1
44.6
91.1
92.9
cospound
1.2
75.0
70.6
det
51.6
WC (Surface) means word content, which checks whether it possible to recover in- formation about the original word from the embedding of the sentence.
advined
2.1
16.5
62.
3K.7
pob:
2.3
9.0
33.9
46.3
nsub]
2.3
13.4
58.2
52.6
2.3
159
31.7
43.5
prep
mark
4.1
25.1
21.3
64.8
BShift (Syntactic) means bigram shift. In this task, two random adjacent words in sen- tence swapped and the goal is to detect if model is sensitive to legal word orders.
cont
4.9
25.1
31.2
33.6
punct
73
25.9
30.3
32.7
advel
7.6
18.4
9.8
20.1
811
29.0
12.4
33.2
short
5.4
28.4
54.3
61.6
ling
3.4
21.0
36.8
Magro
TreeDepth (Syntactic) tests whether model can infer the depth of the syntactic tree of sentences.
Table A3: Evaluations of predictions of de- pendency relations on MRPC dataset. The top 20 common relations distinction of "short" and "long" is whether the average length of relation is greater than 4.
shown. The
TopConst (Syntactic) whether model can recognize the top constituents of the sen- tence, c.g., "[Then] [very dark gray letters on a black screen] (appeared) {-J" has top constituent sequence: "ADVP NP ". This dataset is first introduced by Shi et al. (2016).
each model, as observed by Jawahar et al. (2019). For surface tasks, the surface knowledge is stored more in the bottom layer, syntactic knowledge is in the middle layer and semantic knowledge is in the middle and top layer. Therefore, we conclude that positional encodings play more of role at the syntactic level tasks. On semantic tasks, especially position-independeen ones, contextual encodings are more important.
Tense (Semantic) asks for the tense of the main clause verb.
SubjNum (Semantic) focuses on the number of the subject of the main clause.
ObjNum (Semantic) tests for the number of the direct object of the main clause.
A.6.2 Dependency Analysis of Positional Encodings
SOMO (Semantic) checks the sensitivity of model random replacement of noun or verb.
The detailed scores of each relation are shown in Table A3. We find that positional attentional heads outperform contextual heads short-distance lations, e.g., auxpass and compound. Contextual attention can capture better long-distance relations than positional attention while contextual attention itself has a certain degree of ability to detect some long-distance relations such as conj and punct. Note that there exists head in contextual attention maps attending the token itself, therefore, the score on the Root relation is the best.
CoordInv (Semantic) tests whether model can recognize the order of clauses inverted.
Figure A2 gives the results. We first observe that the combination of contextual and positional encodings can have better performances across all probing tasks (yellow lines). Secondly, compared to contextual encodings, positional encodings per- form better on syntactic tasks (TreeDepth, Top- Const, BShift), which require more information of word orders. On semantic tasks, contextual en- codings outperform positional encodings Tense and ObjNum while performing poorly when the semantic probing tasks require order information (CoordInv). Thirdly, hierarchical structure exists here when we check the peak of probing tasks for
A.7 Visualizations of Positional Weights
In Figure 1, visualize the averaged positional weights of various pre-trained language models and identify they have similar visualized results. However, We find that the behavior of positional encodings very diverse across attention heads. Note that there are 144 attentional heads (12 layers

---

WCC
TreeDooh
TreCooal
DOWS
TEE Tarbe
196
Figure A2: Results of linguistic probing tasks across different layers. BERT-based models are shown a triangle while TUPE-based models are shown in circle. The red, blue and yellow lines represent the of positional weights, contextual weights and combination of both, respectively.
12 heads) for the BERT buse model. For example, the visualizations of BERT (Figure A3) validate this phenomenon. Besides, we observe that BERT exhibits a hierarchical structure: positional weights lower layers are nearly uniform (Layer-4), mid- dle layers attend more to local units (Layer-7) and higher layers demonstrate the asymmetric property (Layer-12). 12). We also visualize all the positional heads in BERT-A" (Figure A4).
Visualization Analysis of BERT-A". BERT-A" outperforms BERT by 3.1 percentage points on av- erage across 10 downstream tasks. The main differ- ence between BERT-A" and BERT is the learnable handcrafted positional encodings. For visualiza- tions, we take the positional weight óúj in Equa- tion instead of Identical Word Probing. Figure A4 shows that most positional heads perfectly satisfy the properties of locality and symmetry, which can bring better inductive bias for sentence represen- tations. Another observation that the diagonal bandwidths are diverse across positional heads af- ter learning, which means proximity units can be combined at different distances. We conclude that, compared to randomly initialized positional encod- ings, the encodings initialized with locality and symmetry properties lead to better sentence repre- sentation models.

---

LANA PARST
LANA IMASS
LAWAY FRAST
Loped BMACE
Leyend
Japeo (ockl
- 1l0
Loundh 1
Loyed Heattt
Leyenc
Lapwc. look
Figure A3: Visualizations of positional weights of BERT across all layers. The weights are computed by Identical
Word Probing. Red color means lower values and blue color means higher values.

---

Road
Head
Heads
Hean
Leyen Meac
Heard
Saper Roudd
Poodd
Feed
Head
Libel
Heww Howk!
Leyed Head
Fnwed Hena
Heen Need
lewe Aordd
2
Hewe
Lewel Head
Lewel leek?
Leee Heek?
Lownl Men.
Leand Heek
Levnd Heed
Levnd Mese?
Laped lenkd
Lweed Headd
Lewed (146
Loand Me6k12
Lower Heads
Lover Heaks
Lower Halie
Liwils Meioe
Leni M4077
Lewel Heice
Loner Heash
Lowers
Livel Heskh1
Lieil Mo
Saper Heals
kew Hewl
18 Heal
Mye Met
Lae 10 Hail
Saye Hear
10 Heal
Lape 13 Heall
kew Heal
Laye Heal
Hew
Heed
Lanett Heod
Figure A4: Visualizations of positional weights of BERT-A" across all layers. Red color means lower values and blue color means higher values.


Reviews:
[{"review_id": "b7icqlS2ur", "reviewer": "Reviewer_5QeA", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 5QeA,\n\nWe appreciate that you found our work interesting and our experiments provided convincing conclusions. We also thank you for your critical comments. \n\n**Q1:**\n\n> *The experiments are merely implemented with BERT-style encoder-only models, whose scales are no larger than BERT-large. The contribution of this work is limited, since many nowadays popular models are large decoder ones like GPT-3.5 and LLaMA.*\n\n**A1:**\n\nOnly Bidirectional Masked Language Models are included in this work because there is an obvious distinction between them and Decoder-only models. \n\nDecoder-only models like GPT predict the next word based on previous words in the sentence (Left-to-Right mode), and therefore implicitly introduce directional information,  which violates the symmetry property. \nFurthermore, recent studies have shown that decoder-only transformers without positional encodings are able to achieve competitive or even better performance than other explicit positional encoding methods [1, 2], which shows the distinction of decoder-only models.\n\nWhile GPT and LLaMA have attracted a lot of attention, bidirectional language models are still widely used today,  e.g., for sentence embeddings [4,5], Named Entity Recognition [6, 7] and Retrieval-based Models [8,9]. Therefore, the research of their positional encodings is valuable.\n\n\n**Q2:**\n\n> *The contributions of pre-training and design of position encodings are not clear. As mentioned in L379-383, learned encodings may not have the best locality and symmetry, but they can still represent contexts. There might be non-positional information but implicitly indicates the positions of tokens, for example, commonsense. We probably say a dog is biting a person, but rarely say a person biting a dog. It might affect the model performance of downstream tasks like NLI, since models can inference merely by part of keywords.*\n\n**A2:**\n\nThank you for raising this interesting question. \nTo study the effect of varying locality and symmetry, we designed the attenuated encoding that can be parameterized along the two metrics. We are able to observe the performance of a model at a particular locality and symmetry value. In the pre-training setting experiment, a model is initialized with locality and symmetry that have the same values as the learned positional encodings in BERT and the results in Table 1 validate that better locality and symmetry can yield better inductive bias.\n\nAs the example you gave, we agree that there exists some implicit position information, e.g., the special token [CLS], which always appears in the first position and can be regarded as a reference point to estimate positions. However, the usage of positional encodings is essential for transformer models, as shown by prior studies [2,3]\n\n[1] The impact of positional encoding on length generalization in transformers. arXiv 2023\n \n[2] What Language Model to Train if You Have One Million GPU Hours?. EMNLP Findings 2022\n\n[3] Constituency Parsing with a Self-Attentive Encoder. ACL 2018\n\n[4] WhitenedCSE: Whitening-based Contrastive Learning of Sentence Embeddings. ACL 23\n\n[5] Composition-contrastive Learning for Sentence Embeddings. ACL 23\n\n[6] An Embarrassingly Easy but Strong Baseline for Nested Named Entity Recognition. ACL 23\n\n[7] Focusing, Bridging and Prompting for Few-shot Nested Named Entity Recognition. ACL Findings 23\n\n[8] RECAP: Retrieval-Enhanced Context-Aware Prefix Encoder for Personalized Dialogue Response Generation. ACL 23\n\n[9] Retrieval-Based Transformer for Table Augmentation. ACL Findings 23", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. It is interesting to study the Symmetry property of position encodings and to discuss its drawback.\n2. The paper contributes new probing tasks of shuffle words, and provides experimental analysis to prove models can be robust against local swaps but sensitive to global swaps.\n3. The authors examine several encoding methods and many pre-trained models, which provides convincing conclusions.", "reasons_to_reject": "1. The experiments are merely implemented with BERT-style encoder-only models, whose scales are no larger than BERT-large. The contribution of this work is limited, since many nowadays popular models are large decoder ones like GPT-3.5 and LLaMA.\n2. The contributions of *pre-training* and *design of position encodings* are not clear. As mentioned in L379-383, learned encodings may not have the best locality and symmetry, but they can still represent contexts. There might be non-positional information but implicitly indicates the positions of tokens, for example, commonsense. We probably say a dog is biting a person, but rarely say a person biting a dog. It might affect the model performance of downstream tasks like NLI, since models can inference merely by part of keywords.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The authors argue that position encodings have two main properties, Locality and Symmetry. They also propose novel probing tasks to quantitatively study these two properties of fixed or pre-trained position encodings. They further reveal existing pre-trained language models are robust to process constituency-shuffling texts but fail to represent semantic-role-shuffling ones.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Z7Crgoc3Y4", "reviewer": "Reviewer_ftCy", "summary": "", "questions": "A. I’m missing a more detailed investigation if really symmetry is the reason that the models perform poorly on the SR probing task. Can't there be other reasons for the observed effect? Maybe I'm misunderstanding, but in Section 3.4 you even claim that PEs matter more at the syntactic level.\n\nB. I also had a look at the code: In classify.py, the if statements for fp / fn / tn (starting in line 364) for one metric (MCC on COLA) appear to be incorrect. This is a major concern if this code was used to compute the values presented in the paper. Can you clarify?\n", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer ftCy,\n\nWe would like to thank you for your time and detailed feedback. We are happy that you found our research topic interesting and our proposed probing tasks useful for future studies.\n\n**Q1:**\n\n>*Eq. 7 and 8 define symmetry, which is one of the key concepts of the paper. However, the equations do not appear to match the original symmetry as proposed by Wang et al.. I'm not convinced that these equations measure proper symmetry of the attention matrix (as only values of a single attention weight vector are used).*\n\n**A1:**\n\nThank you for pointing us to this issue. Equation 7 and 8 is indeed stated incorrectly in the paper. We apologize for this. The code, though, is correct (attached below for your reference). \nOur symmetric metric is different from the one proposed by Wang et al.\nbecause we aim to measure whether a positional weight vector $\\epsilon_i$ is symmetrical to the left and right at the current position $i$, and the original metric is used to check the symmetry of a matrix. \nFor example, we have a sentence *\"cats eat mice\"*  and a positional weight vector for the position of *\"eat\"*, $[0.25, 0.5,0.25]$, which is generated by a particular encoding. Then the vector perfectly satisfies symmetry. However, we would like to argue that the encoding is insensitive to the change of word orders like *\"mice eat cats\"*.\n\nBelow is the code for calculating the symmetry of a matrix, and we average values of all vectors as the matrix-level symmetry, which reflects a general pattern of PEs. \n```\ndef cal_matrix_symmetry(M: List[List[float]]) -> float:\n    seq_length = len(M)\n    sum_symmetry = list()\n    for index in range(seq_length):\n        left, right = M[index][0:index], M[index][index + 1:seq_length]\n        l_len, r_len = len(left), len(right)\n        min_seg_len = min(l_len, r_len)\n        if min_seg_len == 0: continue\n        left_seq, right_seq = left[-min_seg_len:], list(reversed(right[:min_seg_len]))\n        vec_sym = np.array([abs(left_seq[j] - right_seq[j]) for j in range(len(left_seq))]).reshape(-1, 1)\n        normalized_vec_sym = min_max_scaler.fit_transform(vec_sym)\n        sum_symmetry.extend([v[0] for v in normalized_vec_sym])\n    symmetry = 1 - sum(sum_symmetry) / len(sum_symmetry)\n    return symmetry\n```\n\n**Q2:**\n\n>*For the results reported in the non-pre-training setting (Section 3.3.1.), improvements in accuracy are rather small (< 2 pp).*\n\n**A2:**\n\nIn this experiment, our goal is to validate there is a correlation between our proposed two metrics and performances of downstream tasks, and we do not claim that our positional encodings can bring any improvements.\n\nMoreover, we accounted for variance in this experiment and the results are statistically significant. As described in line 310, each model is trained for 5 epochs and we select the best model on validation sets to evaluate on the test set. We repeat this procedure 5 times and use the average score to report.\nTo draw Figure 3, we repeat this procedure ~100 times to obtain the accuracy of models with different locality and symmetry values.  \n\n\n**Q3:**\n\n>*I’m missing a more detailed investigation if really symmetry is the reason that the models perform poorly on the SR probing task. Can't there be other reasons for the observed effect? Maybe I'm misunderstanding, but in Section 3.4 you even claim that PEs matter more at the syntactic level.*\n\n**A3:**\n\nThe symmetry property is one of the main causes of the effect. \nAs the example in A1, we have a sentence *\"cats eat mice\"*  and a positional weight vector for the position of *\"eat\"*, $[0.25, 0.5,0.25]$, which is generated by a particular encoding. Then the vector perfectly satisfies symmetry. However, the encoding is insensitive to the change of word orders like *\"mice eat cats\"*.\nAnother reason is that most existing pre-training tasks like masked token prediction do not explicitly require positional information. In Table 3, the better performance of StructBERT on the Shuffle-SR suggests that introducing additional order-sensitive training tasks may improve this behavior since it explicitly models language structures by forcing it to reconstruct the right order of words during pre-training.\n\nOur empirical results show that there is a clear distinct role between the positional\nand contextual encodings in sentence comprehension: positional encodings play more of a role at the\nsyntactic level while contextual encodings serve more at the semantic level (if the semantic task does\nnot require word order information), as shown in Figure A2 in the appendix. \n\n**Q4:**\n\n> *I also had a look at the code: In classify.py, the if statements for fp / fn / tn (starting in line 364) for one metric (MCC on COLA) appear to be incorrect. This is a major concern if this code was used to compute the values presented in the paper. Can you clarify?*\n\n**A4:**\n\nWe agree with your statement, but the CoLA task is not included in our experiment. \nWe appreciate the time you spent checking our code carefully, and we will use it to correct this piece of code even if it is not currently used.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Research of PEs is an interesting topic and relevant to EMNLP.\n\n- The paper includes an extensive analysis of positional encodings on different NLU tasks. It also shows that initializing BERT with symmetric / localized PEs can significantly improve results on some NLU tasks.\n\n- The proposed probing tasks seem useful to further investigate the effect of word swaps.\n", "reasons_to_reject": "Eq. 7 and 8 define symmetry, which is one of the key concepts of the paper. However, the equations do not appear to match the original symmetry as proposed by Wang et al.. I'm not convinced that these equations measure proper symmetry of the attention matrix (as only values of a single attention weight vector are used).\n\nFor the results reported in the non-pre-training setting (Section 3.5.1.), improvements in accuracy are rather small (< 2 pp).\n\n", "questions_to_authors": "A. I’m missing a more detailed investigation if really symmetry is the reason that the models perform poorly on the SR probing task. Can't there be other reasons for the observed effect? Maybe I'm misunderstanding, but in Section 3.4 you even claim that PEs matter more at the syntactic level.\n\nB. I also had a look at the code: In classify.py, the if statements for fp / fn / tn (starting in line 364) for one metric (MCC on COLA) appear to be incorrect. This is a major concern if this code was used to compute the values presented in the paper. Can you clarify?\n", "paper_topic_and_main_contributions": "The paper investigates positional encodings for transformer-based LMs. It suggests two properties, symmetry and locality, as important properties for analyzing the influence of positional encodings. \n1. by calculating the symmetry and locality of different LMs with learnable PEs, it is shown that not only are the symmetry and locality learned during pre-training, but that these values are also similar to those of fixed PEs.\n2. A function is presented to create PEs with adjustable locality and symmetry values, which is then used to examine the effect of both properties on different NLU tasks (such as Sentiment Analysis) on different datasets. For some datasets, significant improvements are reported by tuning the PE initialization.\n3. Two probing tasks of word-swaps are proposed to study the sensitivity of LMs wrt. word swaps. The first variant swaps words in a phrase of a sentence (bad syntax, semantics tends to be preserved), the second variant swaps agent and patient phrases in a sentence (semantics tends to be changed). By shuffling and evaluating on SNLI, it is shown that LMs can handle the first variant but perform poorly on the second.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "-"}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Only two variants are compared with the original BERT (line 352)\n\n- Theta is not part of Eq. 4 (or any other equation in the paper) (line 290, 292)\n\n- “electicity guitar” → “electric guitar” (line 518)\n\n- Some sentences are incomplete (e.g. line 262, 582)"}}, {"review_id": "3TPK0FsrVZ", "reviewer": "Reviewer_R38Q", "summary": "", "questions": "- A: I still do not fully understand the calculation of the Symmetry metric. For a specific entry with “j”, why is it the difference between the values with indexes of j and n-j+1? Shouldn’t “i” (the current position) also be involved here?\n- B: Better designed positional encodings may also enjoy the benefits of faster convergence, and I’m wondering if such phenomenon could be observed in some of the training experiments?\n", "limitations": "", "rating": 3, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer R38Q,\n\nWe appreciate the time you spent reviewing the manuscript. Also, we are encouraged that you find our analyses are comprehensive and our findings may inspire future studies. \n\nFirst, we would like to clarify one comment you gave in the Main Contribution part:\n\n**Q1:**\n\n>*This work provides a detailed analysis on the positional encodings in the BERT model*\n\n**A1:**\n\nIn this work, we investigated various positional encodings (absolute and relative), not limited to the encodings used in BERT. For example, Figure 2 shows the visualizations of the locality and symmetry for distinct positional encodings.\n\nNext, we address your main concerns and questions.\n\n**Q2:**\n\n>  *Only BERT-styled models are investigated, while it would be better if decoder-only and encoder-decoder models can be analyzed as well*\n\n**A2:**\n\nOnly Bidirectional Masked Language Models are included in this work because there is an obvious distinction between them and Decoder-only models. \n\nDecoder-only models like GPT predict the next word based on previous words in the sentence (Left-to-Right mode), and therefore implicitly introduce directional information,  which violates the symmetry property. While GPT and LLaMA have attracted a lot of attention, bidirectional language models are still widely used today, e.g., sentence embeddings [1,2], Named Entity Recognition [3, 4] and Retrieval-based Models [5,6]. Therefore, the research of their positional encodings is valuable.\n\n**Q3:**\n\n> *The metrics are calculated using the Identical Word Probing method, which takes sequences with repeated identical words. This may be different to the real scenarios with real natural language sequences. It would be nice if more different ways for the measurement of positional encodings can be explored.*\n\n**A3:**\n\nThe Identical Word Probing is used only for visualizations (Figure 1 and 3), and the two metrics of our designed positional encodings can be computed directly, as described in Section 3.3. Hence, the experimental results in Section 3.3.1 and 3.3.2 are unrelated to Identical Word Probing and we reported results across multiple real natural language datasets.\n\nMore importantly, using identical word probing is possible to measure the behavior of positional encodings in Transformers. \nWe theoretically demonstrate that the attentional weight $\\alpha_{i,j}$ can be divided into two components (in Equation 4): contextual correlations $\\gamma_{i,j}$ and positional correlations $\\delta_{i,j}$.  We feed many randomly selected identical words and average $\\alpha$ over many words, then $\\alpha$ is only related to $\\delta_{i,j}$ and can reflect general behaviors of positional encodings.\n\n**Q4:**\n\n>*I still do not fully understand the calculation of the Symmetry metric. For a specific entry with “j”, why is it the difference between the values with indexes of j and n-j+1? Shouldn’t “i” (the current position) also be involved here?*\n\n**A4:**\n\nWe made a mistake in Equation 8 and it does not match the code used in our experiments.\nWe apologize for this.\n\nFor understanding, we provided the code for calculating the symmetry of a matrix, and we average values of all vectors as the matrix-level symmetry, which reflects a general pattern.\nThe current position is not involved since we aim to measure whether the positional weight vector is symmetrical to the left and right at the current position $i$.\n\nThe core idea of this metric is to measure whether a positional weight vector $\\epsilon_i$ is symmetrical to the left and right at the current position $i$. \nFor example, we have a sentence *\"cats eat mice\"*  and a positional weight vector for the position of *\"eat\"*, $[0.25, 0.5,0.25]$, which is generated by a particular encoding. Then the vector perfectly satisfies symmetry. However, we would like to argue that the encoding is insensitive to the change of word orders like *\"mice eat cats\"*.\n\n```\ndef cal_matrix_symmetry(M: List[List[float]]) -> float:\n    seq_length = len(M)\n    sum_symmetry = list()\n    for index in range(seq_length):\n        left, right = M[index][0:index], M[index][index + 1:seq_length]\n        l_len, r_len = len(left), len(right)\n        min_seg_len = min(l_len, r_len)\n        if min_seg_len == 0: continue\n        left_seq, right_seq = left[-min_seg_len:], list(reversed(right[:min_seg_len]))\n        vec_sym = np.array([abs(left_seq[j] - right_seq[j]) for j in range(len(left_seq))]).reshape(-1, 1)\n        normalized_vec_sym = min_max_scaler.fit_transform(vec_sym)\n        sum_symmetry.extend([v[0] for v in normalized_vec_sym])\n    symmetry = 1 - sum(sum_symmetry) / len(sum_symmetry)\n    return symmetry\n```\n\n\n**Q5:**\n> *Better designed positional encodings may also enjoy the benefits of faster convergence, and I’m wondering if such phenomenon could be observed in some of the training experiments?*\n\n**A5:**\n\nWe have done such an experiment. Specifically, we recorded loss values of training and validation and observed that a BERT model equipped with our hand-crafted positional encodings can consistently achieve lower training and validation loss than the original BERT using the same steps, which proves that better designed PEs can have better convergence speed.\n\nThe table below shows training loss values of the original BERT and BERT-A* with better initialized positional encodings.\n\n|  Model   | 5000|10000|15000|20000|25000 (steps)|\n|  ----  | ----  | ----  | ----  | ----  | ----  |\n| BERT  | 7.4 | 7.2 |7.1 |7.0 |7.0 |\n| BERT-A*  | 7.2 |6.9 |6.8 |6.7 |6.6 |\n\n\n\n[1] WhitenedCSE: Whitening-based Contrastive Learning of Sentence Embeddings. ACL 23\n\n[2] Composition-contrastive Learning for Sentence Embeddings. ACL 23\n\n[3] An Embarrassingly Easy but Strong Baseline for Nested Named Entity Recognition. ACL 23\n\n[4] Focusing, Bridging and Prompting for Few-shot Nested Named Entity Recognition. ACL Findings 23\n\n[5] RECAP: Retrieval-Enhanced Context-Aware Prefix Encoder for Personalized Dialogue Response Generation. ACL 23\n\n[6] Retrieval-Based Transformer for Table Augmentation. ACL Findings 23", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This work provides a comprehensive investigation and analysis on the positional encodings in BERT, and some of the findings seem interesting and may inspire more future analysis.\n- The proposed swapping experiments seem interesting, indicating the utilization of linguistic structures in the analysis of pre-trained models.\n", "reasons_to_reject": "- Only BERT-styled models are investigated, while it would be better if decoder-only and encoder-decoder models can be analyzed as well.\n- The metrics are calculated using the Identical Word Probing method, which takes sequences with repeated identical words. This may be different to the real scenarios with real natural language sequences. It would be nice if more different ways for the measurement of positional encodings can be explored.\n", "questions_to_authors": "- A: I still do not fully understand the calculation of the Symmetry metric. For a specific entry with “j”, why is it the difference between the values with indexes of j and n-j+1? Shouldn’t “i” (the current position) also be involved here?\n- B: Better designed positional encodings may also enjoy the benefits of faster convergence, and I’m wondering if such phenomenon could be observed in some of the training experiments?\n", "paper_topic_and_main_contributions": "This work provides a detailed analysis on the positional encodings in the BERT model. Two core properties, locality and symmetry, are investigated. It is discovered that models with suitable properties can lead to better inductive bias and better performance through both non-pretraining and pretraining settings. Finally, two probing tasks are designed, showing a weakness of the encodings with the symmetry property, which may lead to insensitivity to semantic role swapping.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "L262: incomplete sentence."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Paper studies positional encodings of encoder only models for NLP (BERT) and proposes probing tasks and formulation to investigate symetry and locality of the positional encodings. Reviewers raised some concerns about the experimental design and formulation.
Decision: Accept-Findings