Rethinking Positional Encoding in Language Pre-training
Authors: Guolin Ke
Di He
Tie-Yan Liu
Venue: ICLR 2021 Poster (2021)
Keywords: Natural Language Processing
Pre-training

Abstract: In this work, we investigate the positional encoding methods used in language pre-training (e.g., BERT) and identify several problems in the existing formulations. First, we show that in the absolute positional encoding, the addition operation applied on positional embeddings and word embeddings brings mixed correlations between the two heterogeneous information resources. It may bring unnecessary randomness in the attention and further limit the expressiveness of the model.  Second, we question whether treating the position of the symbol \texttt{[CLS]} the same as other words is a reasonable design, considering its special role (the representation of the entire sentence) in the downstream tasks. Motivated from above analysis, we propose a new positional encoding method called \textbf{T}ransformer with \textbf{U}ntied \textbf{P}ositional \textbf{E}ncoding (TUPE). In the self-attention module, TUPE computes the word contextual correlation and positional correlation separately with different parameterizations and then adds them together. This design removes the mixed and noisy correlations over heterogeneous embeddings and offers more expressiveness by using different projection matrices. Furthermore, TUPE unties the \texttt{[CLS]} symbol from other positions, making it easier to capture information from all positions. Extensive experiments and ablation studies on GLUE benchmark demonstrate the effectiveness of the proposed method. Codes and models are released at \url{https://github.com/guolinke/TUPE}.

Published as a conference paper at ICLR 2021
RETHINKING POSITIONAL ENCODING IN LANGUAGE PRE-TRAINING
Guolin Ke, Di He & Tie-Yan Liu
Microsoft Research"
(guolin. gn,elin.ke, dihe, tyliu) eeeieeiiccooo..oo
ABSTRACT
In this work, we investigate the positional encoding methods used in language pre- training (e.g.. BERT) and identify several problems the existing formulations. First, we show that in the absolute positional encoding, the addition operation applied on positional embeddings and word embeddings brings mixed correlations between the two heterogeneous information resources. It may bring unnecessary randomness in the attention and further limit the expressiveness of the model. Sec- ond, question whether treating the position of the symbol [CLS] ] the same other words is a reasonable design, considering its special role (the representation of the entire sentence) in the downstream tasks. Motivated from above analysis, we propose a new positional encoding method called Transformer with Untied Positional Encoding (TUPE). In the self-attention module, TUPE computes the word contextual correlation and positional correlation separately with different parameterizations and then adds them together. This design removes the mixed and noisy correlations over heterogencous embeddings and offers more expres- siveness by using different projection matrices. Furthermore, TUPE unties the [CLS] symbol from other positions, making it easier to capture information from all positions. Extensive experiments and ablation studies on GLUE benchmark demonstrate the effectiveness of the proposed method. Codes and models are released at https: / /ttgtgttttt.co/// com/
INTRODUCTION
The Transformer model (Vaswani al., 2017) is the most widely used architecture in language representation learning (Liu et al., 2019; Devlin et al., 2018; Radford et al., 2019; Bao et al., 2020). In Transformer, positional encoding is an essential component since other main components of the model are entirely invariant to sequence order. The original Transformer uses the absolute positional encoding, which provides each position an embedding vector. The positional embedding added to the word embedding. which is found significantly helpful learning the contextual representations of words at different positions. Besides using the absolute positional encoding, Shaw et al. (2018); Raffel et al. (2019) further propose the relative positional encoding, which incorporates some carefully designed bias term inside the self-attention module to encode the distance between any two positions.
In this work, we revisit and study the formulation of the widely used absolute/relative positional encoding. First, we question the rationality of adding the word embedding with the absolute positional embedding in the input layer. Since the two kinds of embeddings addition operation brings mixed correlations' between the positional information and word semantics. For example, by expanding the dot-production function of keys and values in the self-attention module of the first layer, we find that there are explicit terms that use "word" to query "positions" and vice versa. However, words may only have weak correlations to where they appear in the sentence. Our empirical analysis also supports this by showing that in a well-trained model, such correlation is noisy.
apparently heterogeneous, this
Second, we notice that the BERT model does not only handle natural language words. A special symbol [CLS] is usually attached to the sentence. is widely acknowledged that this symbol
"Correspondence to: gguolin,kk, dihe @microsoft.com
'The term "correlation" mainly refers to the dot product hetween Key and Query the self-attention module.

---

Published as a conference paper at ICLR 2021
Relative positional correlation
Add & Layer Norm
bi
b1
02
FFN
FFN
b-1
Bg
62
b2
b-3
Dv
Add & Layer Norm
b-s
h.
bu
Self-Attention
bj-1
Untie [CLS]
Add & Layer Norm
f
FFN
... FFN
Add & Layer Norm
UQ
U
Self -ttle-tttenion
W/
Pt
Pj
Word embedding input directly
Absolute positional correlation
Figure 1: The architecture of TUPE. The positional correlation and word correlation are computed separately, and added together in the self-attention module. The positional attention related to the [CLS] token is treated more positionless, to encourage it captures the global information.
receives and summarizes useful information from all the positions, and the contextual representation of [CLS] will be used as the representation of the sentence in the downstream tasks. As the role of the [CLS] symbol is different from regular words that naturally contain semanties, argue that it will be ineffective if we treat its position the same as word positions in the sentence. For example, if we apply the relative positional encoding this symbol, the attention distribution of some heads will likely be biased to the first several words, which hurts the understanding of the whole sentence.
Based on the investigation above, we propose several simple, yet effective modifications to the current methods, which lead to a new positional encoding called Transformer with Untied Positional Encoding (TUPE) for language pre-training, see Figure In TUPE, the Transformer only uses the word embedding as input. In the self-attention module, different types of correlations are separately computed to reflect different aspects of information, including word contextual correlation and absolute (and relative) positional correlation. Each kind of correlation has its own parameters and will be added together to generate the attention distribution. specialized positional correlation is further set to the [CLS] symbol, aiming to capture the global representation of the sentence correctly. First, we can see that in TUPE, the positional correlation and word contextual correlation are de-coupled and computed using different parameters. This design successfully removes the randomness in word-to-position (or position-to-word, correlations and gives more expressiveness to characterize the relationship between a pair of words positions. Second, TUPE uses a different function compute the correlations between the [CLS] symbol and other positions. This flexibility can help the model learn an accurate representation the whole sentence.
We provide an efficient implementation of TUPE. To validate the method, we conduct extensive experiments and ablation studies on the GLUE benchmark dataset. Empirical results confirm that our proposed TUPE consistently improves the model performance on almost all tasks. In particular, we observe that by imposing this inductive bias to encode the positional information, the model can be trained more effectively, and the training time of the pre-training stage can be largely improved.
2 PRELIMINARY
2.1 ATTENTION MODULE
The attention module (Vaswani et al., 2017) formulated as querying dictionary with key-value pairs, e.g.. Attention(Q2 K,V) softmax( QK"
Vd )V, where d is the dimensionality of the hidden
2

---

Published as a conference paper at ICLR 2021
representations, and Q (Query), K (Key), V (Value) are specified as the hidden representations of the previous layer. The multi-hean head variant of the attention module is popularly used which allows the model jointly attend the information from different representation sub-spaces, and is defined as
Multi-head (Q, K, V) Concat(heedd, head W
head; = Attention (QW? KWW VW
where W E Rdxdx W E Rdxdx W E Rdxdy, and w E Rdd are learnable project matrices, is the number heads. dx and dy are the dimensionalities of Key and Value.
The self-attention module is one of the key components in Transformer and BERT encoder (Devlin et al., 2018). For simplicity, we use the single- head self-attention module and set dx = dy = d for a demonstration. We denote x' = (œ z2 ... ,za) ,,, as the input to the self-attention module in the I-th layer, where n is the length of the sequence and each vector o E Rd is the contextual representation of the token at position z' (=1, 2/2 :.. znn) is the output of the attention module. Then the self-attention module can be written as
exp(ass) (at) ( where aij = Jaa (
(2)
exp(aay)
As we can see, the self-attento module does not make use of the order of the sequence, i.e., is permutation-innaaian However, natural language is well-structured and word order important for language understanding (Sutskever et al., 2014). In the next section, we show several previous works that proposed different ways of incorporating positional information into the Transformer model.
2.2 POSITIONAL. ENCODING
Generally, there two categories of methods that encode positional information in the Transformer model, absolute positional encoding and relative positional encoding.
Absolute Positional Encoding. The original Transformer (Vaswani et al., 2017) proposes to absolute positional encoding to represent positions. In particular, a (learnable) real-valued vector P; E RRd is assigned each position i. Given sentence, P; will be added to the word embedding w; at position i, and wi + pj will be used as the input to the model, e.g. x² w. + Pr. In such a way, the Transformer differentiate the words coming from different positions and assign each token attention. For example, in the self-attention module in the first layer, we have
Abe
1 Va ((u; p.)) + K.))
Relative Positional Encoding, In absolute positional encoding, using different pt for different position helps Transformer distinguish words different positions. However, pointed out in Shaw et al. (2018), the absolute positional encoding is not effective for the model to capture the relative word orders. Therefore, besides using absolute positional encoding, Shaw al. proposes relative positional encoding as an inductive bias to help the learning of attention modules,
Rel 1 0i
(4)
Vd
where a E Rd is learnable parameter and can be viewed as the embedding of the relative position j - in layer this way, embedding df explicitly models the relative word orders. T5 (Raffel et al., 2019) further simplifies it by eliminating of_ in Query-Ker product.
T5 1
+ by-
(5)
Vd
For each j -, by_ is a learnable scalar2 and shared in all layers.
?Specifically, in Shaw et al. (2018); Raffel al. (2019), the relative position - will be first clipped to pre-defined range, e.g., clip(j - t), = 128. The embedding defined over the possible values of the clipped range, i.c., [-t, t). Besides, Shaw et al. also tried to add vector a,-; to the value V in the output of self-attention, but the experiment results indicate that did not improve much.
V.l
3

---

Published as a conference paper at ICLR 2021
2
1
-1
-2
Figure 2: Visualizations of the four correlations (Eq. (6)) on pre-trained BERT model for sampled batch of sentences. From left right: word-to-word, word-to-position, position-to-wort and position-to-position correlation matrices. In each matrix, the (i-th, j-th) element is the correlation between i-th word/position and j-th word/position. We can find that the correlations between word and position are not strong since the values in the second and third matrices look uniform.
3 TRANSFORMER WITH UNTIED POSITIONAL ENCODING
3.1 UNTIE THE CORRELATIONS BETWERN POSITIONS AND WORDS
In absolute positional encoding, the positional embedding is added together with the word embedding to serves as the input to the neural networks. However, these two kinds of information are hetero- geneous. The word embedding encodes the semantic meanings of words and word analogy tasks can be solved using simple linear arithmetic on word embeddings (Mikolov et al., 2013; Pennington et al., 2014; Joulin et al., 2016). On the other hand, the absolute positional embedding encodes the indices in sequence, which is not semantic and far different from word meanings. We question the rationality of the linear arithmetic between the word embedding and the positional embedding. To check clearly, we take look the expansion of Eq. (3).
((w, P.) Q.1 (w) + K..)WK..JT
Abs
Vd
(w:Wa.)) (w, WK.JJT Vd
(p, Vd
(w, Ko, Vd
(6)
vd
The above expansion shows how the word embedding and the positional embedding are projected and queried in the attention module. We see that there are four terms after the expansion: word-to-word, word-to-position, position-to-word, and tosition-to-position correlations.
We have several concerns regarding this formulation. First, it is easy see that the first and the last term characterize the word-word and position-position relationships respectively. However, the projection matrices WQ and WK.! are shared in both terms. As the positional embedding and the word embedding encode significantly different concepts. it is not reasonable to apply the same projection such different information.
Furthermore, we also notice that the second and the third term use the position (word) as the query to get keys composed of words (positions). As far as we know, there is little evidence suggesting that the word and its location in sentence have a strong correlation. Furthermore, in BERT and recently developed advanced methods such as RoBERTa (Liu et al., 2010), sentences are patched in random way. For example, in BERT, each input contains multiple sentences and some of the sentences are negatively sampled from other documents to form the next sentence prediction task. Due to the random process of batching, it is possible that word the correlations between words and positions could be weak. To further investigate this, we visualize the four correlations in Eq. (6) pre-trained BERT model. We find that the second and the third term looks uniform across positions, as shown in Figure 2. This phenomenon suggests that there are no strong correlations between the word and the absolute position and using such noisy correlation may be inefficient for model training.
appear any positions and
'Some recent works (Yang et al., 2019; He et al., 2020) show that correlations between relative positions and words can improve the performance. Our results have contradiction with theirs study is the correlations between word embeddings and absolute positional embeddings.
4

---

Published as a conference paper at ICLR 2021
W
WA
W
W Wj
U
UK
W; Wi
P:
Pj
Pj
(a) Absolute positional encoding.
(b) Untied absolute positional encoding.
Figure 3: Instead of adding the absolute positional embedding to the word embedding in the input (left), compute the positional correlation and word correlation separately with different projection matrices, and add them together in the self-attention module (right).
Our modification. To overcome these problems, we propose to directly model the relationships between pair of words or positions by using different projection matrices and remove the two terms in the middle. That is, we use
VaAl ( (o
(n.0%)
Oaj
where Uº2 Uº U Idddd are the projection matrice for the positional embedding, and scaling term is used retain the magnitude of a, (Vaswani al., 2017). A visualization is put in Figure Our proposed method can be well combined with the relative positional encoding in Raffel et al. (2019) by simply changing Eq. (5) to
Y2d
1 (z/ (a (at WKiY? + (0, + bbg-i. v2d
Ozj²
3.2 UNTIE THE [CLS] SYMBOL FROM POSITIONS
Note that in language representation learning, the input sequence to the Transformer model is not always a natural sentence. In BERT, special symbol [CLS] ] is attached to the beginning of the input sentence. This symbol designed capture the global information of the whole sentence. Its contextual representation will be used to make predictions in the sentence-level downstream tasks after pre-training (Devlin et al., 2018; Liu et al., 2019).
We argue that there will be some disadvantages if we treat this token the same other natural words the attention module. For example, the regular words usually have strong local dependencies in the sentence. Many visualizations (Clark et al., 2019a; Gong et al., 2019) show that the attention distributions of some heads concentrate locally. If we process the position [CLS] the same as the position of natural language words, according to the aforementioned local concentration phenomenon, [CLS] will be likely biased to focus on the first several words instead the whole sentence. will potentially hurt the performance of the downstream tasks.
Our modification. We give a specific design in the attention module to untie the [CLS] symbol from other positions. In particular, we reset the positional correlations related [CLS). For better demonstration, we denote Vij as the content-free (position-only) correlation between position i and j. For example, when using the absolute positional encoding in Eq. (7), oij Jaa (p.UU)))pp (Ky
when using relative positional encoding in Eq. (8), Ui J/2 (p, + bj-s. We reset the values of Uig by the following equation:
Vij
+ 1.j # (not related to [CLS])
(v,
0
= 1. (from [CLS] to others)
(9)
02
i 1.j = (from others to [CLS])
where 0 (01,02] 02) is learnable parameter. A visualization is put in Figure 4.

---

Published as a conference paper at ICLR 2021
F11
P12
P13
P14
02
0
01
82
1/21
122
P'z3
1/24
Untie [CLS]
0
1/2
V/3
1.44
133
1232
P33
P/1
8/2
1/2
1/3
P34
141
P42
P43
1/4
V42
143
P44
Figure 4: Illustration of untying [CLS] Ui denotes the positional correlation of pair (4,5). The first row and first column are set to the same values respectively.
3.3 IMPLEMENTATION DETAILS AND DISCUSSIONS
In the two subsections above, we propose several modifications to untie the correlations between positions and words (Eq. (7) and Eq. (8)). and untie the [CLS] symbol from other positions (Eq. (9)). By combining them, we obtain a new positional encoding method and call it TUPE (Transformer with Untied Positional Encoding). There are two versions of TUPE. The first version is to the untied absolute positional encoding with the untied [CLS] symbol (Eq. (7) + Eq. (9)), and the second version use an additional relative positional encoding (Eq. (8) Eq. (9)). We call them TUPE-A and TUPE-R respectively and list the mathematical formulations as below.
TUPE-A ij
Vea (at WK.JT + reseto V (n:U9) (PJ (I,))
1
(10)
TUPE-R (i)
(od + reseto( Aza (v:09) + +,),,,;)), V2d
(11)
The multi-head version, parameter sharing, and efficiency. TUPE can be easily extended to the multi-head version. In our implementation, the absolute positional embedding p; for position i is shared across different heads, while for each head, the projection matrices U and U are different. For the relative positional encoding, by-s is different for different heads. The reset parameter 0 also shared across heads. For efficiency, we share the (multi-head) projection matrices UºQ and UK in different layers. Therefore, in TUPE, the number of total parameters does not change much. Taking BERT-Base as an example, we introduce about 1.18M (2 768 x 768) new parameters, which only about 1 % of the 10M parameters in BERT-Base Besides, TUPE introduces little additional computational costs. As the positional correlation term ( (p, is shared in all layers, we only need to compute in the first layer, and reuse its outputs in other layers.
Are absolute/relative positional encoding redundant to each other? One may think that both the absolute/relative positional encoding in Eq. (11) describe the content-free correlation, and thus one of them is redundant. To formally study this, we denote B as an element Br, = by-t- By definition, B is a Toeplitz matrix (Gray, 2006). We also denote P as an n matrix where the i-th row is pı, and thus the absolute positional correlation in matrix form is (PU We study the expressiveness of B and Vzl (PUR) by first showing B can be factorized similarly from the following proposition.
matrix where each
Proposition 1. Any Toeplitz matrix B E C'xn can be factorized into B = GDG", where D is 2n x 2n diagonal matrix. G a n x 2n Vandermonde matrix in the complex space, where each (=+++1) and G is the conjugate transpose of G.
element Gj.ba 2n
The proof can be found in Appendix A. The two terms B and 1 (PUR)(PUK) actually form V2d
different subspaces in Raxn, In the multi-head version, the shape of the matrix Uº and 1 are x # Therefore. (PU)) can characterize low-rank matrices Raxn But from the proposition, we can see that forms a linear subspace in Raxn with only 2n freedoms, which is quite different from the space of 1 ((PU))(PUK)) There are also some practical reasons which make using both terms together essential. As discussed previously, in Raffel al. (2019), the range of the relative distance j i will be clipped up to an offist beyond which all relative positions will be assigned the same value. In such a situation, the relative positional encoding may not be able to differentiate words faraway and Z (p, can be used to encode complementary information.

---

Published as a conference paper at ICLR 2021
Table I: GLUE scores dev set. All settings are pre-trained by BERT-Base (110M) model with 16GB data. TUPE-A (TUPE-R is the intermediate 3t0 checkpoint of TUPE-A (TUPE-R). TUPE-A ti tie-ds removes the reset function from TUPE-A. BERT-A" uses different projection matrices for words and positions, based on BERT-A.
Steps
MNLI-n/mm
QNLI
QQP
SST
CoLA
MRPC
RTE
STS
Avg.
BERT-A
IM
84.93/84.91
91.34
91.04
92.88
55.19
88.29
68.61
89.43
82.96
BERT-R
IM
85.81/85.84
92.16
91.12
92.90
55,43
89.26
71.46
88.94
83.66
TUPE-A
1M
86.05/85.99
91.92
91.16
93.19
63.09
88.37
71.61
88.88
84.47
TUPE-R
1M
86.21/86.19
92.17
91.30
93.26
63.56
89.89
73.56
89.23
85.04
TUPE-A
300k
84.76/84.83
90.96
91.00
92.25
62.13
87.1
68.79
88.16
83.33
TUPE-R and
300k
84.86/85.21
91.23
91.14
92.41
62.47
87.29
69.85
88.63
83.68
TUPE-A"cGR
IM
85.91/85.73
91.90
91.05
93.17
59,46
88.53
69.54
88.97
83.81
BERT-A"
1M
85.26/85.28
91.56
91.02
92.70
59.73
88.46
71.31
87.47
83.64
EXPERIMENT
To verify the performance of the proposed TUPE, we conduct extensive experiments and demonstrate the results in this section. the main body the paper, we study TUPE under the BERT-Base setting (Devlin et al., 2018). We provide all the experimental details and more results on applying TUPE under the BERT-Large setting and the ELECTRA setting (Clark al., 2019b) in Appendix and
4.1 EXPERIMENTAL. DESIGN
We use BERT-Base (110M parameters) architecture for all experiments. Specifically, BERT-Bass consist of 12 Transformer layers. For each layer, the hidden size is set to 768 and the number of attention head is set to 12. To compare with TUPE-A and TUPE-R, we set up two baselines correspondingly: BERT-A, which is the standard BERT-Bas Base with absolute positional encoding (Devlin et al., 2018); BERT-R, which uses both absolute positional encoding and relative positional encoding (Raffel et al., 2019) (Eq. (5)).
Following Devlin et al. (2018), we use the English Wikipedia corpus and BookCorpus (Zhu et al., 2015) for pre-training. By concatenating these datasets, obtain corpus with roughly 16GB in size. We set the vocabulary size (sub-word tokens) as 32,768. We use the GLUE (General Language Understanding Evaluation) dataset (Wang et al., 2018) as the downstream tasks to evaluate the performance of the pre-trained models. All codes are implemented based on fairseq (Ott et al., 2019) in PyTorch (Paszke et al., 2017). All models are run on 16 NVIDIA Tesla V100 GPUs with mixed-preciside (Micikevicius et al., 2017).
4.2
OVERALL COMPARISON
32
BERT-A
16
30
BERT-R
38
TUPE A
10
2.8
63
2.6
22
2.4
10
BERT-A
91
BERT-A
BERT-R
BERT-R
2.2
TUPE-A
90
TUPE-A
TUPEIR
2.0
79
200
400
600
800
1000
0
200
400
600
800
1000
200
400
600
800
1000
Pre-Train Steps (k)
Pre: Trai Steps (k)
Pre-Train Steps (K)
(a) Validation loss in pre-training.
(b) MNLI-m score.
(c) GLUE average score.
Figure 5: Both TUPE-A and TUPE-R converge much faster than the baselines, and achieve better performance in downstream tasks while using much fewer pre-training steps.
7

---

Published as a conference paper at ICLR 2021
The overall comparison results shown in Table 1. Firstly, it is easy to find that both TUPE-A and TUPE-R outperform baselines significantly. In particular, TUPE-R outperforms the best baseline BERT-R by 1.38 points in terms of GLUE average score and is consistently better on almost all tasks, especially on MNLI-m/mm, CoLAA and MRPC. We can also see that TUPE-R outperforms TUPE-A by 0.57 points. As discussed in Sec.3.3, although using the absolute/relative positional encoding together seems to be redundant, they capture complement information to each other.
Besides the final performance, also examine the efficiency of different methods. As shown in Figure 5a, TUPE-A (TUPE-R) achieves smaller validation loss than the baselines during pre-training. As shown in Table and Figure 5c, TUPE-A (TUPE-R) can even achieve a better GLUE average score than the baselines while only using 30% pre-training steps. Similar improvements can be found LERT- and ELECTRA settings Table. 3 and Table. (in Appendix C).
Since the correlations between words and positions are removed in TUPE, we can easily visualize the attention patterns over positions, without considering the variability of input sentences. In TUPE-A (see Figure 6), we find there are mainly five patterns (from 12 heads): (1) attending globally; (2) attending locally; (3) attending broadly; (4) attending to the previous positions: (5) attending to the next positions. Interestingly, the model can automatically extract these patterns from random initialization. As there are some attention patterns indeed have strong local dependencies, our proposed method to untie [CLS] is necessary. Similar patterns could be found in TUPE-R as well.
To summarize, the comparisons show the effectiveness and efficiency of the proposed TUPE. As the only difference between TUPE and baselines is the positional encoding, these results indicate TUPE can better utilize the positional information in sequence. In the following subsection, we will examine each modification in TUPE check whether it useful.
4.3 ABLATION STUDY
Untie the [CLS] symbol from other positions. To study the improvement brought by untying [CLS], 1 we evaluate positional encoding method which removes the reset function in Eq. 10. We call it TUPE-A A de-ds and train this model using the same configuration. We also list the performance that TUPE-A works consistently better than TUPE-A tieds especially for low-resource tasks, such as CoLA and RTE.
of TUPE-A A fie-el in Table 1. From the table,
Untie the correlations between positions and words. Firstly, from Table 1, it is easy to find that TUPE-A outperforms BERT-A. Since the only difference between TUPE-A get
and BERT-A is the way of dealing with the absolution positional encoding, conclusion that untying the correlations between positions and words helps the model training. To further investigate this, we design another encoding method, BERT-A4, which is based on BERT-A and uses different projection matrices for words and positions. Formally, aij = in BERT-A4 Therefore, we can check whether using different projection matrices (BERT-A vs. )) can improve the model and whether removing word-to-position and position-to-word correlations (BRET-A4 vs. TUPE-A iieck)) hurts the final performance. From the summarized results in Table 1, find that TUPE-Aties tiecs is even slightly better (0.17) than BERT-A and is more computationally efficient". BERT-A is the worst one. These results indicate that using different projection matrices improves the model, and removing correlations between words and positions does not affect the performance.
I Vid (Z) + (x + (a, + (p,
Summary. From the above analysis, we find that untying [CLS] helps great deal for the low- resource tasks, such as CoLA and RTE. Untying the positional correlation and word correlation helps high- tasks, like MNL.-m/-mm. By combining them, TUPE can consistently perform better on all GLUE tasks. We also have several failed attempts regarding modifying the positional encoding strategies, see Appendix D.
'BERT-A4 is inefficient compared to TUPE as it needs to additionally compute middle two correlation terms in all layers.
8

---

Published as a conference paper at ICLR 2021
3.2
2.4
1.6
0.8
0.0
-0.8
Figure 6: The visualization of learned positional correlations by TUPE-A.
RELATED WORK
As introduced in Sec.2, Shaw et al. (2018) was the first work leverage relative positional encoding to Transformer. Most of the other works are based on Shaw et al. (2018). For example, Transformer-X (Dai et al., 2019) re- re-parameterize the self-attention to integrate relative positional encoding directly. T5 (Raffel al., 2019) simplified the vector representation of relative positions in Shaw al. (2018) to scalar. He et al. (2020) extended Shaw et al. (2018) by adding the position-to-work correlation for relative position. In (Kitaev & Klein, 2018), the authors show that separating positional and content information in the Transformer encoder can lead to an improved constituency parser. We mathematically show that such disentanglement also improves Transformer in general language pre-training. There are some other parallel works to enhance the absolute positional encoding in Transformer, but not directly related to our work. For example, Shiv & Quirk (2019) extended the sequence positional encoding to tree- tree-based positional encoding in Transformer; Wang et al. (2019) extended positional encoding to complex complex-valued domain; Liu et (2020) modeled the positional encoding by dynamical systems.
6 CONCLUSION
We propose TUPE (Transformer with Untied Positional Encoding), which improves existing methods by two folds: untying the correlations between words and positions, and untying [CLS] from sequence positions. Specifically, we first remove the absolute positional encoding from the input of the Transformer and compute the positional correlation and word correlation separately with different projection matrices in the self-attention module. Then, we untie [CLS] by resetting the positional correlations related [CLS]. Extensive experiments demonstrate that TUPE achieves much better performance on GLUE benchmark. Furthermore, with better inductive bias over the positional information, TUPE can even outperform the baselines while only using 30% pre-training computational costs.
REFERENCES
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.0645 06450, 2016.
Hangbo Bao, Li Dong, Furu Wei, Wenhui Wang, Nan Yang, Xiaodong Liu, Yu Wang, Songhao Piao, Jianfeng Gao, Ming Zhou, et al. Unilmv2: Pseudd-masked language models for unified language model pre-training. arXiv preprint arXiv:2002. /2804, 2020.
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning. What does bert look at? an analysis of bert's attention. ACL 2019, pp. 276, 2019a.
Kevin Clark, Minh-Thang Luong, Quoe V Le, and Christopher Manning. Electra: Pre-training text encoders as discriminators rather than generators. In International Conference on Learning Representations, 2019b.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoe Le, and Ruslan Salakhutdi- Transforrme-xx:: Attentive language models beyond fixed-length context. arXiv preprint arXiv:1901.02860. 2019.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: /0/0. 04.005 2018.

---

Published as a conference paper at ICLR 2021
Linyuan Gong, Di He, Zhuohan Li, Tao Qin, Liwei Wang, and Tieyan Liu. Efficient training of bert by progressively stacking. In International Conference on Machine Learning, pp. 2337-2346, 2019.
Robert M Gray. Toeplitz and circulant matrices: review. now publishers inc, 2006.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhanee bert with disentangled attention. arXiv preprint arXiv:2006.036544 2020.
Armand Joulin, Edouard Grave, Piotr Bojanowski, Maithijs Douze, Hérve Jégou, and Tomas Mikolov. Fasttext, zip: Compressing text classification models. arXiv preprint arXiv: 1612.0331, 2016.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, (trgg/bbs14440 6980.
abs/1412.69880 2014. URL
Nikita Kitaev and Dan Klein. Constituency parsing with self-attentive encoder. arXiv preprint arXiv:1805.01052, 2018.
Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. Moses: Open source toolkit for statistical machine translation. ACL, 2007.
Jun Liu, Lei Yuan, and Jieping Ye. An efficient algorithm for a class fused lasso problems. In Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 233-332.222 ACM, 2010.
Xuanqing Liu, Hsiang-Fu Yu, Inderjit Dhillon, and Cho-Jui Hsieh. Learning to encode position for transformer with continuous dynamical model. arXiv preprint arXiv:2003.09229 2020.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: /907. / 1692, 2019.
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al. Mixed precision training. arXiv preprint arXiv:1710.0331 2017.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. In Advances neural information processing systems, pp. 3111-3111 19,,,,11 2013.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. arXiv preprint arXiv:1904.01038, 2019.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary Vito,o, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In Proceedings of the 20/4 conference on empirical methods in natural language processing (EMNLP), pp. 1532-1543, 2014.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl Blog, 1(8), 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv: 10683, 2019.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. CoRR, abs/ 0009999 2015. URL http: /arxivv.. ctg/abs/////eeeccccc....caaaaeae /1508. 07909.
10

---

Published as a conference paper at ICLR 2021
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representations. arXiv preprint arXiv:1803.021:: 2018.
Vighnesh Shiv and Chris Quirk. Novel positional encodings to enable tree-based transformers, In Advances in Neural Information Processing Systems, pp. 12058-12068, 2019.
llya Sutskever, Oriol Vinyals, and Quoc Le. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pp. 3104-3112, 2014.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, pp. 6000-600,, 2017.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. CoRR, abs/1804. 07461, 2018. URL http: //tpivv/axivv////aa org/ abs/ 1804. 07461.
Benyou Wang, Donghao Zhao, Christina Lioma, Qiuchi Li, Peng Zhang, and Jakob Grue Simonsen. Encoding word order in complex embeddings. arXiv preprint (tt.:/9/2..333.. 2019.
Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tie- Yan Liu. On layer normalization in the transformer architecture. arXiv preprint arXhv:2002.04445, 2020.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint MeXiv:/906.0820 2019.
Yukun Zhu, Ryan Kiros, Richard Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. arxiv preprint arXiv: 06724, 2015.

---

Published as a conference paper at ICLR 2021
A PROOF OF PROPOSITION 1
Proof. We first construct circulant matrix B of shape 2n x 2n using B. Note that B is a Toeplitz matrix consisting of 2n 1 values {b-(n-1), (n--))) ... bo, .(n-)), ((1-1) For ease of
reference, we further define b_n = bn = by- Then B is constructed as
bk-s
-n S j
ByA
< k j 2n bk-j-2n
(12)
bk-j+2n
-2n < k: j
To avoid any confusion, we use kt and] j in the subscription for positions and use i as the imaginary unit. It is easy to check that / is a circulant matrix by showing that B₁ B5+1 and 2
By+1. for any 0 S j. k < 2n 1. Using Theorem 7 in Gray (2006), the matrix B can be factorized
into B- QDQ". Q is an 2n x 2n Vandermonde matrix, where Qk.j 2n
cdx(i+1)k// and D
is 2n x 2n diagonal matrix. Since the top-left n x n submatrix of B is B, we can rewrite
G1
G
B
where is defined in the theorem. Then we have We also rewrite Q = Ca
C2
C3
G
Di
0 |G Ci]. Considering that D is a diagonal matrix, we can obtain D2
QDQ- =
C4
B GDG using block matrix multiplication.
B EXPERIMENTAL DETAILS
Pre-training. Following BERT (Devlin et al., 2018), we use the English Wikipedia corpus and BookCorpus (Zhu al., 2015) for pre-training. By concatenating these two datasets, we obtain a corpus with roughly 16GB in size. We follow a couple of consecutive pre-processing steps: segmenting documents into sentences by Spacy', normalizing, lower-casing, and tokenizing the texts by Moses decoder (Koehn et al., 2007), and finally, applying byte pair encoding (BPE) (Sennrich al., 2015) with setting the vocabulary size as 32,768.
We found the data cleaning is quite important for language pre-training, Specially, we de-duplicate the documents, normalize the punctuations, concatenate the short sequences, replace the URL and other hyperlinks to special tokens, and filter the low-frequency tokens. Therefore, our re-implemented baselines, like BERT, and achieve higher GLUE scores than the original papers.
We use masked language modeling as the objective of pre-training. We remove the next sentence prediction task and use FULL-SENTENCES mode to pack sentences as suggested in RoBERTa (Liu et al., 2019). We train the models for 1000k steps where the batch size is 256 and the maximum sequence length is 512. The masked probability is set to 0.15, with replacing 80% of the masked positions by [MASK], 10% by randomly sampled words, and keep the remaining 10% unchanged. We use Adam (Kingma& & Ba, 2014) as the optimizer, and set the its hyperparameter e to 1e-6 and (81, (22 to (0.9, 0.999). The peak learning rate is set to 1e-4 with 10k-step warm-up stage. After the warm-up stage, the learning rate decays linearly to zero. We set the dropout probability to 0.1, gradient clip norm 1.0, and weight decay 0.01. Besides the final checkpoint, also save intermediate checkpoints (( 100k; 300k, 600k } steps) and fine-tune them on downstream tasks, to check the efficiency of different methods.
Fine-tuning. We use the GLUE (General Language Understanding Evaluation) dataset (Wang et al., 2018) as the downstream tasks to evaluate the performance of the pre-trained models. Particularly, we use nine tasks in GLUE, including CoLA, RTE, MRPC, STS, SST, QNLI, QQP, and MNLI-m/mm For the evaluation metries, we report Matthews correlation for CoLA, Pearson correlation for STS-B, and accuracy for other tasks. We use the same optimizer (Adam) with the same hyperparameters as in pre-training- Following previous works, we search the learning rates during the fine-tuning for each downstream task. The setting details are listed in Table 2. For fair comparison, we do not apply any tricks for fine-tuning. Each configuration will be run five times with different random seeds, and the median of these five results on the development set will be used as the performance of one configuration. We will ultimately report the best number over all configurations.
httpp:: httpss://spacy.ooo 1o
12

---

Published as a conference paper at ICLR 2021
Table Hyperparameters for the pre-training and tine-nuning.
Pre-training
Fine-tuning
Max Steps
1M
Max Epochs
or 10
Learning Rate
le-4
(2e-5, 3e-5, 4e-5, 5e-5)
Batch Size
256
32
Warm-up Ratio
0.01
0.06
Sequence Length
512
512
Learning Rate Decay
Linear
Linear
Adam
le-6
le-6
Adam (319 (32)
(0.9, 0.999)
(0.9, 0.999)
Clip Norm
1.0
1.0
Dropout
0.1
0.1
Weight Decay
0.01
0.01
we use five for the top four high-resource tasks, MNLI-m/-mm, QQP, and QNLI,
the fine-tuning costs. Ten is used for other tasks.
Table 3: GLUE scores on dev set. Different models are pre-trained in the BERT-Large setting (330M)
with 16GB data. Targ-L is the intermediate 300k-step checkpoint of TUPE-Large.
Steps MNLI-m/mm
QNLI QQP
SST
CoLA
MRPC
RTE
STS
Avg.
BERT-Large
IM 88.21/88.18
93.56
91.66
94.08
58.42
90.46
77.63
90.15
85.82
TUPE-Lard
1M
88.22288.21
93.55
91.69
94.98
67.46
90.06
81.66
90.67
87.39
TUPE-Larger
300k
86.92/86.80
92.61
91.48
93.97
63.88
89.26
75.82
89.34
85.56
Normalization and rescaling. Layer normalization (Ba et al., 2016; Xiong et al., 2020) is key component in Transformer. In TUPE, we also apply layer normalization on p; whenever is used. The Va in Eq. (2) is used in the Transformer rescale the dot product outputs into a standard range.
In similarly way, use 1 in the Eq. (7) to both terms to keep the scale after the summation. Furthermore, in order to directly obtain similar scales for every term, we parameterize 01 and 0g by using 8 Vea (Po, (po, )* and 02 Vza UPo, where pa, Po, E IRd are learablee vectors.
V2d
C MORE RESULTS IN THE BERT-LARGE/ELECTRA -BASE SETTINGS
In the main body of the paper, we provide empirical studies TUPE under the BERT-Ban Base setting. Since TUPE only modifies the positional encoding, one expects it to improve all language pre-training methods and deeper models. To demonstrate this, integrate TUPE-R (with relative position) in BERT-LLage and ELECTRA-Bas models, and conduct experiments for comparison. For the BERT-Large setting, we directly apply TUPE-R to the 24-layer Transformer model and obtain the TUPE-Large model. For ELECTRA-Baas we apply our positional encoding to both generator and discriminator and obtain the ELECTRA-TUPE model. We use the same data introduced previously and use the same hyper-parameters as in Liu et al. (2019); Clark et al. (2019b). The experimental results are listed in the Table. 3 and Table. 4 respectively. From the results, we can that TUPE is better than the corresponding baseline methods. These additional experiments further demonstrate that TUPE is better positional encoding solution.
D FAILED ATTEMPTS
We tried to replace the parametric form of the positional correlation ( (2,9)), to the non-parametric form. However, empirically we found that the training of this setting con- verges much slower than the baselines. We also tried to parameterize relative position bias by- by (r) But the improvement is just marginal.
13

---

Published as a conference paper at ICLR 2021
Table 4: GLUE scores on dev set. Different models are pre-trained in the ELECTRA-Base setting (120M) with 16GB data, Electra-TUPE is the intermediate 600k-step checkpoint of ELECTRA- TUPE.
Steps MNLI-m/mm QNLI QQP SST CoLA
MRPC
RTE
STS
Avg.
ELECTRA
TM 86.42/86.
92.19 91.45 92.97 65.92
89.74
73.56
89.99 85.38
ELECTRA-TUR
1M
86.98/87.01
92.03 91.75 93.97 66.51
89.74
76.53
90.13
86.07
ELECTRA-TUPE"T!
600k
86.72/86.74
91.84 91.72 93.30
66.42
89.34
75.64
89.67 85.71
14


Reviews:
[{"review_id": "nQpSoO0xnZf", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: The paper delves into the nature of positional encoding in Transformer and variants (especially BERT). The paper points out that, attention weight computation obtained from the addition of the word embedding and the position embedding in the first layer can be expanded into four terms, namely word-word, position-position, and two word-position (in both directions). The paper points out that word-position relationship is largely meaningless, which is also demonstrated via visualizing the values in pertained BERT. Then the paper proposes to remove these two terms when computing the attention. Secondly, the paper mentions that special tokens in BERT (e.g. [CLS]) should be treated in a different way than typical words. To do so, it proposes to replace the attention logic with a learnable parameter if the attention involves one or two special tokens. Both methods combined show a significant advantage on multiple GLUE tasks when applied to BERT.\n\nStrengths:\n- The observations on the issues of the PE in BERT (Transformer) are interesting and convincing.\n- The proposed method based on the observations result in significant experimental benefit in multiple classification datasets in GLUE.\n\nWeaknesses:\n- I am wondering if all experiments in Table 1 were done with exactly the same setup including random seed. This is because BERT was known for being undertrained and some slight changes with respect to the original BERT could result in non-trivial gains.\n- It would be good to see how the proposed method works in other more recent pertained models including RoBERTa.\n\nOverall: I think the paper nicely touches on positional encoding, an important component of Transformer-based architectures, and the experimental results are encouraging. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "mE57A4Szbz", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the positional encoding in BERT. It argues against the word—position correlations that are implicitly imposed by BERT’s treatment of positional encodings. The paper proposes to decouple the “content attention” (token to token) and “contentless attention” (position to position), and remove the so called heterogeneous interactions between tokens and positions. Further, it points out that it is problematic to treat the special [CLS] symbol as a token in calculating its positional encoding: per standard practice, [CLS]’s representation is used as a summary of the full sequence, and treating it as a token implicitly biases the sentence representation towards those tokens close [CLS]. To resolve this issue, the paper “hides” the position information of [CLS]. The proposed approach is built on top of BERT and evaluated on  the GLUE benchmark. Experimental results show that it outperforms the baselines.\n\nOverall I find this paper interesting and clearly-presented. The experiments are solid and well-executed too. My main concern is that the intuition that “a word does not have strong correlations to its position,” which is one of the key motivations, can be better elaborated (see details below).\n\nPro:\n- The proposed solutions to the issues pointed out by the paper are clean and reasonable.\n- Untying [CLS]’s positional encoding is insightful.\n- Strong performance.\n- Writing is very clear.\n\nCon:\n- The key motivation that a word has little to do with where it appears is debatable.\n- It’s hard to figure out whether or not the improvement could come from using absolute positional encodings at every layer.\n\nDetails:\n- Upfront the paper argues that “without specific context there is little evidence that a word has a strong correlation to where it appears in the sentence,” and cites Köhler et al., (2008) for support. First of all, I find this claim, to the best of my knowledge, counterintuitive and misleading. I am not able to verify this claim from the cited work: it seems to be a book about linguistics and is written in German, which I do not have any expertise. Since this is the key motivation of this paper, I encourage the authors to elaborate it: (1) Please explain what it means by “without specific context.” Clearly in “contextualized” pretraining every word appears in context. (2) Clarify what “correlation” refers to from a linguistic perspective. Footnote 1 indicates that it refers to dot product between vectors, which does not seem to fit in the linguistic argument the paper is trying to make. (3) Verify that this phenomenon, if true at all, applies to all languages or specify the languages it applies. (4) If at all possible, please additionally cite a supporting work written in English so that it is more accessible to a broader audience, especially to those without German expertise like myself.\n- The paper empirically supports the above claim using a visualization experiment in Figure 5. However, I find it hard to interpret without knowing the experimental details. Does this apply to all layers? Is it consistent across all attention heads? Do the authors think this is due to the learning biases of BERT and transformers, or it is an inherent property of the data and the English language?\n- Again about this claim: I don’t think it is well-supported in the experiments either: comparing BERT-A^d against TUPE-A^{tie-cls}, less than 0.15 average improvement can be attributed to removing word-position correlation, and the performance is mixed across the datasets. \n- Adding onto the above three points: this claim serves as a motivation instead of a key research question. So I don’t think the paper’s contribution will be weaker if it tones down this claim or does not talk about it at all.\n- BERT uses the positional encoding at the first layer and propagate it all the way up with the residual connections. In contrast, some works use relative positional encodings at every layer. Equation 7 suggests that TUPE uses absolute positional encodings at every layer. Can the authors comment on how this compares to (1) a TUPE model that uses positional encoding only at the first layer (so that it is more comparable to BERT) and (b) a BERT model that uses absolute positional encodings at every layer?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "gLlh5t5OCyK", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces an approach for the positional encoding of input tokens into pretrained contextual embedding models. Specifically, they compute the positional correlation and the embedding separately, then add them together. In addition, they treat the [CLS] symbol separately from the other positions, which is a long-overdue improvement. \n\nThis paper's motivation is clear, they derive intuitive improvements for the positional encoding of inputs to BERT. Writing out the absolute and relative positional encodings was useful -- they motivate the untying by showing that the absolute positional encoding is actually composed of four terms in equation 6, two of which are not encoding useful information. They then introduce in equation 9 their modification to the positional encoding of the CLS symbol. Again, this is fairly straightforward, and likely leaves room for improvement but is a reasonable first step. \n\nTheir experiments support their hypotheses that this builds a better representation than the standard positional encoding schemes. They present results on the GLUE dev set, and while for some of the smaller datasets in GLUE results can be sometimes unstable (even between runs changing only the random seed), the performance improvements are so consistent that there is clear evidence in favor of the hypothesis that this change is an improvement. \n\nIn general, I don't see any clear faults, and I expect this work will be built on quickly or fairly widely adopted. It's novel, clear, and as far as I'm aware original, and even though it's a relatively simple idea, it's well supported by the experimental evidence. It's possible adding other LMs to the experiments would add to the strength of the argument, but to me this is enough.\n\nEdit: After reviewing the author response, I leave my score unchanged. I believe this paper should be accepted. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "DipCD8Mav5p", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper takes a stab at fixing the correlation problem between the positional encoding and word embedding when they are added together in a transformer. The paper fixes this by untying the position encoding from the word embedding and passing them through separate routes of projection key and value matrices before merging them. The paper also tackles the problem of untying [CLS] token from rest of the position tokens due to its significance in downstream tasks. Authors claim that [CLS] token if not properly tied is susceptible to only gathering information from local starting words in the sentence. The authors fix this by resetting the positional correlations related to CLS from others and to others from [CLS]. The authors provide nice theoretical intuitions behind their claims which I find justified.\n\nThe proposed method is called TUPE short for Transformers with Untied Positional Encoding. TUPE outperforms the BERT baseline easily on GLUE tasks and shows empirically that this was quite necessary and semantically word embeddings and position embedding were indeed not similar. I recommend this paper for acceptance as this is an important advancement in Transformer architecture which is becoming a de-facto standard in NLP and elsewhere. \n\nQuestions to authors:\n- It would be great if qualitative analysis similar to Figure 4. Can be shown for original correlation in positional embeddings and word-embeddings to qualitatively support the intuitions. Similarly, for the original [CLS] token.\n\n- Original BERT model was evaluated on SQuAD and SWAG as well. Do these claims hold there as well?\n\nEdit after rebuttal: I have read the author response and I am thankful to authors for answering my questions. I keep my rating as it is and believe that the paper should be accepted. I hope this will be widely adopted in NLP community.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper revisits the design of positional embedding in the pre-trained language models, and propose a new approach to handling the positional encoding. 

Overall, the paper is well-motivated. The authors have addressed most comments based on the review. The method proposed in the paper is simple and effective. Experiments are comprehensive and demonstrate the effectiveness of the proposed approaches. 

Decision: Accept (Poster)