Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The paper takes a stab at fixing the correlation problem between the positional encoding and word embedding when they are added together in a transformer. The paper fixes this by untying the position encoding from the word embedding and passing them through separate routes of projection key and value matrices before merging them. The paper also tackles the problem of untying [CLS] token from rest of the position tokens due to its significance in downstream tasks. Authors claim that [CLS] token if not properly tied is susceptible to only gathering information from local starting words in the sentence. The authors fix this by resetting the positional correlations related to CLS from others and to others from [CLS]. The authors provide nice theoretical intuitions behind their claims which I find justified.

The proposed method is called TUPE short for Transformers with Untied Positional Encoding. TUPE outperforms the BERT baseline easily on GLUE tasks and shows empirically that this was quite necessary and semantically word embeddings and position embedding were indeed not similar. I recommend this paper for acceptance as this is an important advancement in Transformer architecture which is becoming a de-facto standard in NLP and elsewhere. 

Questions to authors:
- It would be great if qualitative analysis similar to Figure 4. Can be shown for original correlation in positional embeddings and word-embeddings to qualitatively support the intuitions. Similarly, for the original [CLS] token.

- Original BERT model was evaluated on SQuAD and SWAG as well. Do these claims hold there as well?

Edit after rebuttal: I have read the author response and I am thankful to authors for answering my questions. I keep my rating as it is and believe that the paper should be accepted. I hope this will be widely adopted in NLP community.

review_title: Advancement to Transformer Architecture

final_score: rating: 7: Good paper, accept
review_content: This paper introduces an approach for the positional encoding of input tokens into pretrained contextual embedding models. Specifically, they compute the positional correlation and the embedding separately, then add them together. In addition, they treat the [CLS] symbol separately from the other positions, which is a long-overdue improvement. 

This paper's motivation is clear, they derive intuitive improvements for the positional encoding of inputs to BERT. Writing out the absolute and relative positional encodings was useful -- they motivate the untying by showing that the absolute positional encoding is actually composed of four terms in equation 6, two of which are not encoding useful information. They then introduce in equation 9 their modification to the positional encoding of the CLS symbol. Again, this is fairly straightforward, and likely leaves room for improvement but is a reasonable first step. 

Their experiments support their hypotheses that this builds a better representation than the standard positional encoding schemes. They present results on the GLUE dev set, and while for some of the smaller datasets in GLUE results can be sometimes unstable (even between runs changing only the random seed), the performance improvements are so consistent that there is clear evidence in favor of the hypothesis that this change is an improvement. 

In general, I don't see any clear faults, and I expect this work will be built on quickly or fairly widely adopted. It's novel, clear, and as far as I'm aware original, and even though it's a relatively simple idea, it's well supported by the experimental evidence. It's possible adding other LMs to the experiments would add to the strength of the argument, but to me this is enough.

Edit: After reviewing the author response, I leave my score unchanged. I believe this paper should be accepted. 
review_title: An improvement to positional encoding for contextual language models

final_score: rating: 7: Good paper, accept
review_content: Summary: The paper delves into the nature of positional encoding in Transformer and variants (especially BERT). The paper points out that, attention weight computation obtained from the addition of the word embedding and the position embedding in the first layer can be expanded into four terms, namely word-word, position-position, and two word-position (in both directions). The paper points out that word-position relationship is largely meaningless, which is also demonstrated via visualizing the values in pertained BERT. Then the paper proposes to remove these two terms when computing the attention. Secondly, the paper mentions that special tokens in BERT (e.g. [CLS]) should be treated in a different way than typical words. To do so, it proposes to replace the attention logic with a learnable parameter if the attention involves one or two special tokens. Both methods combined show a significant advantage on multiple GLUE tasks when applied to BERT.

Strengths:
- The observations on the issues of the PE in BERT (Transformer) are interesting and convincing.
- The proposed method based on the observations result in significant experimental benefit in multiple classification datasets in GLUE.

Weaknesses:
- I am wondering if all experiments in Table 1 were done with exactly the same setup including random seed. This is because BERT was known for being undertrained and some slight changes with respect to the original BERT could result in non-trivial gains.
- It would be good to see how the proposed method works in other more recent pertained models including RoBERTa.

Overall: I think the paper nicely touches on positional encoding, an important component of Transformer-based architectures, and the experimental results are encouraging. 

review_title: Nice analysis and improvement on Transformer's position encoding 

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper studies the positional encoding in BERT. It argues against the word—position correlations that are implicitly imposed by BERT’s treatment of positional encodings. The paper proposes to decouple the “content attention” (token to token) and “contentless attention” (position to position), and remove the so called heterogeneous interactions between tokens and positions. Further, it points out that it is problematic to treat the special [CLS] symbol as a token in calculating its positional encoding: per standard practice, [CLS]’s representation is used as a summary of the full sequence, and treating it as a token implicitly biases the sentence representation towards those tokens close [CLS]. To resolve this issue, the paper “hides” the position information of [CLS]. The proposed approach is built on top of BERT and evaluated on  the GLUE benchmark. Experimental results show that it outperforms the baselines.

Overall I find this paper interesting and clearly-presented. The experiments are solid and well-executed too. My main concern is that the intuition that “a word does not have strong correlations to its position,” which is one of the key motivations, can be better elaborated (see details below).

Pro:
- The proposed solutions to the issues pointed out by the paper are clean and reasonable.
- Untying [CLS]’s positional encoding is insightful.
- Strong performance.
- Writing is very clear.

Con:
- The key motivation that a word has little to do with where it appears is debatable.
- It’s hard to figure out whether or not the improvement could come from using absolute positional encodings at every layer.

Details:
- Upfront the paper argues that “without specific context there is little evidence that a word has a strong correlation to where it appears in the sentence,” and cites Köhler et al., (2008) for support. First of all, I find this claim, to the best of my knowledge, counterintuitive and misleading. I am not able to verify this claim from the cited work: it seems to be a book about linguistics and is written in German, which I do not have any expertise. Since this is the key motivation of this paper, I encourage the authors to elaborate it: (1) Please explain what it means by “without specific context.” Clearly in “contextualized” pretraining every word appears in context. (2) Clarify what “correlation” refers to from a linguistic perspective. Footnote 1 indicates that it refers to dot product between vectors, which does not seem to fit in the linguistic argument the paper is trying to make. (3) Verify that this phenomenon, if true at all, applies to all languages or specify the languages it applies. (4) If at all possible, please additionally cite a supporting work written in English so that it is more accessible to a broader audience, especially to those without German expertise like myself.
- The paper empirically supports the above claim using a visualization experiment in Figure 5. However, I find it hard to interpret without knowing the experimental details. Does this apply to all layers? Is it consistent across all attention heads? Do the authors think this is due to the learning biases of BERT and transformers, or it is an inherent property of the data and the English language?
- Again about this claim: I don’t think it is well-supported in the experiments either: comparing BERT-A^d against TUPE-A^{tie-cls}, less than 0.15 average improvement can be attributed to removing word-position correlation, and the performance is mixed across the datasets. 
- Adding onto the above three points: this claim serves as a motivation instead of a key research question. So I don’t think the paper’s contribution will be weaker if it tones down this claim or does not talk about it at all.
- BERT uses the positional encoding at the first layer and propagate it all the way up with the residual connections. In contrast, some works use relative positional encodings at every layer. Equation 7 suggests that TUPE uses absolute positional encodings at every layer. Can the authors comment on how this compares to (1) a TUPE model that uses positional encoding only at the first layer (so that it is more comparable to BERT) and (b) a BERT model that uses absolute positional encodings at every layer?

review_title: Review

Meta-review: This paper revisits the design of positional embedding in the pre-trained language models, and propose a new approach to handling the positional encoding. 

Overall, the paper is well-motivated. The authors have addressed most comments based on the review. The method proposed in the paper is simple and effective. Experiments are comprehensive and demonstrate the effectiveness of the proposed approaches. 

Decision: Accept (Poster)