Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposed a novel attention mechanism and a new objective function that mitigates the distribution shifts caused by masked tokens for downstream tasks in MLM. It demonstrates superior performance across benchmarks.

Pros:
1. Good empirical results are demonstrated across an extensive suite of benchmarks. Ablation studies are well done. Hence I am willing to give a score of 6 despite of the following concerns.

Cons:
1. My major concern is about the novelty of this paper. 
In transformer-XL[1], the idea of relative positional information in the form of Eq (2) was already introduced. The paper somehow intentionally omit the discussion following (2), only mentioning two earlier works of (Shaw et al., 2018; Huang et al., 2018). I think the author should be honest and compare with relative positional information introduced in transformer-XL in the forefront. 
That being said, there is obviously still differences between transformer-XL and the proposed methods. And also the introduction of novel objectives in addition to the attention mechanism. 

2. However, the previous concern brought up the second concern I have about the evaluations. Since the modification relative positional information of transformer-XL to the proposed method is not too large, I wonder if there is a reason to explain the better performances of the proposed methods. Hence I am worried if the baseline such as XLNet was well-tuned. We can see that for example in [2], the performance of XLNet was much better than originally reported. I think the author should try to carefully evaluate the relative positional mechanisms of prior works with authors' own infrastructure, while having everything else fixed.

3. I find the word "disentangled" a bit misleading in this context. Disentanglement in ML [3] often refers to the ability to disentangle factors of variations of the data. The work does not make use of any disentangled techniques, or have disentanglement representation/architectures. It simply use a relative position mechanism that's the sum of four matrix products. 

[1] Dai et. al. Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context

[2] Dai et. al. Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing

[3] Locatello et. al., Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations
review_title: Good empirical performance but requires a more careful comparisons to prior works.

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary and Contributions

The authors proposed an extension to the word representation transformer architecture  that takes into account disentangle features for position and content. The disentangle of attention is based on the composition of a content and position parameter matrices, in addition with combinations of both.  The main contribution is to tackle issues with the relative position embeddings used on standard transformer architectures. The proposed model shows improvements on some benchmarks by using less pre-training data compared to the baseline.

Strengths

- The proposed model tackles a known issue in transformer architectures.
- The authors perform a comprehensive comparison on standard text benchmarks as well as an ablation study.
- The findings show that disentangle attention improves results on some text benchmarks.

Weaknesses

- Related work on disentangle representations for text, and the further motivation for using disentanglement into the attention model are not discussed.
- Missing results of the variance in metrics with multiple runs on the downstream tasks. As an extra contribution, the authors could  show if the improvements are due to the proposed model or variance in parameter initialisation. 

Questions to the Authors

- Could you elaborate on disentangled representations and how they relate to the proposed attention model? 
- How does it compare the enhanced masked language model with the masked language model?
- How does the relative position parameter matrix is initialised, and how does it affect the language model performance? 

review_title: DeBERTa: Decoding-Enhanced BERT with Disentangle Attention

final_score: rating: 7: Good paper, accept
review_content: In this paper, an improvement of BERT model is proposed. It relies on the disentanglement of contents and relative positions in the encoding layers and on the incorporation of absolute positions in the decoding layer.


Strengths:
* The paper is well written, the positioning to the state of the art is clear and the method is rigorously described. 
* The paper provides a complete evaluation using the existing benchmarks for NLP and including ablation studies and evaluation of pre-training efficiency and Deberta improves results in the major part of the cases.


Weaknesses:
* The proposed method is a relative increment of previous methods.
* In Section 4.1.1., the way performance increase or decrease is reported is not exact (1.1% -> 1.1 points)
* Do we have an idea of the statistical significance of the improvements?
* It would be interesting to have the rationale for the mitigated result obtained on Table 1. Is Deberta more relevant for specific tasks?
* The authors claim that they evaluate their results on generation task but it rather seems that they evaluate language modeling using perplexity. 
*The use of non documented acronyms (ppl, for example) that could be not understandable outside the NLP community.
*They are some redundancy in the text (second paragraph of 3.2 and fourth paragraph of the introduction) that is not necessary.


review_title: In this paper, an improvement of BERT model is proposed. It relies on the disentanglement of contents and relative positions in the encoding layers and on the incorporation of absolute positions in the decoding layer.

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposes a BERT-inspired model that adds a two main different architectural decisions: different content and position representations (instead of a sum), and absolute positions in the decoding layer. The authors run the standard suite of GLUE benchmark experiments, on both “large” and “base” setups, as well as a generation setup (Wikitext-103). 

The modifications proposed are not game-changing, but the evaluations are interesting in terms of understanding the impact of these modifications. One thing that I find disingenuous is fact that their disentangled approach does introduce additional parameters, which is not quantified (or even mentioned) in the main paper. I had to dig into the Appendix to see that this introduces about 49M additional parameters (increment of 13%).

Another problem that I have is with their experimental comparisons, especially the ones in main part, Sec 4.1.1. I’m listing below the most important issues in this section:

“RoBERTa and XLNet are trained for 500K steps with 8K samples in a step, which amounts to four billion passes over training samples”. This is confusing; what you mean to say is that the models see about four billion training examples. The term “passes” is used usually as an equivalent to “epochs”, ie how many times the model goes over the entire training set.

“[...] Table 1, which compares DeBERTa with previous models with around 350M parameters: BERT, RoBERTa, XLNet, ALBERT and ELECTRA.” Note that ALBERT is actually around 235M parameters, significantly less than all the others. You cannot simply bundle all together and claim they are equivalent parameter-size--wise.

“DeBERTa still outperforms them [ALBERT_xxlarge] in term  of the average “GLUE” score.” Note that the difference here wrt ALBERT_xxlarge is from 89.96 to 90.00, ie 0.04 for the average, with a tie 3-3 in terms of wins for specific tasks. Unless you can show that the 0.04 difference is statistically significant, you need to tone down the claim about “outperforming”.


“We summarize the results in Table 2. Compared to the previous SOTA models with similar sizes, including BERT, RoBERTa, XLNet and Megatron336M, DeBERTa consistently outperforms them in all the 7 tasks. Taking RACE as an example, DeBERTa is significantly better than previous SOTA XLNet with an improvement of 1.4% (86.8% vs. 85.4%).”
For whatever reason, the authors omit ALBERT from the comparison done for Table 2, in spite of its even smaller size compared to the included ones, and the fact that the ALBERT numbers for these tasks are readily available in the paper. Taking RACE as an example: ALBERT (single model) has 86.5% accuracy, therefore nullifying the claim of 1.4% improvement.


Re: References

A lot of the references use the Arxiv version for papers that have been peer-reviewed and published. Please fix.

review_title: DEBERTA: DECODING-ENHANCED BERT WITH DISENTANGLED ATTENTION

Meta-review: All reviewers gave, though not very strong,  positive scores for this work.  Although the technical contribution of the paper is somewhat incremental, the reviewers agree that it solidly addresses the known important issues in BERT, and the experiments are extensive enough to demonstrate the empirical effectiveness of the method.  The main concerns raised by the reviewers are regarding the novelty and the discussion with respect to related work as well as some unclear writings in the detail,  but I think the pros outweigh the cons and thus would like to recommend acceptance of the paper.

We do encourage authors to properly take in the reviewers' comments to further polish the paper in the final version.


Decision: Accept (Poster)