DEBERTA: DECODING-ENHANCED BERT WITH DISENTANGLED ATTENTION
Authors: Pengcheng He
Xiaodong Liu
Jianfeng Gao
Weizhu Chen
Venue: ICLR 2021 Poster (2021)
Keywords: Transformer
Attention
Natural Language Processing
Language Model Pre-training
Position Encoding

Abstract: Recent progress in pre-trained neural language models has significantly improved the performance of many natural language processing (NLP) tasks. In this paper we propose a new model architecture DeBERTa (Decoding-enhanced BERT with disentangled attention) that improves the BERT and RoBERTa models using two novel techniques. The first is the disentangled attention mechanism, where each word is represented using two vectors that encode its content and position, respectively, and the attention weights among words are computed using disentangled matrices on their contents and relative positions, respectively. Second, an enhanced mask decoder is used to incorporate absolute positions in the decoding layer to predict the masked tokens in model pre-training. In addition, a new virtual adversarial training method is used for fine-tuning to improve models’ 
 generalization. We show that these techniques significantly improve the efficiency of model pre-training and the performance of both natural language understand(NLU) and natural langauge generation (NLG) downstream tasks. Compared to RoBERTa-Large, a DeBERTa model trained on half of the training data performs consistently better on a wide range of NLP tasks, achieving improvements on MNLI by +0.9% (90.2% vs. 91.1%), on SQuAD v2.0 by +2.3% (88.4% vs. 90.7%) and RACE by +3.6% (83.2% vs. 86.8%). Notably, we scale up DeBERTa by training a larger version that consists of 48 Transform layers with 1.5 billion parameters. The significant performance boost makes the single DeBERTa model surpass the human performance on the SuperGLUE benchmark (Wang et al., 2019a) for the first time in terms of macro-average score (89.9 versus 89.8), and the ensemble DeBERTa model sits atop the SuperGLUE leaderboard as of January 6, 2021, outperforming the human baseline by a decent margin (90.3 versus
89.8). The pre-trained DeBERTa models and the source code were released at: https://github.com/microsoft/DeBERTa.


Published as a conference paper at ICLR 2021
DEBERTA: DECODING-ENHANCER BERT WITH DIS- ENTANGLED ATTENTION
Pengcheng He', Xiaodong Liu", Jianfeng Gao?, Weizhu Chen'
1 Microsoft Dynamics 365 Al 2 Microsoft Research
(penhe, xiaodl, jfgao, wzchen @microsoft. com
ABSTRACT
Recent progress in pre-trained neural language models has significantly improved the performance of many natural language processing (NLP) tasks. In this pa- per we propose new model architecture DeBERTa (Decoding-enhanced BERT with disentangled attention) that improves the BERT and RoBERTa models using two novel techniques. The first is the disentangled attention mechanism, where each word is represented using two vectors that encode its content and position, respectively, and the attention weights among words are computed using disen- tangled matrices on their contents and relative positions, respectively, Second, an enhanced mask decoder is used to incorporate absolute positions in the de- coding layer to predict the masked tokens in model pre-training, In addition, new virtual adversarial training method used for fine-tuning to improve models' generalization. We show that these techniques significantly improve the efficiency model pre-training and the performance of both natural language understand (NLU) and natural langauge generation (NLG) downstream tasks. Compared to RoBERTa-LLage. a DeBERTa model trained on half of the training data per- forms consistently better on a wide range of NLP tasks, achieving improvements on MNLI by +0.9% (90.2% vs. 91.1%), on SQuAD v2.0 by +2.3% (88.4% vs. 90.7%) and RACE by +3.6% (83.2% vs. 86.8%) Notably, we scale up DeBERTa by training larger version that consists of 48 Transform layers with 1.5 bil- lion parameters. The significant performance boost makes the single DeBERTa model surpass the human performance on the SuperGLUE benchmark (Wang al., 2019a) for the first time in terms of macro-average score (89.9 versus 89.8), and the ensemble DeBERTa model sits atop the SuperGLUE leaderboard as of Jan- uary 6, 2021, outperforming the human baseline by a decent margin (90.3 versus 89.8). The pre-trained DeBERTa models and the source code were released https: / com/ mi crosoft, /DeBRTT
INTRODUCTION
The Transformer has become the most effective neural network architecture for neural language modeling. Unlike recurrent neural networks (RNNs) that process text in sequence, Transformers apply self-attention to compute in parallel every word from the input text an attention weight that gauges the influence each word has on another, thus allowing for much more parallelization than RNNs for large-scale model training /Vaswani et al. 2017). Since 2018 8, we have seen the rise of set of large-scale Transformer-batt based Pre-trained Language Models (PLMs), such as GPT (Radford eet al. 2019 Brown al. 2020), BERT (Devlin al. 2019). RoBERTa (Liu et al. 2019c). XeN Yang et al. 2019), UniLM (Dong et al. 2019), ELECTRA (Clark et al. 2020). T5 (Raffel et al. 2020), StructBERT (Wang al. 2019c) and ERINE (Sun al. 2019) These PLMs have been fine-tuned using task-specific labels and created new state of the art in many
2020), ALOM (Liu et al.
downstream natural language processing (NLP) tasks (Liu et al. 2019b Minace al. 2020 Tiang
et al. 2020 He et al. 2019a Shen et al. 2020
!Our code and models also available HuggingFace Transformers: https::/github.co com/ musgiingfacc//rransaaaa http:::/hugiingffcce.oom or

---

Published as a conference paper at ICLR 2021
In this paper, propose new Transformer-based neural language model DeBERTa (Decoding- enhanced BERT with disentangled attention), which improves previous state-of state-of-the-ar ahe- PLMs using two novel techniques: disentangled attention mechanism, and an enhanced mask decoder.
Disentangled attention. Unlike BERT where each word in the input layer is represented using a vector which is the sum of its word (content) embedding and position embedding, each word in DeBERTa is represented using two vectors that encode its content and pasition, respectively, and the attention weights among words are computed using disentangled matrices based on their contents and relative positions, respectively. This is motivated by the observation that the attention weight of word pair depends on not only their contents but their relative positions. For example, the dependency between the words "deep" and "learning" is much stronger when they occur next to each other than when they occur in different sentences.
Enhanced mask decoder. Like BERT, DeBERTa is pre-trained using masked language modeling (MLM). MLM is fill-i-tthe-blank task. where model is taught to use the words surrounding a mask token to predict what the masked word should be. DeBERTa uses the content and position information of the context words for MLM. The disentangled attention mechanism already considers the contents and relative positions of the context words, but not the absolute positions of these words, which in many cases are crucial for the prediction. Consider the sentence "a new store opened beside the new mall" with the italicized words "store" and "mall" masked for prediction. Although the local contexts of the two words are similar, they play different syntactic roles in the sentence. (Here, the subject of the sentence is "store" not "mall," for example.) These syntactical nuances depend, to a large degree, upon the words' absolute positions in the sentence, and so it is important to account for a word's absolute position in the language modeling process. DeBERTa incorporates absolute word position embeddings right before the softmax layer where the model decodes the masked words based on the aggregated contextual embeddings of word contents and positions.
In addition, we propose new virtual adversarial training method for fine-tuning PLMs to downstream NLP tasks. The method is effective in improving models' generalization.
We show through comprehensive empirical study that these techniques substantially improve the efficiency of pre-training and the performance of downstream tasks. In the NLU tasks, compared to RoBERTa-Large, a DeBERTa model trained on half the training data performs consistently better on a wide range of NLP tasks, achieving improvements on MNLI by +0.9% (90.2% vs. 91.1%) 1)..) on SQuAD v2.0 by (88.% vs. 90.7%) and RACE by +3.6% (83.2% vs. 86.8%). In the NLG tasks, DeBERTa reduces the perplexity from 21.6 to 19.5 on the Wikitext- 103 dataset. We further scale up DeBERTa by pre-training larger model that consists of 48 Transformer layers with 1.5 billion parameters. The single 1 .arr-errr DeBERTa model substantially outperforms T5 with 11 billion parameters on the SuperGLUE benchmark [Wang et al. 2019a) by 0.6%(89.39) vs. 89.9%), and surpasses the human baseline (89.9 vs. 89.8) for the first time. The ensemble DeBERTa model sits atop the SuperGLUE leaderboard as of January 6, 2021, outperforming the human baseline by decent margin (90.3 versus 89.8).
BACKGROUND
2.1 TRANSFORMER
A Transformer-based language model is composed of stacked Transformer blocks (Vaswani et al. 2017). Each block contains a multi-head self-attention layer followed by a fully connected positional feed-forward network. The standard self-attention mechanism lacks natural way to encode word position information. Thus, existing approaches add positional bias to each input word embedding so that each input word is represented by a vector whose value depends on its content and position. The positional bias can be implemented using absolute position embedding (Vaswani et al. 2017 7 Radford et al. 2019 Devlin et al. 2019) relative position embedding (Huang et al. 20188] Yang ct al. 2019). It has been shown that relative position representations are more effective for natural language understanding and generation tasks (Dai et al. 2219 Shaw al. 2018). The proposed disentangled attention mechanism differs from all existing approaches in that we represent each input word using two separate vectors that encode word's content and position, respectively, and
2

---

Published as a conference paper at ICLR 2021
attention weights among words computed using disentangled matrices their contents and relative positions, respectively.
2.2 MASKED LANGUAGE MODEL
Large-scale Transformer-basee PLMs typically pre-trained large amounts of text learn contextual word representations using a self-supervision objective, known as Masked Language Model (MLM) (Devlin et al., 2019). Specifically, given sequence X {zz). corrupt into X by masking 15% of its tokens at random and then train a language model parameterized by 0 to reconstruct X by predicting the masked tokens & conditioned on X:
max log po X X) max Σ log po = 2 X) 0
(1)
#
where C is the index set of the masked tokens in the sequence. The authors of BERT propose keep 10% of the masked tokens unchanged, another 10% replaced with randomly picked tokens and the rest replaced with the [MASK] token.
3 THE DEBERTA ARCHITECTURE
3.1 DISENTANGLED ATTENTION: A TWO-VECTOR APPROACI TO CONTENT AND POSITION EMBEDDING
For token at position in sequence, we represent it using two vectors, {H;} and (Paitt which represent its content and relative position with the token at position j, respectively. The calculation of the cross attention score between tokens and j can be decomposed into four components
Arj (H, Pag) x P
H. + H.P + HJ Pay + Jis
That is, the attention weight of a word pair can be computed as a sum of four attention scores using disentangled matrices on their contents and positions as comtent-to-contenl, content-to-position. position-to-comlen, and voxition-to-position
Existing approaches to relative position encoding relative position bias in computing attention weights (Shaw et al.] 2018 Huang et al.) 2018). This equivalent to computing the attention weights using only the conteni-to-content and content-to- position terms in equation? 2 We argue that the position-to-come
separate embedding matrix compute the
term is also important since the attention weight of word pair depends not only on their contents but on their relative positions, which can only be fully modeled using both the content-to-post Since we use relative position embedding,
and position-to co-ceettnt terms. position-to-positiee term does not provide much additional information and is removed from equation 2 in our implementation.
Taking single-head attention as an example, the standard self-attention operation (Vaswani al. 2017) can be formulated as:
Q HW K HW V HW. A QK
Vd
Ho softmax(A)V
where H Nxd represents the input hidden vectors, H. Rxd the output of self-attention, Wg. We, W. Ráxa the projection matrices, E NNN X the attention matrix, the length of the input sequence, and the dimension of hidden states.
Denote as the maximum relative distance, 5(i,j) [0, 2k) as the relative distance from token to token j. which is defined as:
0 for
i- < k i-
8(i i-
2k
1 for
others.
II this sense, our model shares some similarity to Tensor Product Representation (Smolensky 1990 Schlag
al. 2019 Chen et al. (219)) where word is represented using tensor product of its filler (content) vector and role (position) vector.

---

Published as a conference paper at ICLR 2021
We represent the disentangled self-attention with relative position bias equation 4 where Qc. Ke and Ve are the projected content vectors generated using projection matrices W 9.cr WA. k.c. W., Rdxd respectively. P R2axd represents the relative position embedding vectors shared across all layers (i.e., staying fixed during forward propagation), and Qr and K, projected relative position vectors generated using projection matrices Wa Wg,r; Wk, krr Raxd respectively.
Q. = HW q,e, Ke = H,k,e Vo=
HWo.e: Qr
PW a., K, = PWer
Ar, = QiKg
QiKoo +
K K5
(4)
(a) content-to co-ceeeet
(b) content-oo-posiiee
(c) position-to-content
À VV
Ho = softmax(
v3d
Aig is the element of attention matrix Á, representing the attention score from token i to token Of the i-th row of Qc Kÿ is the j-th row of Ke. KT ((.5)) the jii row of K,, with regarding to relative distance (i. (agja) is the 8(j, t)-t row of Qr with regarding to relative distance 8(j, i). Note that we use 5(j, i) rather than 5(i,j) here. This is because for given position i. position-to-conter
computes the attention weight of the key content at j with respect to the query position at i, thus the relative distance is ò(j. The position-to-content term is calculated as K))) T. The content-to-p contenttto-position posotoooioonn term is calculated in a similar way.
Finally, we apply a scaling factor v3d on Å. The factor is important for stabilizing model training (Vaswani et al. 2017), especially for large-scale PLMs.
Algorithm Disentangled Attention
Input: Hidden state H, relative distance embedding P, relative distance matrix Content projec-
tion matrix Wk.c. W q.e. Wr,es position projection matrix Wk.r, W q.
Wg,e.
K. HWkee. k. Qc HW q,e- Ve HW vee- K, PW k. Qr PW q.
2: Ac-c QcKI
3: for iori=0 0,
do Ac-pli, -i, = Qcli,
4:
5: end for
6: for 0 O.... N do
7: for j
..... N do
8: Ac-pli [1, A Ac-p peppl 12, ,i
9: end for
end for
11: for j
= O..., N do
12: A Ap-sel:n [ = QT
end for
14: for j 0,.., N - do
for i 0 .., 1 do
Ap-eli 51 Ap-e i1.b
17: end for
end for
19:
A A + Ac-p + Ap
20: H. = softmax
Ve
v3d
Output: H.
3.1.1 EFFICIENT IMPLEMENTATION
For input sequence of length N. it requires space complexity of O(N2d) (Shaw et al. 2018 Huang et al. 2018: Dai et al. 2019) to store the relative position embedding for each token. However, taking critent-to-positionn
example, note that since 5(i,j) [0, 2k) and the embeddings
4

---

in the Published as a conference paper at ICLR 2021
of all possible relative positions are always subset of K, E R2kxd then reuse K, attention calculation for all the queries.
In our experiments, set the maximum relative distance & to 512 for pre-training. The disentangled attention weights can be computed efficiently using Algorithm [ Let ò be the relative position matrix according to equation 3 i.e., o[ j(..j). Instead of allocating a different relative position embedding matrix for each query, we multiply each query vector Qo[i. :] by KT E Rdx2k, as line 3 5. Then, we extract the attention weight using the relative position matrix
the index, in line 6 10. To compute the position-to-connent attention score, calculate Ap-el--j.. i.e., the column vector of the attention matrix Ap- by multiplying each key vector KoLi. :] by QT, as in line 11 13. Finally,
extract the corresponding attention score via the relative position matrix as the index, as in line 14 18. In this way, we do not need to allocate memory to store a relative position embedding for each query and thus reduce the space complexity to (kkd) (for storing K, and Qr)
3.2 ENHANCED MASK DECODER ACCOUNTS FOR ABSOLUTE WORD POSITIONS
DeBERTa is pretrained using MLM, where model is trained to the words surrounding mask token to predict what the masked word should be. DeBERTa uses the content and position information of the context words for MLM. The disentangled attention mechanism already considers the contents and relative positions of the context words, but not the absolute positions of these words, which in many cases are crucial for the prediction.
Given a sentence "a new store opened beside the new mall" with the words "store" and "mall" masked for prediction. Using only the local context (e.g., relative positions and surrounding words) is insufficient for the model to distinguish store and mall in this sentence, since both follow the word new with the same relative positions. To address this limitation, the model needs take into account absolute positions, as complement information to the relative positions. For example, the subject of the sentence is "store" not "mall". These syntactical nuances depend, large degree, upon the words' absolute positions in the sentence.
There are two methods of incorporating absolute positions. The BERT model incorporates absolute positions in the input layer. In DeBERTa, we incorporate them right after all the Transformer layers before the sofimax layer for masked token prediction, as shown in Figure In this way, DeBERTa captures the relative positions in all the Transformer layers and only uses absolute positions as complementary information when decoding the masked words. Thus, we call DeBERTa's 's decoding component an Enhanced Mask Decoder (EMD). In the empirical study, we compare these two methods of incorporating absolute positions and observe that EMD works much better. We conjecture that the early incorporation of absolute positions used by BERT might undesirably hamper the model from learning sufficient information of relative positions. In addition, EMD also enables us to introduce other useful information, in addition to positions, for pre-training. We leave future work.
SCALE INVARIANT FINE-TUNING
This section presents a new virtual adversarial training algorithm, Scale-invariant-Fnnee -inn--unine--nnege (SiFT), a variant to the algorithm described in Miyato et al. (2018); [Ting et al. (2020), for fine-tuning.
Virtual adversarial training is regularization method for improving models' generalization. It does so by improving model's robustness to adversarial examples, which are created by making small perturbations the input. The model is regularized so that when given task-specific example, the model produces the same output distribution as it produces on an adversarial perturbation of that example.
For NLP tasks, the perturbation is applied to the word embedding instead of the original word sequence. However, the value ranges (norms) of the embedding vectors vary among different words and models. The variance gets larger for bigger models with billions of parameters, leading to some instability of adversarial training.
5

---

Published as a conference paper at ICLR 2021
Inspired by layer normalization (Ba et al. 2016), propose the SiFT algorithm that improves the training stability by applying the perturbations to the normalized word embeddings. Specifically, when fine-tuning DeBERTa to downstream NLP task in experiments, SiFT first normalizes the word embedding vectors into stochastic vectors, and then applies the perturbation to the normalized embedding vectors. We find that the normalization substantially improves the performance of the fine-tuned models. The improvement is more prominent for larger DeBERTa models. Note that we only apply SiFT to DeBERTa, L.B on SuperGLUE tasks in our experiments and will provide more comprehensive study of SiFT in our future work.
EXPERIMENT
This section reports DeBERTa results on various NLU tasks.
5.1
MAIN RESULTS ON NLU TASKS
Following previous studies of PLMs, we report results using large and base models.
5.1.1
PERFORMANCE ON LARGE MODELS
CoLA Mcc
QQP
MNLI-m/mm
SST-2
STS-B
QNLI
RTE
MRPC
Avg.
Model
Acc
Acc
Acc
Corr
Acc
Acc
Acc
BERT BERTiarge large
60.6
91.3
86.6/-
93.2
90.0
92.3
70.4
88.0
84.05
RoBERTa TataReee laferge
68.0
92.2
90.2/90.2
96.4
92.4
93.9
86.6
90.9
88.82
XLNet gerge
69.0
92.3
90.8/90.8
97.0
92.5
94.9
85.9
90.8
89.15
ELECTR Aarge
69.1
92.4
90.9/-
96.9
92.6
95.0
88.0
90.8
89.46
DeBERTa Tatarge
70.5
92.3
91.1/91.1
96.8
92.8
95.3
88.3
91.9
90.00
Table 1: Comparison results on the GLUE development set.
We pre-train our large models following the setting of BERT (Devlin et al., 2019), except that we use the BPE vocabulary of Radford et (2019): Liu al. (20 9c) For training data, we use Wikipedia (English Wikipedia dump") 12GB), BookCorpus (Zhu et al. 2015) (6GB), OPENWEBTEXT (public Reddit content (Gokaslan & Cohen 2019) 38GB), and STORIES (a subset of CommonCrawl (Trinh & Le 2018); 31 3IGB). The total data size after data deduplication (Shoeybi et al., 2019) is about 78G. Refer Appendix A.2 for a detailed description of the pre-training dataset.
We use DGX-2 2 machines (96 V100 GPUs) to train the models. A single model trained with 2K batch size and MM steps takes about 20 days. Refer to Appendix A for the detailed hyperparamters.
We summarize the results on eight NLU tasks of GLUE (Wang et al. 2019b) in Table I where DeBERTa is compared DeBERTa with previous Transform- based PLMS similar structures (i.e. 24 layers with hidden size of 1024) including BERT, RoBERTa, XLNet, ALBERT and ELECTRA. Note that RoBERTa, XLNet and ELECTRA are pre-trained on 160G training data while DeBERTa is pre- trained on 78G training data. RoBERTa and XLNet pre-trained for 500K steps with 8K samples in step, which amounts to four billion training samples. DeBERTa pre-trained for one million steps with 2K samples in each step. This amounts to billion training samples, approximately half of either RoBERTa or Xe..t. Table [ showss that compared to BERT and RoBERTa, DeBERTa performs consistently better across all the tasks. Meanwhile, DeBERTa outperforms XLNNe in six of eight tasks. Particularly, the improvements on MRPC (1.1% over XLNet and 1.0% over RoBERTa), RTE (2.4% over XLNet and 1.7% over RoBERTa) and CoLA (1.5% over XLNet and 2.5% over RoBERTa) are significant. DeBERTa also outperforms other SOTA PLMs, i.e., ELECTRA and XLNet in terms of average GLUE score.
Among all GLUE tasks, MNLI is most often used as an indicative task to monitor the research progress of PLMss DeBERTa significantly outperforms all existing PL.Ms of similar size on MNLI and creates new state of the art.

---

Published as a conference paper at ICLR 2021
MNN.I-m/mm
SQuAD vI.1 SQuAD v2.0
RACE
ReCoRD FI/EM
SWAG
NER
Model
Acc
FI/EM
FI/EM
Acc
Acc
Fl
BERT large
86.6/-
90.9/84.1
81.8/79.0
72.0
86.6
92.8
ALBERT large
Laagg
86.5/-
91.8/852
84.9/81.8
75.2
SaBERTaiorpa
90.2/90.2
94.6/88.9
89.4/86.5
83.2
90.6/90.0
89.9
93.4
XLNettarge
90.8/90.8
95.1/89.7
90.6/87.9
85.4
Megatronæ36M
89.7/90.0
94.2/88.0
88.1/84.8
83.0
DcBERTajarge
91.1/99.1
95.5/90.1
90.7/88.0
86.8
91.4/91.0
90.8
93.8
aateReett ALBERT mustarge
90.8/-
94.8/89.3
90.2/87.4
86.5
Megatronj,ss
90.9/91.0
94.9/89.1
90.2/87.1
87.3
Megatron3,98
91.4/91.4
95.5/90.0
91.2/88.5
89.5
Table 2: Results on MNLI in/out-domain, SQuAD v1.1. 1. SQuAD v2.0, RACE, ReCoRD, SWAG, CoNLL 2003 NER development set. Note that missing results literature are signified by "-".
In addition GLUE, DeBERTa is evaluated three categories of NLU benchmarks: (1) Question Answering: SQuAD v1.1 (Rajpurkar et al. (016). SQuAD v2.0 (Rajpurkar et al. 2018), RACE (Lai et al. 2017), ReCoRD (Zhang et al. 2018 and SWAG (Zellers al. 2018): (2) Natural Language Inference: MNLI (Williams et al. 2018);; ; and (3) NER: CoNLL-2003. For comparison, we include ALBERT xxlarge (Lan et al. 2019) and Megatron (Shoeybi et al., 2019) with three different model sizes, denoted as Megatrons16M: Megatron and Megatron 9B respectively, which are trained using the same dataset as RoBERTa. Note that Megatron has similar model size as other models mentioned above
We summarize the results in Table Compared to the previous SOTA PLMs with a similar model size (i.e., BERT, RoBERTa, XLNet, ALBERT large, and Megatronsssx), DeBERTa shows superior performance in all seven tasks. Taking the RACE benchmark as an example, DeBERTa significantly outperforms XLNet by +1.4% (86.8% vs. 85 85.4%). Although Megatron; 33 is three times larger than DeBERTa, DeBERTa outperforms three of the four benchmarks. We further report DeBERTa on text generation tasks in Appendix A.4
5.1.2 PERFORMANCE ON BASE MODELS
Our setting for base model pre-training similar to that for large models. The base model structure follows that of the BERT base model, i.e., L 12, H 768, A = 12. We use 4 DGX-2 with 64 V100 GPUs to train the base model. It takes 10 days to finish a single pre- pre-training of 1M training steps with batch size 2048. We train DeBERTa using the same 78G dataset, and compare it to RoBERTa and Xeteet trained on 160G text data.
We summarize the base model results in Table Across all three tasks, DeBERTa consistently outperforms RoBERTa and XLNet by a larger margin than that in large models. For example, on MNLI-m, DeBERTanse obtains +1.2% (88.8% vs. 87.6%) over RoBERTanse. and +2% (88.8% vs. 86.8%) over XLNettase
Model
MNLI-m/mm (Acc)
SQuAD vI.1 (FI/EM)
SQuAD v2.0 (FI/EM)
RoBERTaiaa
87.6/-
91.5/84.6
83.7/80.5
XLNetbase
86.8/-
-/-
-/80.2
DeBERTatee
88.8/88.5
93.1/87.2
86.2/83.1
Table 3: Results on MNLI in/out-domain (m/mm), SQuAD v1.1 and v2.0 development set.
"The hidden dimension of ALBERT sxlage is 4 times of DeBERTa and the computation cost is about times of DeBERTa.
*T5 (Raffel et al. 2020) has more parameters (11B). Raffel et al. 2020 only report the test results of T5 which comparable with other models.
7

---

Published as a conference paper at ICLR 2021
5.2 MODEL ANALYSIS
In this section, we first present an ablation study to quantify the relative contributions of different components introduced in DeBERTa. Then, we study the convergence property characterize the model training efficiency. We run experiments for analysis using the base model setting: a model is pre-trained using the Wikipedia + Bookcorpus dataset for 1M steps with batch size 256 in days on DGX-2 machine with 16 V-100 GPUs. Due to space limit, we visualize the different attention patterns of DeBERTa and RoBERTa in Appendix A.7.
5.2.1 ABLATION STUDY
To verify experimental setting, pre-train the RoBERTa base model from scratch. The re-pre- trained RoBERTa model is denoted as RoBERTa-Relmppees To investigate the relative contributions of different components in DeBERTa, we develop three variations:
-EMD is the DeBERTa base model without EMD.
-C2P is the DeBERTa base model without the content-to-positee term ((c) in Eq.
-P2C is the DeBERTa base model without the position-to-conten term ((b) in Eq.
As
XLNet also uses the relative position bias, this model is close to XLNet plus EMD.
MNLI-m/mm Acc
SQuAD v1.1
SQuAD v2.0
RACE Acc
Model
F1/EM
FI/EM
BERT base Devlin al.
2019)
84.3/84.7
88.5/81.0
76.3/73.7
65.0
RoBERTa T
LiU et a
2019c
84.7/-
90.6/-
79.7/-
65.6
XLNetbase
Yang al. 2019)
85,8/85,4
-/-
81.3/78.5
66.7
RoBERTa-Relmpp
84.9/85.1
91.1/84.8
79.5/76.0
66.8
DeBERTabose
86.3/86.2
92.1/86.1
82.5/79.3
71.7
-MD
16.1/
91.8/85.8
81.3/78.0
70.3
C2P
85.9/85.7
91.6/85.8
81.3/78.3
69.3
P2C
86.0/85.8
91.7/85.7
80.8/77.6
69.6
(EMD+C2P)
85,8/85.9 85,8/85.8
91.5/85.3
80.3/77.2
68.1
(EMD+P2C)
91.3/85.1
80.2/77.1
68.5
Table 4: Ablation study of the DeBERTa base model.
Table summarizes results on four benchmark datasets. First, RoBERTa-Relmp performs similarly to RoBERTa across all benchmark datasets, verfiying that our setting is reasonable. Second, we see that removing any one component in DeBERTa results sheer performance drop. For instance, removing EMD (-EMD) results in loss of 1.4% (7 (71.7% 70.3%) RACE, 0.3% (92.1% vs. 91.8%) on SQuAD v1.1, 1.2% (82.5% vs. 81.3%) on SQuAD v2.0, 0.2% (86.3% vs. 86.1%) and 0.1 % (86.2% vs. 86.1%) MNLI-m/mm, respectively. Similarly, removing either content-to-positiee or position-to nosition-to-conter leads to inferior performance in all the benchmarks. As expected, removing two components results in even more substantial loss in performance.
5.3 SCALE UP 1.5 BILLION PARAMETERS
Larger pre-trained models have shown better generalization results (Raffel et al. 2020 Brown et al. 2020, (Shoeybi et al.) 2019). Thus, we have built larger version of DeBERTa with 1.5 billion parameters, denoted as DeBERTaI.sR. The model consists of 48 layers with hidden size of 1.536 and 24 attention heads 6 DeBERTaI55 is trained on pre-training dataset amounting to 160G, vocabulary of size 128K constructed using the dataset.
similar to that in Liu al. 2019c). with
To train DeBERTa1.5 we optimize the model architecture as follows. First, we share the projection matrices of relative position embedding W W with We.. W Wec respectively. in all attention layers to reduce the number of model parameters. Our ablation study in Table [3 on base models shows that the projection matrix sharing reduces the model size while retaining the model performance.
"See Table in Appendix for the model hyperparameters.
8

---

Published as a conference paper at ICLR 2021
Second, a convolution layer is added aside the first Transformer layer to induce n n-gram knowledge of sub-word encodings and their outputs are summed up before feeding to the next Transformer layer]
Table rreports the test results of SuperGLUE (Wang et al.
2019a) which is one of the most popular tasks, including Question Answering
NLU benchmarks. SuperGLUE consists of a wide of NLU
(Clark et al., [019 Khashabi (019/ahashabi et al. 2018 Zhang al. 2018).
Natural Language Inference (Dagan
et 20066 Bar- Bar-Haim et al. 2006;; Giampiccolo et al. 2007 Bentivogli et al. 2200), Word Sense
Disambiguation (Pilchvar & Camacho-Collados 2019), and Reasoning (Levesque
al. [01 I
2011). Since its release in 2019, top research teams around the world have been developing large-scale PLMs that have driven striking performance improvement on SuperGLUE.
Roemmele et al.
The significant performance boost due to scaling DeBERTa to a larger model makes the single DeBERTa 5B surpass the human performance on SuperGLUE for the first time in terms of macro- average score (89.9 versus 89.8) as of December 29, 2020, and the ensemble DeBERTa model (DeBERTagnsenete sits atop the SuperGLUEE benchmark rankings as of January 6, 2021, outper- forming the human baseline by decent margin (90.3 versus 89.8). Compared to T5, which consists of 11 billion parameters, the 1.5-billionpprrmeee and maintain, and it is easier to compress and deploy to apps of various settings.
DeBERTa is much more energy efficient to train
BoolQ Acc
CB
COPA
MultiRC Fla/EM
ReCoRD FI/EM
RTE
WiC
WSC|
Average Score
Model
F1/Acc
Acc
Acc
Acc
Acc
RoBERTa aaaage
87.1
90.5/95.2
90.6
84.4/52.5
90.6/90.0
88.2
69.9
89.0
84.6
NEXHA-Plus
87.8
94.4/96.0
93.6
84.6/55.1
90.1/89.6
89.1
74.6
93.2
86.7
T511B
91.2
93.9/96.8
94.8
88. 1/6.3
94.1/93.4
92.5
76.9
93.8
89.3
Tinst+Meena
91.3
95.8/97.6
97.4
88.3/63.0
94.2/93.5
92.7
77.9
95.9
90.2
Human
89.0
95.8/98.9
100.0
81.8/51.9
91.7/91.3
93.6
80.0
100.0
89.8
90.4
94.9/97.2
96.8
88.2/63.7
94.5/94.1
93.2
76.4
95.9
89.9
DeBERTa Ensemble
90.4
95.7/97.6
98.4
88.2/63.7
94.5/94.1 1
93.2
77.5
95.9
90.3
Table SuperGLUUE test set results scored using the SuperGI.UE evaluation server. All the results are obtained from httpss//supeegguueeeeennmaarokka on January 2021.
CONCLUSIONS
This paper presents a new model architecture DeBERTa (Decoding-enhanced BERT with disentangled attention) that improves the BERT and RoBERTa models using two novel techniques. The first is the disentangled attention mechanism, where each word is represented using two vectors that encode its content and position, respectively, and the attention weights among words are computed using disentangled matrices on their contents and relative positions, respectively. The second is an enhanced mask decoder which incorporates absolute positions in the decoding layer to predict the masked tokens model pre-training. addition, new virtual adversarial training method used for fine-tuning to improve model's generalization on downstream tasks.
We show through a comprehensive empirical study that these techniques significantly improve the efficiency of model pre-training and the performance of downstream tasks. The DeBERTa model with 1.5 billion parameters surpasses the human performance on the SuperGLUE benchmark for the first time in terms of macro-average score.
DeBERTa surpassing human performance on SuperGLUE marks an important milestone toward general AI. Despite its promising results SuperGLUU,, the model is by means reaching the hevan-level intelligence of NL.U. Humans are extremely good at leveraging the knowledge learned from different tasks solve a new task with no or little task-specific demonstration. This is referred to as compositional generalization, the ability to generalize to novel compositions (new tasks) of familiar constituents (subtasks or basic problem-solving skills). Moving forward, it worth exploring how to make DeBERTa incorporate compositional structures in more explicit manner, which could allow combining neural and symbolic computation of natural language similar to what humans do.
? Please refer to Table 12 in Appendix A.6 for the ablation study of different model sizes, and Table 13 in
Appendix fo the ablation study of new modifications.

---

Published as a conference paper at ICLR 2021
7 ACKNOWLEDGMENTS
We thank Jade Huang and Nikos Karampatziakis for proofreading the paper and providing insightful comments. We thank Yoyo Liang, Saksham Singhal, Xia Song, and Saurabh Tiwary for their help with large-scale model training. We also thank the anonymous reviewers for valuable discussions.
REFERENCES
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprini arXiv:1607.06450, 2016.
Roy Bar-Haim, Ido Dagan, Bill Dolan, Lisa Ferro, and Danilo Giampiccolo. The second PASCAL recognising textual entailment challenge. In Proceedings of the Second PASCAL Challenges Workshop on Recognising Textual Entailment, 01 2006.
Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.051100 2020.
Luisa Bentivogli, Ido Dagan, Hoa Trang Dang, Danilo Giampiccolo, and Bernardo Magnini. The fifth pascal recognizing textual entailment challenge. In Proc Text Analysis Conference (TAC'09, 2009.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005. 4665, 2020.
Daniel Cer, Mona Diab, Eneko Agirre. Inigo Lopez-Gazpie and Lucia Specia. Semeval-200 2017 task and cross-lingual focused evaluation. arXiv preprint
1: Semantic textual
arXiv: 00055, 2017.
Kezhen Chen, Qiuyuan Huang, Hamid Palangi. Paul Smolensky, Kenneth D Forbus, and Jianfeng Gao. Natural-to formal- language generation using tensor product representations. arXiv preprint 1rXiv:1910.02339, 2019.
Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv: /904. /0509, 2019.
Christopher Clark, Kenton Lee, Ming-Was Wei Chang. Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. Proceedings of NAACL-HLT 2019, 2019.
Kevin Clark, Minh- Minh-Thang Luong, Quoc Le, and Christopher D. Manning. ELECTRA: Pre-training text encoders as discriminators rather than generators, In ICLR, 2020.
Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textual entailment challenge. In Proceedings of the First International Conference on Machine Learning Chal- lenges: Evaluating Predictive Uncertainty Visual Object Classification, and Recognizing Textual Entailment, MLCW'05, Berlin, Heidelberg, 2006.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G Carbonell, Quoc Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 21p.2978-2990 2019.
De Marneffe, Mandy Simons, and Judith Tonhauser. The commitmentbank: In- vestigating projection in naturally occurring discourse. In proceedings of Sinn und Bedeutung. volume 23, pp. 107- 2019.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 4171-4186, 2019.
10

---

Published as a conference paper at ICLR 2021
William B Dolan and Chris Brockett. Automatically constructing corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005), 2005.
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao- Wuen Hon. Unified language model pre- tre-training for natural language understanding and generation. In Advances in Neural Information Processing Systems, pp. 13042-13054, 2019.
Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. The third PASCAL recognizing textual entailment challenge. In Proceedings of the ACI--PASCAL Workshop Textual Entailment and Paraphrasing, pp. 1-9, Prague, June 2007. Association for Computational Linguistics. URL
https: www. ac web. org/ antho wacethhoogyyyyoo ogy 0ooggy///0000000 /07-140111111111
Aaron Gokaslan and Vanya Cohen. Openwebtext corpus. http://Skylion007gitthhal 2019.
Pengeheng He, Xiaodong Liu, Weizhu Chen, and Jianfeng Gao. A hybrid neural network model for commonsense reasoning. arXiv preprint arXiv: /907. 107../1983, /983, 2019a.
Pengcheng He, Yi Mao, Kaushik Chakrabarti, and Weizhu Chen. X-sql: reinforce schema representa- tion with context. arXiv preprint arXiv: /9080080111111 / 13, 2019b.
Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Ian Simon, Curtis Hawthorne, Noam Shazeer, Andrew M Dai, Matthew Hoffman, Monica Dinculescu, and Douglas Eck. Music transformer: Generating music with long-term structure. 2018.
Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Tuo Zhao. SMART: Robust and efficient fine-tuning for pre-trained natural language models through principled regu- larized optimization. In ACL, July 2020. doi: 1e.18655/112220..cm motd-mmai..19))
Mandar Joshi, Danqi Chen, Yinhan Liu. Daniel Weld, Luke Zettlemoyer, and Omer Levy. Spanbert: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8:64-77, 2020.
Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. Looking beyond the surface: A challenge set for reading comprehension over multiple sentences. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Com- putational Linguisties: Human Language Technologies, Volume (Long Papers), pp. 252-262, 2018.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 2014.
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In International Conference Learning Representations, 2019.
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. Race: Large-scale reading comprehension dataset from examinations. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 785-794, 2017.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: lite bert for self-supervised learning of language representations. International Conference Learning Representations, 2019.
Hector Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. In Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning, 2012.
Hector Levesque, Ernest Davis, and Leora Morgenstem. The Winograd schema challenge. In AAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning, volume 46, pp. 47, 2011.
Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. On the variance of the adaptive learning rate and beyond. In International Conference on Learning Representations, 2019a.
11

---

Published as a conference paper at ICLR 2021
Xiaodong Liu, Pengeheng He, Weizhu Chen, and Jianfeng Gao. Multi-task deep neural networks for natural language understanding. In Proceedings of the 57th Annual Meeting of the Associa- tion for Computational Linguistics. pp. 4487-4490 Florence, Italy, July 2019b. Association for Computational Linguistics. URL http:::ttp//.o..... www. ac web.
441
Xiaodong Liu, Hao Cheng, Pengcheng He, Weizhu Chen, Yu Wang, Hoifung Poon, and Jianfeng Gao. Adversarial training for large neural language models. arXiv preprint arXiv:2004.08994, 2020.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach, arXiv preprint arXiv: 1907. 1692, 2019c.
Ilya Loshchilov and Frank Hutter. Fixing weight decay regularization in adam. 2018.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv, pp. arXiv-1609, 2016.
Shervin Minaee, Nal Kalchbrenner, Erik Cambria, Narjes Nikzad, Meysam Chenaghlu, and Jian- feng Gao. Deep learning based text classification: comprehensive review. arXiv preprint 0300: 2020.
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: regularization method for supervised and semi-supervised learning. IEEE transactions on pattern analysis and machine intelligence, 11999)1979,1993, 2018
Mohammad Taher Pilehvar and Jose Camacho- Comaach-Collados... Wic: the word-in pord-in-contex- dataset for evaluating context-sensitive meaning representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume 1 (Long and Short Papers), pp. 1267-1273, 2019.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl Blog, 1(8), 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020. URL
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference Empirical Methods in Natural Language Processing, November 2016.
Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't know: Unanswerable questions for squad. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 784-789, 2018.
Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S. Gordon. Choice of plausible alternatives: evaluation of commonsense causal reasoning. 20/ AAAI Spring Symposium Series, 2011.
Imanol Schlag, Paul Smolensky, Roland Fernandez, Nebojsa Jojic, Jürgen Schmidhuber, and Jianfeng Gao. Enhancing the transformer with explicit relational encoding for math problem solving. arXiv preprint arXiv 1010006900006111 2019.
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pp. 464 468, 2018.
Tao Shen, Yi Mao, Pengcheng He, Guodong Long, Adam Trischler, and Weizhu Chen. Ex- ploiting structured knowledge in text via graph-guided representation learning. arXiv preprint arXiv:2004. 14224, 2020.
12

---

Published as a conference paper at ICLR 2021
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-Im: Training multi-billion parameter language models using gpu model parallelism. arXiv preprint arX9v:1909.008 2019.
Paul Smolensky. Tensor product variable binding and the representation of symbolic structures in connectionist systems. Artificial intelligence, 46(1-2):1 1990.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference empirical methods in natural language processing, pp.
1631-1642, 2013.
Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng. Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian, and Hua Wu. Ernie: Enhanced representation through knowledge integration. arXiv preprint (EKiv::99040922... 2019.
Tricu H Trinh and Quoc V Le. A simple method for commonsense reasoning. arXiv preprint a8Xiv: 02847, 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman, Superglue: A stickier benchmark for general-purpose language understanding systems. In Advances in neural information processing systems, pp. 3266-3280, 2019a.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman, Glue: A multi-task benchmark and analysis platform for natural language understanding. In 7th Interna- tional Conference on Learning Representations, ICLR 2019, 2019b.
Wei Wang, Bin Bi, Ming Yan. Chen Wu, Zuyi Bao, Liwei Peng, and Luo Si. Structbert: Incor- porating language structures into pre-training for deep language understanding. arXiv preprint /908. 2019c.
Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. Neural network acceptability judgments. arXiv preprint 1805. 12471, 2018
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo- gies, Volume 1 (Long Papers), pp. 1112-1122... Association for Computational Linguisties, 2018. http:///aaaaahoooooo//o11-110
htRl:htt::////
Zhilin Yang, Zihang Dai, Yiming Yang. Jaime Carbonell, Russ R Salakhutdinov, and Quoe V Le. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, pp. 5754-5764, 2019.
Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. Swag: A large-scale adversarial dataset for grounded commonsense inference, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 93-104, 2018
Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. ReCoRD: Bridging the gap between human and machine commonsense reading comprehension. arXiv preprini 1810. / 12885, 2018.
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international conference on computer vision, pp. 19-27, 2015.
13

---

Published as a conference paper at ICLR 2021
A APPENDIX
A.1 DATASET
Corpus
Task
#Train
#Dev
#Test
#aaeel
Metrics
General I Language
Understanding Evaluation (GLUE)
CoLA
Acceptability
8.5k
lk
lk
2
Matthews corr
SST
Sentiment
67k
872
1.8k
2
Accuracy
MNLI
NLI
393k
20k
20k
Accuracy
RTE
NLI
2.5k
276
3k
2
Accuracy
WNLI
NLI
634
71
146
2
Accuracy
QQP
Paraphrase
364k
40k
391k
2
Accuracy/Fi
MRPC
Paraphrase
3.7k
408
1.7k
2
Accuracy/F1
QNLI
QA/NLI
108k
5.7k
5.7k
2
Accuracy
STS-B
Similarity
7k
1.5k
1.4k
Pearson/Spearman corr
SuperGLUE
WSC
Coreference
554k
104
146
2
Accuracy
BoolQ
QA
9,427
3,270
3,245
Accuracy
COPA
QA
400k
100
500
2
Accuracy
CB
NLI
250
57
250
3
Accuracy/FI
RTE
NLI
2.5k
276
3k
2
Accuracy
WIC
WSD
2.5k
276
3k
2
Accuracy
ReCoRD
MRC
101k
10k
10k
Exact Match (EM)/F1
MultiRC
Multiple choice
5,100
953
1, 1,800
Exact Match (EMyFI
Question Answering
SQuAD vL.1
MRC
87.6k
10.5k
9.5k
Exact Match (EM)/FI
SQuAD v2.0
MRC
130.3k
11.9k
8.9k
Exact Match (EM)/F
RACE
MRC
87,866
4,887
4,934
4
Accuracy
SWAG
Multiple choice
73.5k
20k
20k
4
Accuracy
Token Classification
CONLL 2003
NER
14,987
3,466
3,684
8
F1
Table 6: Summary information of the NLP application benchmarks.
GLUE. The General Language Understanding Evaluation (GLUE) benchmark is a collection of nine natural language understanding (NLU) tasks. As shown in Table 6 it includes question answer-
ing (Rajpurkar et al. 2016), linguistic acceptability (Warstadt et al. 2018). sentiment analysis (Socher
al. (013). text similanity (Cer et al. 2017), paraphrase detection (Dolan & Brockett] (2055). and
natural language inference (NLI) (Dagan et al. 2006 Bar-Haim et al. 2006) Giampiccolo et al.
2007 Bentivogli al.) 2009 Levesque
2012) Williams al. al.] 2018)) >) The diversity of the tasks makes GLUE very suitable for evaluating the generalization and robustness of NLU models.
SuperGLUE. SuperGLUE is an extension of the GLUE benchmark, but more difficult, which is collection of eight NLU tasks. It covers various of tasks including question answering (Zang
al. 2018 Clark et al. 2019 Khashabi et al. 2018), natural language inference (Dagan al. 2007 Bentivogli et al. 22099 De Marneffe et al. 2019), coreference resolution (Levesque et al. 2012) and word sense disambiguation (Pilehvar &
2006
Bar-Haim et al. 2006 Giampiccolo et al.
Canacho-Collados] 2019).
RACE is large-scale machine reading comprehension dataset, collected from English examinations in China, which designed for middle school and high school students (Lai et al. 2017).
SQuAD v1.1/v2.0 is the Stanford Question Answering Dataset (SQuAD) v1.1 and v2.0 (Rajpurkar et al. 2016: 2018) are popular machine reading comprehension benchmarks. Their passages come from approximately 500 Wikipedia articles and the questions and answers are obtained by crowd- sourcing. The SQuAD v2.0 dataset includes unanswerable questions about the paragraphs.
14

---

Published as a conference paper at ICLR 2021
SWAG is large-scale adversarial dataset for the task of grounded commonsense inference, which unifies natural language inference and physically grounded reasoning (Zellers et al. 2018). SWAG consists of 113k multiple choice questions about grounded situations.
CoNLL 2003 an English dataset consisting text from a wide variety of sources. has 4 types of named entity.
A.2 PRE-TRAINING DATASET
For DeBERTa pre-training, we use Wikipedia (English Wikipedia dump 12GB), BookCorpus Zhu et al. 2015 (6GB), OPENWEBTEXT (public Reddit content (Gokaslan & Cohen 2019); 38GB) and S STORIES (a subset CommonCrawi (Trinh & Le 2018); 31GB). The total data size after data deduplication(Shuee data as the validation set to monitor the training process. Table 7 compares datasets used different pre-trained models.
et al., 2219)) is about 78GB. For pre pre-traimin mre-training, also sample 5% training
Model
Wiki+Book 16GB
OpenWeTText 38GB
Stories
CC-News
Giga5 16GB
ClueWeb 19GB
Common Crawl 110GB
31GB
76GB
BERT
XLNet
RoBERTa
v
DeBERTa
V
DcBERTaI.58
Table 7: Comparison of the pre-training data.
A.3 IMPLEMENTATION DETAILS
Following RoBERTa (Liu et al. 2019c), adopt dynamic data batching. We also include span masking (Joshi et al. 2020) as an additional masking strategy with the span size up to three, We list the detailed hyperparameters of pre-training in Table 8 For pre-training, we use Adam (Kingma &
Ba 2014) as the optimizer with weight decay (Loshchilov & Hutter 2018). For fine-tuning, even though we can get better and robust results with RAdam(Liu et al.) (019a) on some tasks, c.g. CoLA, RTE and RACE, we use Adam(Kingnm) & Ba 2014) as the optimizer for a fair comparison. For fine-tuning, we train each task with
search procedure, each run takes about 1-2 hours on DGX-2 node. All the prprmmprrseetess are presented in Table The model selection is based on the performance on the tppc-fee development sets.
Our code is implemented based on Huggingface Transformer! !1 ct al. 2019 13
FainSed
12
and Megatron Shheybi
A.3.1 PRE-TRAINING EFFICIENCY
To investigate the efficiency of model pre-training, we plot the performance of the fine-tuned model on downstream tasks as a function of the number pre-training steps. As shown Figure for RoBERTa-Relmpuus. and DeBERTasase: we dump checkpoint every 150K pre-training steps, and then fine-tune the checkpoint on two representative downstream tasks, MNLI and SQuAD v2.0, and then report the accuracy and Fl score, respectively. As reference, also report the final model performance both the original RoBERTabase (Liu et al. 2019c) and XLNetbaas (Yang et al. 2019). The results show that DeBERTaaase consistently outperforms RoBERTa -erTaReeembase during the course of pre-training.
bookcorpus
12 hhtpp://gthhb..cmmiyyocccaaaaaa
13 https://gthhb.ccm/////DAAAggggg
15

---

Published as a conference paper at ICLR 2021
Hyper-parameter
DeBERTay.58
DeBERTatarge
DeBERTapuse DeBERTT GeBERTabasee - ablation
Number Layers
48
24
12
Hidden size
1536
1024
768
768
FNN inner hidden size
6144
4096
3072
3072
Attention Heads
24
16
12
12
Attention Head size
64
64
64
64
Dropout
0.1
0.1
0.1
0.1
Warmup Steps
10k
10k
10k
10k
Learning Rates
1,5e-4
2e-4
2e-4
le-4
Batch Size
2k
2k
2k
256
Weight Decay
0.01
0.01
0.01
0.01
Max Steps
IM
IM
IM
IM
Learning Rate Decay
Linear
Linear
Linear
Linear
Adam
le-6
le-6
le-6
le-6
Adam $1
0.9
0.9
0.9
0.9
Adam 82
0.999
0.999
0.999
0.999
Gradient Clipping
1.0
1.0
1.0
1.0
Number DGX-2 nodes
16
Training Time
30 days
20 days
10 days
7 days
Table Hyper-parameters for pre-training DeBERTa.
DeBERTa1.52
DeBERTatarge
DeBERTabase
Dropout of task layer
10,0.15,0.3)
(0.0.1.0.15)
(0,0.1,0.15)
Warmup Steps
150,100,500,
(501100.500,1000
150.100,500,1000
Learning Rates
{ (le-6, 3e-6, 5e-6]
(5e-6, 8e-6, 9e-6, le-5)
(1.5e-5,2e-5, 3e-5, 4e-5) (16,32,48,64)
Batch Size
(16,32,64)
(16.32,48,64)
Weight Decay
0.01
0.01
Maximun Training Epochs
10
10
10
Learning Rate Decay
Linear
Linear
Linear
Adam
le-6
Ie-6
Ie-6
Adam 31
0.9
0.9
0.9
Adam 3g
0.999
0.999
0.999
Gradient Clipping
1.0
1.0
1.0
Table Hyper-parameters for fine-tuning DeBERTa on down-streaming tasks.
82
86
Accuracy (MNLI)
(SQuAD v2.0)
84
RoBERTa-RelmPoer
RoBERTa-Relmp.se
DeBERTanae
76
DeBERTaase
82
RoBERTaessa
RoBERTamis
XLNetaaie
XLNetoase
150k 250k 350k 450k 550k 650k 750k 850k 1M Number of pre-training steps
150k 250k 350k 450k 550k 650k 750k 850k 1M Number of pre-training steps
(a) Results MNI. development
(b) Results SQuAD v2.0 development
Figure 1: Pre-training performance curve between DeBERTa and its counterparts the MNLI and SQuAD v2.0 development set.
16

---

Published as a conference paper at ICLR 2021
A.4 MAIN RESULTS ON GENERATION TASKS
In addition NLU tasks, DeBERTa can also be extended handle NLG tasks. To allow DeBERTa operating like auto-regressive model for text generation, we use triangular matrix for self- attention and set the upper triangular part of the self-attention mask -c,, following Dong et al. (2019).
We evaluate DeBERTa on the task of auto-regressive language model (ARLM) using Wikitext- 103 (Merity et al. 2016). To do so, train new version of DeBERTa, denoted as DeBERTa-MT. is jointly preitreined using the MLM and ARLM tasks as in UniLM (Dong et al. 2019). The pre-training syper-parametery follows that of DeBERTatase except that use fewer training steps (200k). For comparison, we use RoBERTa as baseline, and include GPT-2 and Transformer-Xi as additional references. DeBERTa-AP is variant of DeBERTa where absolute position embeddings are incorporated in the input layer as RoBERTa. For a fair comparison, all these models are base models pre-trained in similar setting.
Model
RoBBRTa DeBERTa-AP DeBERTa DeBERTa-MT GPT-2 Transformer- XL
Dev PPL
21.6
20.7
20.5
19.5
23.1
Test PPL
21.6
20.0
19.9
19.5
37.50
24
Table 10: Language model results in perplexity (lower better) on Wikitext-100
Table 10 summarizes the results on Wikitext- 103. We see that DeBERTagne obtains lower perplexities on both dev and test data, and joint training using MLM and ARLM reduces perplexity further. That DeBERTa-AP is inferior to DeBERTa indicates that it is more effective to incorporate absolute position emheddings of words in the decoding layer the EMD in DeBERTa than in the input layer as RoBERTa.
A.5 HANDLING LONG SEQUENCE INPUT
With relative position bias, we choose to truncate the maximum relative distance to & as in equation 3 Thus in each layer, each token can attend directly to at most 2(k 1) tokens and itself. By stacking Transformer layers, each token in the /-th layer can attend to at most (2k: 1)l tokens implicitly. Taking DeBERTatorge as an example, where = 512, = 24, in theory, the maximum sequence length that can be handled is 24,528. This is byproduct benefit of our design choice and we find it beneficial for the RACE task. A comparison of long sequence effect on the RACE task is shown in Table 11
Sequence length
Middle High Accuracy
512
88.8
85.0
86.3
768
88.7
86.3
86.8
Table 11: The effect of handling long sequence input for RACE task with DeBERTa
Long sequence handling is an active research area. There have been a lot of studies where the Transformer architecture is extended for long sequence handling (Beltgy et al.] [2220: Kitaev et al.) 2019, Child et al. 2019 DDai et al. 2019). One of our future research directions to extend DeBERTa to deal with extremely long sequences.
A.6 PERFORMANCE IMPROVEMENTS OF DIFFERENT MODEL SCALES
In this subsection, study the effect of different model sizes applied to large models on GLUE. Table summarrize the results, showing that larger models can obtain a better result and SiFT also improves the model performance consistently.
17

---

Published as a conference paper at ICLR 2021
CoLA
QQP
MNLI-m/mm
SST-2
STS-B
QNLI
RTE
MRPC
Avg.
Model
Mec
Acc
Acc
Acc
Cor
Acc
Acc
Acc
DeBERTa targge
70.5
92.3
91.1/91.1
96.8
92.8
95.3
88.3
91.9
90.00
DeBERTaggok. M
71.1
92.3
91.7/91.6
97.5
92.0
95.8
93.5
93.1
90.86
DeBERTa L.5
72.0
92.7
91.7/91.9
97.2
92.9
96.0
93.9
92.0
91.17
DeBERTa 1
73.5
93.0
92.0/92.1
97.5
93.2
96.5
96.5
93.2
91.93
Table 12: Comparison results of DeBERTa models with different sizes on the GLUE development set.
Model
Parameters
MNLI-m/mm
SQuAD v1.1
SQuAD v2.0
Acc
F1/EM
F1/EM
GaBERTa-Relmpoaar
120M
84.9/85.1
91.1/84.8
79.5/76.0
DeBERTaa
I34M
86.3/86.2
92.1/86.1
82.5/79.3
+ ShareProjection
120M
86.3/86.3
92.2/86.2
82.3/79.5
+ Conv
122M
86.3/86.5
92.5/86.4
82.5/79.7
+ 128k Vocab
190M
86.7/86.9
93.1/86.8
83.0/80.1
Table 13: Ablation study of the additional modifications in DeBERTa 158 and DeBERTag00M models. Note that we progressively add each component on the top of DeBERTapase:
A.7 MODEL COMPLEXITY
With the disentangled attention mechanism, introduce three additional sets of parameters War. Wk, Rd and P E R2 The total increase model parameters 2T. x + 2k For the large model (d= 1024, L 24, k 512). this amounts about 49.M additional parameters, an increase of 13%. For the base (d = 768, L = 12, k 512), this amounts to 14M additional parameters, an increase of 12% However, by sharing the projection matrix between content and position embedding, i.e. War Wq.c. Wk, = Wk,e, the number of parameters of DeBERTa is the same as RoBERTa. Our experiment on base model shows that the results are almost the same, as in Table K
The additional computational complexity is ( due to the calculation of the additional position- to-content and content-re rontent-to-positeee to-position attention scores. Compared with BERT or RoBERTa, this increases the computational cost by 30%. Compared with XLNet which also uses relative position embedding. the increase of computational cost is about 15%. A further optimization by fusing the attention computation kernel can significantly reduce this additional cost. For EM since the decoder in pre-traine pre-training only reconstructs the masked tokens, it does not introduce additional computational cost for unmasked tokens. In the situation where 15% tokens are masked and we use only two decoder layers, the additional cost 0.15 x 2/ which results in an additional computational cost of only 3% for base model(I. = 12) and 2% for large model(L. = 24) in EMD.
A.8 ADDITIONAL DETAILS OF ENHANCED MASK DECODER
The structure EMD shown in Figure 2b There are two inputs for EMD, (i.c., I, H). H denotes the hidden states from the previous Transformer layer, and can be any necessary information for decoding, e.g., H, absolute position embedding or output from previous EMD layer. n denotes n stacked layers of EMD where the output of each EMD layer will be the input for next EMD layer and the output of last EMD layer will be fed to the language model head directly. The n layers can share the same weight. In our experiment we share the same weight for a layers reduce the number of parameters and use absolute position embedding as of the first EMD layer. When = H and 1, EMD is the same as the BERT decoder layer. However, EMD more general and flexible as it can take various types of input information for decoding.
A.9 ATTENTION PATTERNS
To visualize how DeBERTa operates differently from RoBERTa, we present in Figure the attention patterns (taken in the last self-attention layers) of RoBERTa, DeBERTa and three DeBERTa variants.
18

---

Published as a conference paper at ICLR 2021
Language Model Head
Language Model Head
Transformer Layer
Transformer Layer
Xn
K
H
(a) BERT decoding layer
(b) Enhanced Mask Decoder
Figure 2: Comparison of the decoding layer.
DeBERTa
RoBERTa
DeBERTa-END
DeBERTa- C2P
DeBERTa-PZZ
08
0.6
141210 8
0.2
12
1012
80
Figure 3: Comparison of attention patterns the last layer among DeBERTa, RoBERTa and DeBERTa variants (i.e., DeBERTa without EMD, C2P and P2C respectively).
We observe two differences. First, RoBERTa has clear diagonal line effect for token attending to itself. But this effect is not very visible DeBERTa. This can be attributed the use of EMD, in which the absolute position embedding is added to the hidden state of content the query vector, as verified by attention pattern DeBERTa-ERE than that of the original DeBERTa. Second, we observe vertical strips in the attention patterns of RoBERTa, which are mainly caused by high-frequent functional words or tokens (e.g., "a", "the", and punctuation). For DeBERTa, the strip only appears in the first column, which represents the [CLS] token. We conjecture that a dominant emphasis on [CLS] desirable since the feature vector of [CLS] is often used contextual representation of the entire input sequence in downstream tasks. We also observe that the vertical strip effect quite obvious in the patterns of the three DeBERTa variants.
where the diagonal line effect more visible
We present three additional examples to illustrate the different attention patterns of DeBERTa and RoBERTa in Figures and 5
19

---

Published as a conference paper at ICLR 2021
1.0
0.6
0.4
12 10
0.2
14
DeRERTa
RoBERTs
0.8
30
0.6
0.4
60
10
(b)
DeBERla
RoBERTa
20
30
0.8
0.6
80
90
0.4
320
0.2
130
140
36
104
120
128
0
8
48
98
27
38
88
104
1-2
120
120
138
144
Figure 4: Comparison attention patterns of the last layer between DeBERTa and RoBERTa.
20

---

Published as a conference paper at ICLR 2021
DeBERTa
DeBERTeEMD
DeBERTa-C2P
DeBERTe-P2C
0.5
32
(a)
DeBERTa
DeBERTI-EMD
DeBERTO-C2P
0
08
06
40
58
60
02
0 CORARE
0
0.8
40
50
90
100
02
150
140
3
Figure Comparison on attention patterns of last layer between DeBERTa and its variants (i.e. DeBERTa without EMD, C2P and P2C respectively).
A. ACCOUNT FOR THE VARIANCE IN FINE-TUNING
Accounting for the variance of different runs of fine-tuning, our experiments, we always follow Liu et al. (001cc) report the results downstream tasks by averaging over five with different random initialization seeds, and perform significance test when comparing results. As the examples shown in Table 14 DeBERTamss significantly outperforms RoBERTa (p-value < 0.05).
Model RoBERTaboss
MNLI-matched (Min/Max/Avg)
SQuAD vI.1 (Min/Max/Avg)
p-value
84. 84.7/85.0/84
90.8/91.3/91.1 91.8/92.2/92.1
0.02
DcBERTatase
86. 1/86./6688
0.01
Table 14: Comparison of DeBERTa and RoBERTa on MNLI-matched and SQuAD v1.1.
21


Reviews:
[{"review_id": "tM7jxs-mDwp", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, an improvement of BERT model is proposed. It relies on the disentanglement of contents and relative positions in the encoding layers and on the incorporation of absolute positions in the decoding layer.\n\n\nStrengths:\n* The paper is well written, the positioning to the state of the art is clear and the method is rigorously described. \n* The paper provides a complete evaluation using the existing benchmarks for NLP and including ablation studies and evaluation of pre-training efficiency and Deberta improves results in the major part of the cases.\n\n\nWeaknesses:\n* The proposed method is a relative increment of previous methods.\n* In Section 4.1.1., the way performance increase or decrease is reported is not exact (1.1% -> 1.1 points)\n* Do we have an idea of the statistical significance of the improvements?\n* It would be interesting to have the rationale for the mitigated result obtained on Table 1. Is Deberta more relevant for specific tasks?\n* The authors claim that they evaluate their results on generation task but it rather seems that they evaluate language modeling using perplexity. \n*The use of non documented acronyms (ppl, for example) that could be not understandable outside the NLP community.\n*They are some redundancy in the text (second paragraph of 3.2 and fourth paragraph of the introduction) that is not necessary.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "tAB4kErmoV", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a BERT-inspired model that adds a two main different architectural decisions: different content and position representations (instead of a sum), and absolute positions in the decoding layer. The authors run the standard suite of GLUE benchmark experiments, on both “large” and “base” setups, as well as a generation setup (Wikitext-103). \n\nThe modifications proposed are not game-changing, but the evaluations are interesting in terms of understanding the impact of these modifications. One thing that I find disingenuous is fact that their disentangled approach does introduce additional parameters, which is not quantified (or even mentioned) in the main paper. I had to dig into the Appendix to see that this introduces about 49M additional parameters (increment of 13%).\n\nAnother problem that I have is with their experimental comparisons, especially the ones in main part, Sec 4.1.1. I’m listing below the most important issues in this section:\n\n“RoBERTa and XLNet are trained for 500K steps with 8K samples in a step, which amounts to four billion passes over training samples”. This is confusing; what you mean to say is that the models see about four billion training examples. The term “passes” is used usually as an equivalent to “epochs”, ie how many times the model goes over the entire training set.\n\n“[...] Table 1, which compares DeBERTa with previous models with around 350M parameters: BERT, RoBERTa, XLNet, ALBERT and ELECTRA.” Note that ALBERT is actually around 235M parameters, significantly less than all the others. You cannot simply bundle all together and claim they are equivalent parameter-size--wise.\n\n“DeBERTa still outperforms them [ALBERT_xxlarge] in term  of the average “GLUE” score.” Note that the difference here wrt ALBERT_xxlarge is from 89.96 to 90.00, ie 0.04 for the average, with a tie 3-3 in terms of wins for specific tasks. Unless you can show that the 0.04 difference is statistically significant, you need to tone down the claim about “outperforming”.\n\n\n“We summarize the results in Table 2. Compared to the previous SOTA models with similar sizes, including BERT, RoBERTa, XLNet and Megatron336M, DeBERTa consistently outperforms them in all the 7 tasks. Taking RACE as an example, DeBERTa is significantly better than previous SOTA XLNet with an improvement of 1.4% (86.8% vs. 85.4%).”\nFor whatever reason, the authors omit ALBERT from the comparison done for Table 2, in spite of its even smaller size compared to the included ones, and the fact that the ALBERT numbers for these tasks are readily available in the paper. Taking RACE as an example: ALBERT (single model) has 86.5% accuracy, therefore nullifying the claim of 1.4% improvement.\n\n\nRe: References\n\nA lot of the references use the Arxiv version for papers that have been peer-reviewed and published. Please fix.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "hbRWP5lM16H", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposed a novel attention mechanism and a new objective function that mitigates the distribution shifts caused by masked tokens for downstream tasks in MLM. It demonstrates superior performance across benchmarks.\n\nPros:\n1. Good empirical results are demonstrated across an extensive suite of benchmarks. Ablation studies are well done. Hence I am willing to give a score of 6 despite of the following concerns.\n\nCons:\n1. My major concern is about the novelty of this paper. \nIn transformer-XL[1], the idea of relative positional information in the form of Eq (2) was already introduced. The paper somehow intentionally omit the discussion following (2), only mentioning two earlier works of (Shaw et al., 2018; Huang et al., 2018). I think the author should be honest and compare with relative positional information introduced in transformer-XL in the forefront. \nThat being said, there is obviously still differences between transformer-XL and the proposed methods. And also the introduction of novel objectives in addition to the attention mechanism. \n\n2. However, the previous concern brought up the second concern I have about the evaluations. Since the modification relative positional information of transformer-XL to the proposed method is not too large, I wonder if there is a reason to explain the better performances of the proposed methods. Hence I am worried if the baseline such as XLNet was well-tuned. We can see that for example in [2], the performance of XLNet was much better than originally reported. I think the author should try to carefully evaluate the relative positional mechanisms of prior works with authors' own infrastructure, while having everything else fixed.\n\n3. I find the word \"disentangled\" a bit misleading in this context. Disentanglement in ML [3] often refers to the ability to disentangle factors of variations of the data. The work does not make use of any disentangled techniques, or have disentanglement representation/architectures. It simply use a relative position mechanism that's the sum of four matrix products. \n\n[1] Dai et. al. Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context\n\n[2] Dai et. al. Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing\n\n[3] Locatello et. al., Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations", "extra_scores": {}, "extra_text": {}}, {"review_id": "WYwTmQDzGb3", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary and Contributions\n\nThe authors proposed an extension to the word representation transformer architecture  that takes into account disentangle features for position and content. The disentangle of attention is based on the composition of a content and position parameter matrices, in addition with combinations of both.  The main contribution is to tackle issues with the relative position embeddings used on standard transformer architectures. The proposed model shows improvements on some benchmarks by using less pre-training data compared to the baseline.\n\nStrengths\n\n- The proposed model tackles a known issue in transformer architectures.\n- The authors perform a comprehensive comparison on standard text benchmarks as well as an ablation study.\n- The findings show that disentangle attention improves results on some text benchmarks.\n\nWeaknesses\n\n- Related work on disentangle representations for text, and the further motivation for using disentanglement into the attention model are not discussed.\n- Missing results of the variance in metrics with multiple runs on the downstream tasks. As an extra contribution, the authors could  show if the improvements are due to the proposed model or variance in parameter initialisation. \n\nQuestions to the Authors\n\n- Could you elaborate on disentangled representations and how they relate to the proposed attention model? \n- How does it compare the enhanced masked language model with the masked language model?\n- How does the relative position parameter matrix is initialised, and how does it affect the language model performance? \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers gave, though not very strong,  positive scores for this work.  Although the technical contribution of the paper is somewhat incremental, the reviewers agree that it solidly addresses the known important issues in BERT, and the experiments are extensive enough to demonstrate the empirical effectiveness of the method.  The main concerns raised by the reviewers are regarding the novelty and the discussion with respect to related work as well as some unclear writings in the detail,  but I think the pros outweigh the cons and thus would like to recommend acceptance of the paper.

We do encourage authors to properly take in the reviewers' comments to further polish the paper in the final version.


Decision: Accept (Poster)