DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing
Authors: Pengcheng He
Jianfeng Gao
Weizhu Chen
Venue: ICLR 2023 poster (2023)

Abstract: This paper presents a new pre-trained language model, NewModel, which improves the original DeBERTa model by replacing mask language modeling (MLM) with replaced token detection (RTD), a more sample-efficient pre-training task. Our analysis shows that vanilla embedding sharing in ELECTRA hurts training efficiency and model performance. This is because the training losses of the discriminator and the generator pull token embeddings in different directions, creating the “tug-of-war” dynamics. We thus propose a new gradient-disentangled embedding sharing method that avoids the tug-of-war dynamics, improving both training efficiency and the quality of the pre-trained model. We have pre-trained NewModel using the same settings as DeBERTa to demonstrate its exceptional performance on a wide range of downstream natural language understanding (NLU) tasks. Taking the GLUE benchmark with eight tasks as an example, the NewModel Large model achieves a 91.37% average score, which is 1.37% over DeBERTa and 1.91% over ELECTRA, setting a new state-of-the-art (SOTA) among the models with a similar structure. Furthermore, we have pre-trained a multi-lingual model mNew-Model and observed a larger improvement over strong baselines compared to English models. For example, the mNewModel Base achieves a 79.8% zero-shot cross-lingual accuracy on XNLI and a 3.6% improvement over XLM-R Base, creating a new SOTA on this benchmark. We will make our model and code publicly available.

Published as a conference paper at ICLR 2023
DEBERTAV3:
IMPROVING DEBERTA
USING
ELECTRA-STYLE TRAINIGGN WITH GRADIENT-DISENTANGLED EMBEDDING SHAR-
ING
Pengcheng He', Jianfeng Gao2, Weizhu Chen'
Microsoft Azure Al Microsoft Research (penhe, jfgao, wzchen @ com
ABSTRACT
This paper presents a new pre-trained language model, DeBERTaV3, which im- proves the original DeBERTa model by replacing masked language modeling (MLM) with replaced token detection (RTD), more sample-efficient pre-training task. Our analysis shows that vanilla embedding sharing in ELECTRA hurts train- ing efficiency and model performance, because the training losses of the discrimina- tor and the generator pull token embeddings different directions, creating the "tug- of-war" dynamics. We thus propose new ing method that avoids the -f- dynamics, improving both training efficiency and the quality of the pre-trained model. We have pre-trained DeBERTaV3 using the same settings as DeBERTa to demonstrate its exceptional performance on a wide range of downstream natural language understanding (NLU) tasks. Taking the GLUE benchmark with eight tasks as an example, the DeBERTaV3 Large model achieves a 91.37% average score, which is 1.37% higher than DeBERTa and 1.91% % higher than ELECTRA, setting new state-of-the-an (SOTA) among the models with similar structure. Furthermore, we have pre-trained multilingual model mDeBERTaV3 and observed a larger improvement over strong baselines compared to English models. For example, the mDeBERTaV3 Base achieves a 79.8% zero-shot cross- lingull accuracy XNLI and 3.6% improvement over XLM-R Base, creating a new SOTA on this benchmark. Our models and code are publicly available at htttts://// thub. com/m crosoott
embedding shar-
INTRODUCTION
Recent advances in Pre-trained Language Models (PLMs) have created new state-of-the-as results on many natural language processing (NLP) tasks. While scaling up PLMs with billions trillions of parameters (Raffel et al. 2020; Radford et al.) [019]] Brown et al. 2020 He et al. 2020 Fedus et al.) 2021) is a well-proved way improve the capacity of the PLMs, it more important explore more energy-efficient approaches to build PLMs with fewer parameters and less computation cost while retaining high model capacity.
Towards this direction, there are a few works that significantly improve the efficiency of PLMs. The first RoBERTa (Liu et al.] (019)) which improves the model capacity with a larger batch size and more training data. Based on RoBERTa, DeBERTa (He et al. (020) further improves the pre-training efficiency by incorporating disentangled attention which an improved relative- posittion encoding mechanism. By scaling up to 1.5B parameters, which is about eighth of the parameters of xxlarge T5 RRffel et al. 2020), DeBERTa surpassed human performance on the SuperGLUE (Wang et al.) 2019a) leaderboard for the first time. The second new pre-training approach to improve efficiency is Replaced Token Detection (RTD), proposed by ELECTRA (Clark et al.] 2020). Unlike BERT (Devlin et al., 2019), which uses transformer encoder to predict corrupt tokens with masked language modeling (MLM), RTD uses generator to generate ambiguous corruptions and a discriminator to distinguish the ambiguous tokens from the original inputs, similar Generative Adversarial

---

Published as a conference paper at ICLR 2023
Networks (GAN). The effectiveness of RTD is also verified by several works, including CoCo-LM (Meng et al., (022). XLM-E (Chi et al. 2021). (Feng et al. (020) and SmallBenchNLP (Kanakarajan et al.
202I
In this paper, we explore two methods of improving the efficiency of pre-training DeBERTa. Follow- ing ELECTRA-style training, we replace MLM in DeBERTa with RTD where the model is trained as a discriminator to predict whether a token in the corrupt input is either original or replaced by a generator. We show that DeBERTa trained with RTD significantly outperforms the model trained using MLM. The second is a new embedding sharing method. In ELECTRA, the discriminator and the generator share the same token embeddings. However, our analysis shows that embedding sharing hurts training efficiency and model performance, since the training losses of the discriminator and the generator pull token embeddings into opposite directions. This is because the training objectives between the generator and the discriminator are very different. The MLM used for training the generator tries to pull the tokens that are semantically similar close to each other while the RTD of the discriminator tries to discriminate semantically similar tokens and pull their embeddings as far as possible to optimize the binary classification accuracy, causing a conflict between their training objectives. In other words, this creates the "tug-of- war" dynamics that reduces the training efficiency and the model quality, as illustrated in Haddsell et al. (2020). On the other hand, we show that using separated embeddings for the generator and the discriminator results in significant performance degra- dation when we fine-tune the discriminator on downstream tasks, indicating the merit of embedding sharing, e.g., the embeddings of the generator are beneficial to produce better discriminator, as argued in Clark et al. (202)). To balance these tradeoffs, we propose a new gradient-disentany ditt--diinnnnedd embedding sharing (GDES) method where the generator shares its embeddings with the discriminator but stops the gradients from the discriminator to the generator embeddings. This way, we avoid the tug-of seg-of-war war effect and preserve the benefits of embedding sharing. We empirically demonstrate that GDES improves both pre-training efficiency and the quality of the pre-trained models.
We pre-train four variants of DeBERTaV3 models, i.e., DeBERTa Iarge. DeBERTa Busee DeBERTaV33mal and DeBERTaV3 We evaluate them on various representative natural language understanding (NLU) benchmarks and set new state-of-the-an art numbers among models with a similar model structure. For example, DeBERTaV V3 surpasses previous SOTA models with similar model structure on GLUE (Wang et al., 2019b> benchmark with an average score over +1.37% which is significant. DeBERTaV 3 achieves a 90.6% accuracy score the MNLI-matched (Williams) et al,, (211)) evaluation set and an 88.49 4%.4% Fl score on the SQuAD v2.0 (Rajpurkar et al. (011)) evaluation set. This improves DeBERTa Tassse by 1.8% and 2.2%, respectively. Without knowledge distillation, DeBERTaV3 small and 3 surpasses previous SOTA models with a similar model structure on both MNLI-matched and SQuAD v2.0 evaluation set by more than 1.2% in accuracy and 1.3% in F1, respectively. We also train DeBERTaV3 on the CC100 (Conneau et al. 2020) multilingual data using a similar setting as XLM-R (Conneau et al.] 2020) but with only a third of the training passes. We denote the model as 3DeBERRTa3 buse- Under the cross-lingual transfer setting, mDeBBERTaV3 3haase achieves a 79.8% average accuracy score on the XNLI (Conneau et al. 2018) task, which outperforms XLM-Rrase and mT (Xue et al.] [021] by 3.6% and 4.4%, respectively. This makes mDeBERTaV3 the best model among multi-lingual models with a similar model structure. All these results strongly demonstrate the efficiency of DeBERTaV3 models and set a good base for future exploration towards more efficient PLMs.
BACKGROUND
2.1 TRANSFORMER
Transformer--aasee language model is composed of L. stacked Transformer blocks (Vaswani et al. 2017). Each block contains multi-head self-attention layer followed by a fully connected positional feed-forward network. The standard self-attention mechanism lacks a natural way to encode word position information. Thus, existing approaches add positional bias to each input word embedding so that each input word is represented by a vector whose value depends on both its content and position. The positional bias can be implemented using absolute position embedding (Vaswani et al. 2017; Brown et al. 2020 Devlin et al.) 2019) or relative position embedding (Huang et al. [2018] Yang et al. al.[[220 2019). Several studies have shown that relative position representations are more effective
2

---

Published as a conference paper at ICLR 2023
for natural language understanding and generation tasks (Dai al.
2019 Shaw et al. 2018 He et al.
2020
2.2 DEBERTA
DeBERTa improves BERT with two novel components: DA (Disentangled Attention) and enhanced mask decoder. Unlike existing approaches that use single vector to represent both the content and the position of each input word, the DA mechanism uses two separate vectors: one for the content and the other for the position. Meanwhile, the DA mechanism's 's attention weights among words are computed via disentangled matrices on both their contents and relative positions. Like BERT, DeBERTa is pre-trained using masked language modeling. The DA mechanism already considers the contents and relative positions of the context words, but not the absolute positions of these words, which in many cases crucial for the prediction. DeBERTa improve MLM by adding absolute position information of the context words at the MLM decoding layer.
enhanced mask decoder to
2.3 ELECTRA
2.3.1
MASKED LANGUAGE MODEL.
Lcrge-scale Transformer-based PLMs are typically pre-trained on large amounts of text to learn contextual word representations using a self-supervision objective, known as MLM (Devlin et al. 2019). Specifically, given a sequence X = (2z). we corrupt it into X by masking 15% of its tokens random and then train a language model parameterized by reconstruct X by predicting the masked tokens å conditioned X:
max log pa X X max Σ log po xz x 8EC
where C the index set of the masked tokens in the sequence. The authors BERT propose to keep 10% of the masked tokens unchanged, another 10% replaced with randomly picked tokens and the rest replaced with the [MASK] token.
2.3.2 REPLACED TOKEN DETECTION
Unlike BERT, which uses only one transformer encoder and trained with MLM, ELECTRA was trained with transformer encoders in GAN style. One is called generator trained with MLM; the other is called discriminator trained with a token-level binary classifier. The generator is used to generate ambiguous tokens to replace masked tokens in the input sequence. Then the modified input sequence is fed to the discriminator. The binary classifier in the discriminator needs to determine if corresponding token either an original token token replaced by the generator. We use 0g and O to represent the parameters of the generator and the discriminator, respectively. The training objective in the discriminator is called RTD (Replaced Token Detection). The loss function of the generator can be written as,
LMLM = E - log Pea (i..o= = IzJ xxl
(2)
sEC
where XG is the input to the generator by randomly masking 15% tokens in X.
The input sequence of the discriminator is constructed by replacing masked tokens with new tokens sampled according to the output probability from the generator:
i Pea ( "5..=x1(aG))
iE C
(3)
Tta
i $ C
The loss function of the discriminator is written as,
LRTD = E - Σ log pao (1 (1 = I:)) 1x),)
(4)
3

---

Published as a conference paper at ICLR 2023
where 1(-) is the indicator function and Xp is the input to the discriminator constructed via equation 3 In ELECTRA, LMLM and LRTD are optimized jointly, L = LMLM + ALRTD, where A is the weight of the discriminator loss LETD, D. which set to 50 in ELECTRA.
DEBERTAV3
This section describes DeBERTaV3, which improves DeBERTa by using the RTD training loss of Clark al. 2020) and a new weight-sharing method.
3.1 DEBERTA WITH RTD
Since RTD in ELECTRA and the disentangled attention mechanism in DeBERTa have proven to be sample-efficient for pre-training, we propose a new version of DeBERTa, referred to as DeBERTaV. V3, by replacing the MLM objective used in DeBERTa with the RTD objective combine the strengths of the latter.
In this implementation, Wikipedia and the bookcorpus (Zhu et al. 2015) used training data, following the base model configuration of Devlin et al. (2019). The generator is the same width as the discriminator but is half the depth. The batch size set 2048, and the model is trained for 125,00 25,000 steps with learning rate of 5e-4 and warmup steps of 10,000. Following Clark et al. (2020), we use À= 50 with the same optimization hyperparameters. We validate the effectiveness of DeBERTaV3 on two representative NLU tasks, i.e., MNNI and SQuAD v2.0. The results, presented in ! of Table 2 show that DeBERTaV3 significantly outperforms DeBERTa, i.e., +2.5% on the MNLI-m accuracy and +3.8% on the SQuAD v2.0 F1.
In the next two subsections, we will show that the performance of DeBERTaV3 can be further improved by replacing token Embedding Sharing (ES) used for RTD, originally proposed in Clark iet al. (2020), by a new Gradient- Diseett-Diett Embedding Sharing (GDES) method. We start with an analysis of ES in Section 3.2
3.2 TOKEN EMBEDDING SHARING IN ELECTRA
To pre-train ELECTRA,
generator and discriminator that share token embeddings, as (a). This method, called Embedding Sharing (ES), allows the generator to provide
shown in Figure I informative inputs for the discriminator and reduces the number of parameters to learn. However, it also creates a multitask learning problem, where the generator's and the discriminator's objectives interfere with each other and slow down the training convergence.
Let and JE be the token embeddings and their gradients, respectively. In each training step of ELECTRA, we compute gE by paokppropggating. the errors from both the generator's Masked Language Modeling (MLM) loss and the discriminator's ' Replaced Token Detection (RTD) loss, as @Larr + Sung This equation means that the token embeddings are updated by balancing tug-of-war procedure (Hadsell et al., 2020). The procedure can eventually converge if we control the update speed carefully (c.g., using a small learning rate or gradient clipping). but can very inefficient if the two tasks have different optimal directions for the embeddings. This is indeed the case for MLM and RTD, because they have opposite effects on the token embeddings. MLM encourages the embeddings of semantically similar tokens to be close to each other, while RTD tries to separate them to make the classification easier.
gE 2E
the gradients from the two tasks, which can be
To verify our hypothesis, we implement a variant of ELECTRA that does not share token embeddings between the generator and the discriminator, which we refer to as No Embedding Sharing (NES), as illustrated in Figure (b). In NES, we update the generator and the discriminator alternately each training step. First, run the generator to generate inputs for the discriminator, and then update the generator's parameters, including its token embeddings EG. by back- pook-propgggting the MLM loss. Next, the discriminator using the inputs from the generator and update the discriminator's parameters, including its token embeddings Ep. by back- packppropgggting the RTD loss.
We compare ES and NES on three aspects: convergence speed, quality of the token embeddings. and performance on downstream Natural Language Understanding (NLU) tasks. Figure shows that NES converges faster than ES, expected, because avoids the confficting gradients between the two
4

---

Published as a conference paper at ICLR 2023
X,
R
8,
6G
0/2
2g
0p
0/1
%
90 0/1
Baachams kackkward with
Backward with
fycx Algro
(a)
(b)
(c)
Figure I: Illustaaiion of different embedding sharing methods. (a) ES: E, 0g and 0D will be jointly
updated in single backward pass with regards to LMEM ALTDD (b) NES: Eg and f will first be updated via the backward pass with regards to LMLM. then En and On will be updated via the backward pass with regards ALRTD (c) GDES: Eg and 0g will first be updated in the backward pass with regards to LMM, then Es and 0D will be updated via the backward pass with regards to ALRDD and Eg sg is the stop gradient operator that prevents the discriminator from updating Eg
Table 1: Average cosine similarity of word beddings of the generator and the discrimina- tor with different embedding sharing methods.
Table 2: Fine-tuning results MNLI and SQuAD v2.0 tasks of base models trained with different embedding sharing methods.
Word Emhedding Sharing
Eg
Ep
Ea
MNLI-m/mm
SQuAD v2.0
Model
Acc
FI/EM
① ES
0.02
0.02
NES
0.45
0.02
BERT base
84.3/84.7
76.3/73.7
3 GDES
0.45
0.29
0.02
ELECTRA
85.8/-
+/-
DcBERTab
86.3/86.2
82.5/79.3
DeBERTa-RTDos
ES
88.8/88.4
86.3/83.5
2 NES
88.3/87.9
85,3/82.7
GDES
89.3/89.0
87.2/84.5
downstream tasks. We then measure the average cosine similarity scores of the token embeddings [ shows that NES produces two distinct embedding models, with EG being more semantically coherent than Ep. This confirms our hypothesis. However, the embeddings learned by NES do not lead to any significant improvement two representative downstream NLU tasks (i.e., MNLI and SQuAD v2.0), as shown in T This result supports the argument of Clark et al. (2200) that ES has the advantage of making the discriminator benefit from the generator embeddings, addition to saving parameters. In the next subsection, we propose new embedding sharing method that combines the strengths of ES and NES, while avoiding their drawbacks.
3.3 GRADIENT- DISNIIDIIEENNGEEDD EMBEDDING SHARING
We propose the Gradient- Drraieet-Diisentanee Disentangled Embedding Sharing (GDES) method to overcome the draw- backs of ES and NES while retaining their advantages. As shown in Figure[(c) GDES shares the token embeddings between the generator and the discriminator, which enables the two models to learn from the same vocabulary and leverage the rich semantic information encoded in the embeddings. However, unlike ES, GDES does not allow the RTD loss to affect the gradients of the generator, thus avoiding the interference and inefficiency caused by the conficctngg objectives. Instead, GDES only updates the generator embeddings with the MLM loss, which ensures the consistency and coherence of the generator output. As a result, GDES can achieve the same converging speed as NES, but without sacrificing the quality of the embeddings.
To implement GDES, we re-parameterize the discriminator embeddings as ED sg( Ea) + Es. where the stop gradient operator sg prevents the gradients from flowing through the generator embeddings Eg and only updates the residual embeddings Es. We initialize Eλ as a zero matrix each iteration, we first generate the inputs for
and train the model following the NES procedure. the discriminator using the generator and update both EG and ED with the MLM loss. Then, we run the discriminator on the generated inputs and update En with the RTD loss, but only through Es. After training, we add EA to EG and save the resulting matrix as ED for the discriminator.
We sample 10% of the word pieces from the vocabulary and calculate the average oosine similarity of every pair of two word pieces.
5

---

Published as a conference paper at ICLR 2023
We conduct extensive experiments to evaluate the effectiveness of GDES compared to ES and NES. We measure the converging speed, the quality of the token embeddings. and the per- formance downstream tasks. The results in Figure 2 Tables and 2 demonstrate that GDES outperforms both ES and NES in various as- pects. First, Figure 2 reveals that GDES con- verges faster than ES and matches the efficiency of NES. As GDES, ES and NES only differ at the way of embedding sharing, the computation cost of each step is the same. Second, Table indicates that GDES produces two distinctive token embedding matrices, with the generator embeddings having higher average similarity scores than the discriminator embeddings. However, the difference is smaller than that in NES, suggesting that GDES preserves more semantic information in the discriminator embeddings through the partial weight sharing. Third, Table 2 shows that after fine-tuning, the model pre-trained with GDES achieves the best performance on two downstream tasks, MNLI and SQuAD v2.0. These results confirm that GDES an effective weight-sharing method for language model pre-trained with MLM and RTD.
ES
GDES
6.P00PP
3x 10"
2x
20000
Steps
Figure 2: MLM training loss of the generator with different word embedding sharing methods.
EXPERIMENT
4.1 MAIN RESULTS ON NLU TASKS
To further verify the effectiveness of those technologies, we combine RTD, GDES and DA (Disentan- gled Attention) train models of different sizes(i.e., large, base and small) using standard pre-training settings. Since all of our experiments are modified based on DeBERTa code base and follow most of the settings of DeBERTa, we denote the new models as DeBERTa V lurge. DeBERTaV buse and DeBERTaV3 The discriminator part of DeBERTa V3lagg and DeBERTaV3 DeBERTa and DeBERTabse: respectively. The discriminator DeBERTaV3 has the same width and attention heads DeBERTa and half the depth of DeBERTa Tatass i.e., 6 laaerrs with 768 hidden size and 12 attention heads. The generator of DeBERTaV3 has the same width as the discriminator and half the depth of the discriminator. We train those models with 160GB data, which is the same as DeBERTaV2 and RoBERTa, and use the same SentencePiece (Kudo 2018 Sennrich et al. 2016) vocabulary DeBERTaV2 (He et al. 2020) which contains 1 128.000 tokens. All the models are trained for 500,000 steps with a batch size of 8192 and warming up steps of 10,000, The learning rate for base and small model 5e-4, while the learning rate for large model is 3e-4. Following the DeBERTa setting, we use the Adam (Loshchilov & Hutter] 2018 optimizer which fixed version of Adam (Kingma & Ba 2014) with weight decay, and set 31 0.9, B2 0.98 for the optimizer. After pre-training, the discriminators of those models are used for downstream task fine-tuning following the same paradigm as Transformer PLMs, such as BERT, RoBERTa, ELECTRA, and DeBERTa. We provide more details on the hyper parameters of pre-training and fine-tuning in the Appendix.
the same as
4.1.1 PERFORMANCE ON LARGE MODELS
Following previous studies on PLMs, we first evaluate our model on the eight NLU tasks in GLUE (Wang et al. 2019b), which are the most representative sentence classification tasks. We fine tune the pre-trained models on those tasks by plugging a classification head on top of the hidden states of the [CLS] token at the last layer. We summarize the results in Table where DeBERTaV3 compared with previous Transformer-base PLMs similar structures (i.e., 24 layers with hidden size of 1024), including BERT, RoBERTa, XLNet (Yang et al. 2019), ALBERT (Lan al. 2019), ELECTRA and DeBERTa. Compared to previous SOTA models, DeBERTaV3 performs consistently comparable or mostly better across all the tasks. Meanwhile, DeBERTaV3 outperforms XLNet in seven out of eight tasks. In terms of average GLUE score, DeBERTaV3 outperforms other SOTA PLMMs with large margin (> 1.3%). Particularly, compared with previous best numbers, there big

---

Published as a conference paper at ICLR 2023
jumps the performance of low resource tasks (i.e., RTE (+4.4%), CoLA (+4.8%)) This indicates DeBERTaV3 is more data efficient and has a better generalization performance. We also note that the improvements SST-2, STS-B, and MRPC are relatively small (< 0.3%). We conjecture this is due to the performance on those tasks is close to be saturated, and thus even small but consistent improvements on them are valuable.
Table 3: Comparison results on the GLUE development set.
CoLA
QQP
MNLI-n/mm Acc
SST-2
STS-B
QNLI
RTE
MRPC
Avg.
Model
Mee
Acc
Acc
Corr
Acc
Acc
Acc
#Train
8.5k
364k
393k
67k
7k
108k
2.5k
3.7k
BERT large
60.6
91.3
86.6/-
93.2
90.0
92.3
70.4
88.0
84.05
RoBERTalage Tatage
68.0
92.2
90.2/90.2
96.4
92.4
93.9
86.6
90.9
88.82
NeNee
69.0
92.3
90.8/90.8
97.0
92.5
94.9
85.9
90.8
89.15
ELECTRAuage
69.1
92.4
90.9/-
96,9
92.6
95.0
88.0
90.8
89,46
DeBERT DaBaRe
70.5
92.3
91.1/91.1
96.8
92.8
95.3
88.3
91.9
90.00
DeBERTaV V3rage
75.3
93.0
91.8/91.9
96.9
93.0
96.0
92.7
92.2
91.37
To further evaluate the model performance, in addition GLUE, DeBERTa is evaluated on three categories of representative NLU benchmarks: (1) Question Answering: SQuAD v2.0, RACE (Lai et al., 2017). ReCoRD (Zhang et al.) (0188), and SWAG (Zellers al., 2018); (2) Natural Language Inference: MNLI; and (3) NER: CoNLL-2003 (Sang & De Meulder (003). Among those tasks, RACE, SWAG, and MNLI are fine-tuned using a same way as sentence classification tasks. SQuAD v2.0, ReCoRD and NER are fine-tuned as sequence tagging tasks, where token classification head is plugged on top of the hidden states of each token at the last layer. For comparison, we include ALBERT sxlarge DeBERTaaage- DeBERTaj, and Megatron (Shoeybi et al. (201)) with three different model sizes, denoted as Megatrons36M: Megatron and Megatron3.9B. which are trained using the same dataset RoBERTa. Note that Megatrons3os, has similar model size as other models mentioned above
We summarize the results in Table Compared to the previous
Table 4: Results on MNLI in/out-domain, SQuAD v2.0, RACE, ReCoRD, SWAG, CoNLL 2003 NER development set. Note that missing results in literature are signified by "_"
MNLI-m/mm
SQuAD v2.0
RACE
ReCoRD FI/EM
SWAG
NER F1
Model
Acc
FI/EM
Acc
Acc
BERT large
86.6/-
81.8/79.0
72.0
86.6
92.8
ALBERT
86.5/-
84.9/81.8 8
75.2
RoBERTan Taaage
90.2/90.2
89.4/86.5
83.2
90.6/90.0
89.9
93.4
XLNettare
90.8/90.8
90.6/87.9
85.4
ELECTRABre 90.9/-
-/88.1
MegatronraM
89.7/90.0
88. 1884.88.8
83.0
DeBERTamne
91.1/91.1
90.7788.0
86.8
91.4/91.0
90.8
93.8
DeBERTa Vaagg
91.8/91.9
91.5/89.0
89.2
92.3/91.8
93.4
93,9
ALBERT sslarge
90.87-
90.2/87.4
86.5
Megatron
90.9/91.0
90.2/87.1
87.3
Megatron 9B
91.4/91.4
91.2/88.5
89.5
DeBERTai.s 91.7/91.9
92.2/89.7
90.8
94.5/94.0
92.3
SOTA PLMS with a similar model size (i.e., BERT, RoBERTa, XLNet, ALBERT Megatron and D DeBERTaV large shows superior performance all six tasks. We see big performance jump on RACE (+2.4%) and SWAG (+2.6%), which require the models to have non- trivial reasoning capability and common-sense knowledge (Lai et al. 2017 Zellers al.. 2018). We conjecture those improvements indicate DeBERTaV3 Iarge has better capability of reasoning and common sense knowledge. Although it is well proved to improve the model capacity by increasing the number of parameters (Raffel et al. 2020 Fedus et al. 202T), compared with larger models,
'The hidden dimension of AL salagge is times of DeBERTa and the computation cost is about times of DeBERTa.
²T5 (Raffel et al. 2020) has more parameters (11B) Raffel et al. 2020 only reported the test results of T5 which are comparable with models mentioned above.
7

---

Published as a conference paper at ICLR 2023
DeBERTa SIEERTa33rre large outperforms ALBERT xxlarge and Megatron by large margin all three tasks, as well as outperform Megatrons.38 on both MNLI and SQuAD v2.0. Compared with DeBERTaj, 1.55 which used to be the SOTA NLU models on GLUE and SuperGLUE leaderboards, DeBERTa V3 1arge is still on par with on MNLJ but outperforms on SWAG.
4.1.2 PERFORMANCE ON BASE AND SMALLER MODELS
We evaluate DeBERTa bases DeBERTaVas V3 and DeBERTaV:s ssmall on two representative tasks, i.c. MNLI and SQuAD v2.0, and summarize the results in Table 5 DeBERTa base consistently outperforms DeBERTabas and ELECTRABe by a larger margin than that in the Large models. For example, on MNLI-m, DeBERTaV3 obtains improvement of +1.8(90.69)) vs. 88.8%) over both DeBERTagase and ELECTRABase On SQuAD v2.0 in terms of the EM score, DeBERTaV V achieves an improvement of +4.9% (85.4% vs. 80.5%) over ELECTRApse and +2.3% (85.4% vs 83.1% 1%) over DeBERTanne Taaass.
Table 5: Results on MNLI in/out-domain (m/mm) and SQuAD v2.0 development set. TinyBERT.oo small (Jiao et al. (019), Minil.Mv2smal and Mvnil. models are pre-trained with knowledge distillation while BERT small. DeBERTaV3 and DeBERTaV3 rsmall are trained from scratch with MLM and RTD objective, respectively.
Vocabulary Size(K)
Backhone PPrrams(M)
MNLI-m/mm ACC
SQuAD v2.0 FI/EM
Model
Base models:11 layers,768 hidden size, heads
BERTbas base
30
86
84.3/84.7
76.3/73.7
RoBERTa Tabase
50
86
87.6/-
83.7/80.5
XLNebbuee
32
92
86.87-
-/80.2
ELECTRANas
30
86
88.87-
-/80.5
DeBERTabe Aiuse
50
100
88. 88.8/88.5
86.2/83.1
DeBERTaV V3ese
128
90.6/90.7
88.4/85.4
Small models:6 layers, 768 hidden size, heads
TinyBERT small
30
44
84.3/-
77.7/-
MiniLMvv.mal
30
44
87.0/-
81.6/
BERT 'mall
81.87-
73.2/-
DeBERTaV3nma
128
44
88.2/87.9
82.9/80.4
XSmall models:12 layers,384 hidden size,6 heads
MiniLMM2ssmall
22
86,9/-
82.3/
DeBERTaVa Vasmall
128
22
1881/88.3
84,8/82.0
Compared with smaller size models with similar model structures, DeBERTaV33.mm 3mall outperforms BERT small (Wang et al., 2020b) by a large margin on those two tasks (i.e., a 6.4% improvement on MNLI-m and a 9.7% Fl score improvement
SQuAD v2.0). Surprisingly, even though DeBERTaV3: 3 has only half the parameters of DeBERTaV3 better than DeBERTaV3..m has deeper layers which allows to extract better semantic features. Without knowledge distillation, DeBERTaV3 small outperforms MiniLMvv.u.. Mv2 (Wang et al.) 2020a) by 1.2% and 1.3% MNL.-m and SQuAD v2.0, respectively. DeBERTaV'S & outperforms MiniL.Mv2ssmall (Wang et al. 2020a) by 1.2% and 2.5% on MNLI-m and SQuAD v2.0, respectively. It's worth noting that, even though DeBERTaV V xsmall has only 1/4 backbone parameters of RoBERTapes and XLNelnaee the former significantly outperforms both models on these two representative tasks (i.e., 0.5% improvement on MNLI-m and 1.5% EM score improvement on SQuAD v2.0). This further demonstrates the efficiency of the DeBERTaV3 models.
performs on par or even these two tasks. We conjecture that this is due to DeBERTaV V3 X
4.2 MULTILINGUAL MODEL
As an important extension, extend DeBERTaV3 to multi-lingual. We train the multi-lingual model with the 2.5T CC100 multi-lingual dataset which is the same as XLM-R. We denote the model as mDeBERTa V We use the same SentencePiece vocabulary mT5 which has 250k tokens. The model structure is the same as our base model, i..., 768 hidden size, 12 layers, and 12 attention heads. It is worth noting that, unlike XL.M XLM-E, have not trained our model with any parallel data
8

---

Published as a conference paper at ICLR 2023
The pre- pre-training settings are similar to XLM-R, except that we only train the model with 500k steps instead of 1.5M steps.
Table Results on XNLI test set under the cross-lingual transfer and the translate-train-all settings.
Model
fr
de
tr'
vi
th
zh
Avg
Cross lingual transfer
XLM
83.2
76.7
77.7
74.0
72.7
74.1
72.7
68.7
68.6
72.9
68.9
72.5
65.6
58.2
62.4
70.7
mTS
84.7
79.
80.3
77.4
77.1
78.6
77.
72.8
73.3
742
73.2
74.1
70.8
69.4
68.3
75.4
XI.
85.8
79.7
80.7
78.7
77.5
79.6
78.
74.2
73.8
76.5
74.6
76.7
72.4
66.5
68.3
76.2
mDeBERRTa
88.2
82.6
84.4
82.7
82.3
82.4
80.8
79.5
78.5
78.1
76.4
79.5
75.9
73.9
72.4
79.8
Trunslate train all
XLM
84.5
80.1
81.3
79.3
78.6
79.4
77.5
75.2
75.6
78.3
75.7
78.3
72.1
69.2
67.7
76.9
82.0
77.9
79.1
77.7
78.
78.5
76.5
74.8
74.4
74.5
75.0
76.0
72.2
71.5
70.4
75.9
XLM-Raaa
85.4
81.4
82.2
80.4
79.7
78.0
77.3
79.7
77.9
80.2
76.
73.1
73.0
79.1
mDeBERTaV Vruse
88.9
84.4
85.3
84.8
84.0
84.5
83,2
82.0
81.6
82.0
79.8
82.6
79.3
77.3
73.6
82.2
As XNLI is one of the major benchmarks to measure multi-lingual model generalization performance, we evaluate the performance of mDeBERTaV3 on XNLI across 15 languages. Following previous multi-lingual PLMs, we report both the zero-shot cross-lingual transfer performance and the translate- train-all performance. Zero-shot cross-lingual transfer to fine-tune the model with English data only and evaluate it on moulti-lingual test sets, connslae-ttrin-aae is to fine-tune the model with English data and multi Manlti-iingul data translated from English data which provided together with the XNLI dataset( et al. 2018), and then evaluate the model on multi-lingual test sets. As shown in Table [6] mDeBERTa Vasse significantly outperforms previous SOTA model XLMM-rass on all languages under both settings. With regards to the average score, mDeBERTa V3EETVV3bma V obtains an improvement of 3..6% (79.8% v.s. 76.2%) compared with XLaeR in the cross-lingual transfer setting, as well achieves improvement of +3.1% (82.2% v.s. 79.1%) compared with XLM-R - under the translate-train setting. These results clearly show the effectiveness of DeBERTaV3 and the disentanglement is simultaneously valuable to multi-lingue pre-training.
All this clearly demonstrates the efficiency of the DeBERTaV3 models. The consistent improvements over a large range of the downstream tasks also show the huge value of improving pre-trained language models.
5 CONCLUSIONS
In this paper, we propose a novel pre-training paradigm for language models based on the combination of DeBERTa and ELECTRA. two state-of-the-art models that use relative position encoding and replaced token detection (RTD) respectively. We show that simply combining these two models leads to pre- pre-training instability and inefficiency. due critical interference issue between the generator and the discriminator in the RTD framework which is well known as the "tug-of-war" dynamics. To address this issue, introduce novel embedding sharing paradigm called GDES, which is the main innovation and contribution of this work. GDES allows the discriminator to leverage the semantic information encoded in the generator's 's embedding layer without interfering with the generator's ' gradients and thus improves the pre-training efficiency. GDES defines a new way of sharing information between the generator and the discriminator in the RTD framework, which can be easily applied to other RTD-based language models. We conduct extensive analysis and experiments to compare GDES with other alternatives verify its effectiveness.
Furthermore, we show that DeBERTaV3 V3 with GDES achieves significant improvements over previous tht (SOTA) models on various NLU tasks that cover different aspects of natural language understanding. For example, DeBERTaV3S 3 surpasses other models with similar architecture by more than 1.37% on the GLUE average score and mDeBERTaVas 33 beats XLM-Rease by 3.6% on the cross lingual transfer accuracy of the XNLI task. These results highlight the effectiveness of all the DeBERTaV3 models and establish DeBERTaV3 as the new SOTA pre-trained language models (PLMs) for natural language understanding different model scales, i.e., Large, Base, Small and XSmall. Meanwhile, this work clearly shows huge potential to further improve model's parameter efficiency and provide some direction for future studies of far more parameter-efficient pre-trained language models.
'we expect to have hetter model by continually pre-training parallel data and will release new model when available

---

Published as a conference paper at ICLR 2023
REFERENCES
Roy Bar-Haim, Ido Dagan, Bill Dolan, Lisa Ferro, and Danilo Giampiccolo. The second PASCAL recognising textual entailment challenge. In Proceedings of the Second PASCAL Challenges Workshop on Recognising Textual Entailment, 01 2006.
Luisa Bentivogli, Ido Dagan, Hoa Trang Dang, Danilo Giampiccolo, and Bernardo Magnini. The fifth pascal recognizing textual entailment challenge. In In Proc Text Analysis Conference ('AAC'09, 2009.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005. 14165, 2020.
Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. Semeval-2017 task 1: Semantic textual smmilaity-multilinguus and cross-lingual focused evaluation. arXiv preprint arXiv: 1708,00055 2017.
Zewen Chi, Shaohan Huang, Li Dong, Shuming Ma, Saksham Singhal, Payal Bajaj, Xia Song, and Furu Wei. Xlm-e: Cross-lingual language model pre-training via electra. arXiv preprint arXiv:2/06. 16138, 2021.
Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. ELECTRA: Pre-training text encoders as discriminators rather than generators. In ICLR, 2020.
Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. Xnli: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2475-2485, 2018.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Fran- cisco Guzmán, Edouard Grave, Myle Out, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross-lingual representation learning scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 1440-8451, 2020.
Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textual entailment challenge. Proceedings of the First International Conference on Machine Learning Chal- lenges: Evaluating Predictive Uncertainty Visual Object Classification, and Recognizing Textual Entailment, MLCW'05, Berlin, Heidelberg, 2006.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G Carbonell, Quoc Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 2978-2988, 2019.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep
bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter the Association for Computational Linguisties: Human Language Technologies, Volume I (Long and Short Papers), pp. 4171-4186, 2019.
William B Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (ZWP2005). 2005.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. arXiv preprint arXiv:2101.03961 2021.
Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al. Codebert: A pre-trained model for programming and natural languages. arXiv preprint XiX::2002.08155, 2020.
Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. The third PASCAL recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL Workshop Textual Entailment and Paraphrasing, pp. 1-9, Prague. June 2007. Association for Computational Linguistics. URL https: www. a web. orgg ant ogy /wwp:///0014400
10

---

Published as a conference paper at ICLR 2023
Raia Hadsell,, Dushyant Rao, Andrei A Rusu, and Razvan Pascanu. Embracing change: Continual learning in deep neural networks. Trends Cognitive Sciences, 24(12):1028-1040, 2020.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Recoding-enhancee bert with disentangled attention. In International Conference on Learning Representations, 2020.
Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Ian Simon, Curtis Hawthomne, Noam Shazeer, Andrew M Dai, Matthew Hoffman, Monica Dinculescu, and Douglas Eck. Music transformer: Generating music with long-term structure. 2018.
Xiaoqi Jiao, Yichun Yin, Lifeng Shang. Xin Jiang, Xiao Chen, Linlin Li, Fang Wang. and Qun Liu. Tinybert: Distilling bert for natural language understanding. arXiv preprint arXiv: /909, 10351, 2019.
Kamal Raj Kanakarajan, Bhuvana Kundumani, and Malaikannan Sankarasubbu. Small-bench nlp: Benchmark for small single gpu trained models in natural language processing. ArXiv, abs/2109.10847 2021.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Taku Kudo. Subword regularization: Improving neural network translation models with multiple sub- word candidates. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers), pp. 66-75, 2018.
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. Race: Large-scale reading comprehension dataset from examinations. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 785-794, 2017.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations. International Conference on Learning Representations, 2019.
Hector Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. In Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning. 2012.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint
2019.
Ilya Loshchilov and Frank Hutter. Fixing weight decay regularization in adam. 2018.
Yu Meng, Chenyan Xiong, Payal Bajaj, Saurabh Tiwary, Paul Bennett, Jiawei Han, and Xia Song. Coco-Im: Correcting and contrasting text sequences for language model pretraining. arXiv preprint arXiv:2102.08473 2021.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl Blog, 188), 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020. URL
http:://mir wwt.popers..a.......oo.o.. wt1p//22112222222000
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference Empirical Methods in Natural Language Processing, November 2016.
Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't know: Unanswerable questions for squad. In Proceedings of the 56th Anmual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 784-789, 2018.
11

---

Published as a conference paper at ICLR 2023
Erik Tjong Kim Sang and Fien De Meulder. Introduction to the conll-2003 shared task: Language- independent named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pp. 142-147, 2003.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pp. 1725- Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10. 118655//1/P11-1-1 1162. URL htttps://cclanthol..ot org/P16-1162
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pp. 464-468, 2018.
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-Im: Training multi-billion parameter language models using gpu model parallelism. arXiv preprint 1909, 08053,, 2019.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods natural language processing, pp. 1631-1642, 2013.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-50088 2017.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. In Advances in neural information processing systems, pp. 3266-3280, 2019a.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman, Glue: A muulti-takk benchmark and analysis platform for natural language understanding. In 7th Interna- tional Conference on Learning Representations, ICLR 2019, 2019b.
Wenhui Wang, Hangbo Bao, Shaohan Huang, Li Dong, and Furu Wei. Minilmv2: Multi-head self-attention relation distillation for compressing pretrained transformers. arXiv preprint arXiv:2012.15828. 2020a.
Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang. and Ming Zhou. Minilm: Deep self- attention distillation for task-agnostic compression of pre-trained transformers. arXiv preprint arXiv:2002. 10957, 2020b.
Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. Neural network acceptability judgments. arXiv preprint 1805. 12471, 2018
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo- gies, Volume 1 (Long Papers), pp. 1112-1122, Association for Computational Linguisties, 2018. org/ httg//anttooogy//1-111000
URL /
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami AI-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mt5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics; Human Language Technologies, pp. 483-498, 2021.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in neural information processing systems, pp. 5754-5764, 2019.
12

---

Published as a conference paper at ICLR 2023
Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. Swag: A large-scale adversarial dataset for grounded commonsense inference. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 93-104, 2018.
Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. ReCoRD: Bridging the gap between human and machine commonsense reading comprehension. arXiv preprint 1810. 12885, 2018.
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international conference on computer vision, pp. 19-27, 2015.
13

---

Published as a conference paper at ICLR 2023
A APPENDIX
A.1 DATASET
Table 7: Summary information of the NLP application benchmarks.
Corpus
Task
#Train
#Dev
#Test
#Label
Metries
General I. Language Understanding Eval Avaluation (GLUE)
CoLA
Acceptability
8.5k
lk
lk
2
Matthews corr
SST
Sentiment
67k
872
1.8k
2
Accuracy
MNLI
NLA
393k
20k
20k
Accuracy
RTE
NLI
2.5k
276
3k
2
Accuracy
WNLI
NLI
634
71
146
2
Accuracy
QQP
Paraphrase
364k
40k
391k
Accuracy/F1
MRPC
Paraphrase
3.7k
408
1.7k
2
Accuracy/Fi
QNLI
QA/NII
108k
5.7k
5.7k
2
Accuracy
STS-B
Similarity
1.5k
1.4k
Pearson/Spearman corr
Question Answering
SQUAD vI.I
MRC
87.6k
10.5k
9.5k
Exact Match (EM)/FI
SQuAD v2.0
MRC
130.3k
11.9k
8.9k
Exact Match (EMMFL
ReCoRD
MRC
10Tk
10k
10k
Exact Match (EMWFI
RACE
MRC
87,866
4.887
4,934
4
Accuracy
SWAG
Muliiple choice
73.5k
20k
20k
Accuracy
Token
Classification
CoNLL 2003
NER
14,987
3,466
3,684
F1
Multi-lingual Natural Language Inference(XNLI)
XNLlgus-innsal
NLI
393k
37k
75k
Accuracy
XNLlrranstte train
NLI
5.9M
37k
75k
Accuracy
GLUE The General Language Understanding Evaluation (GLUE) benchmark is collection of nine natural language understanding (NLU) tasks. As shown in Table 7 it includes question answer-
ing (Rajpurkar et al. 2016), linguistic acceptability (Warstadt al. 2018). ) sentiment analysis (Socher
et al. (013), text similarity (Cer et al.) 2017), paraphrase detection (Dolan & Brockett) (2005), and al.) 2007 Bentivogli et al.) 2009 Levesque et al. 2012) Williams et al. 2018) The diversity of the tasks
natural language inference (NLI) (Dagan et al. (006)] Bar-Haim et al. 2006] [iampiccolo
makes GLUE very suitable for evaluating the generalization and robustness of NLU models.
RACE is a large-scale machine reading comprehension dataset collected from English examinations in China designed for middle school and high school students (Lai et al. 2017).
SQuAD v1.1/v2.0 is the Stanford Question Answering Dataset (SQuAD) v1.1 and v2.0 (Rajpurkar al., 22016] 2018). two popular machine reading comprehension benchmarks from approximately 500 Wikipedia articles with questions and answers obtained by crowdsourcing. The SQuAD v2.0
dataset includes unanswerable questions about the same paragraphs.
SWAG is large-scale adversarial dataset for the task of grounded commonsense inference, which unifies natural language inference and physically grounded reasoning (Zellers et al. 2018). SWAG consists of 113k multiple choice questions about grounded situations.
CoNLL 2003 (Sang & De Meulder 2003) an English dataset consisting of text from wide variety of sources. It has 4 types of named entities.
XNLI (Conneau al.) 2018) comes with ground truth dev and test sets in 15 languages, and ground-truth English training set which is same as MNLI training set. The training set has been machine-translated to the remaining 14 languages, providing synthetic training data for these languages well.
A.2 PRE-TRAINING DATASET
For DeBERTaV3 pre- pre-training, we use same data as RoBERTa and DeBERTai,nn which a combi- nation of Wikipedia, Bookcorpus, CCNews, Stories and Wetn The multi-lingual version
14

---

Published as a conference paper at ICLR 2023
of DeBERTaV3 is trained with 2.5TB CC100 data which is the same XLM-R. For pre-training, we also sample 5% the training data as the validation set to monitor the training process. Table compares datasets used in different pre-trained models.
Table 8: Comparison of the pre-training data.
Model
Wiki+Book OpenWetText Stories(CC-News CC-News
Gigad
Clue TWWee/Coomon Common Craw Craw1|CC100
16GB
38GB
31GB
76GB
16GB
19GB
110GB
2.5TB
BERT
XLNet
v
ELECTRA
v
v
RoBERTa
V
4
d
DeBERTa
V
DeBERTaiss
/
/
V
DeBERTaV3
v
v
V
mDeBFRTaVa Vaus
A.3 GENERALITY OF GDES
To demonstrate the generality GDES as enhancement for RTD, we applied the ELECTRA setting, which we reproduced with base model of 12 layers for the discriminator and 6 layers for the generator, both with the same hidden size. We used wikippedia-bookcrrpus data to pre-train the model from scratch with learning rate of 5e-4 and batch size of 2k for 125k steps. We then evaluated the pre-trained models on MNLI and SQuAD v2.0 using the same setting as in Table The results in Table 9 show that GDES can also improve the training efficiency of RTD in this setting, consistent with the findings in Table
Table 9:
Fine-tuning results on MNLI and SQuAD v2.0 tasks of base ELECTRA models trained with different embedding sharing methods.
MNLI-m/mm Acc
SQuAD v2.0 FI/EM
Model
ELECTRAPE
85.8/-
ELECTRAMa
!
Reimplemented (ES)
87.9/87.4
85.0/82.3
NES
86.3/85.6
81.7/78.9
GDES
88.3/87.8
85.9/83.1
A.4 IMPLEMENTATION DETAILS
Our pre-training almost follows the same setting DeBERTa (He al. 2020). The generators are trained with MLM where we randomly replace 15% input tokens with [MASK] tokens. The discriminator is trained with RTD which is the same ELECTRA. The experiments in Section 3 are trained using Wikipedia English data and Bookcorpus data with a batch size of 2k for 125k steps. The experiments in Section are trained using data listed in Table 8 with batch size of 8k for 500k steps. We list the detailed hyper parameters of pre-training in Table TO For pre-training, we use Adam (Kingma & Ba 2014) as the optimizer with weight decay (Loshchilov Hutter 2018). For fine-tume tneettunng, we use Adam (Kingma & Baj 2014) as the optimizer for a fair comparison. For fine-tuning, we train each task with pyper-paramettr search procedure, each run taking about 1-2 hours on a DGX-2 node. All the hyper-parameters are presented in Table 11 The model selection is based on the performance on the task-specific development sets.
Our code is implemented based on DeBERTa (He et al. 2020 and ELECTRA (Clark et al. 2020
"https//githu..com/iii http::///ftthh.ommmmmiccsf/////f hhanss//ggthhhhcmmmmogggrerrrree
15

---

Published as a conference paper at ICLR 2023
Table 10: Hyper-parameters for pre-training DeBERTaV3,
Hyper-parametee
DeBERTaV3srg
DeBERTaV3 3cBERTaV3.mat mDcBERTaV Brow
DeBERTaVS 3
Branzomslyii
Number of Layers
24
12
6
12
12
Hidden size
1024
768
768
FNN inner hidden size
4096
3072
3072
3072
3072
Attention Hcads
12
12
12
Attention Head size
64
64
64
Dropout
0.1
0.1
0.1
0.1
0.1
Warmup Steps
10k
10k
10k
10k
10k
Learning Rates
3e-4
6e-4
6e-4
6e-4
5e-4
Batch Size
8k
8k
8k
8k
2k
Weight Decay
0.01
0.01
0.01
0.01
0.01
Max Steps
500k
500k
500k
500k
125k
Learning Rate Decay
Linear
Linear
Linear
Linear
Lincar
Adam
lc-6
le-6
Ic-6
le-6
le-6
Adam 81
0.9
0.9
0.9
0.9
Adam 82
0.98
0.98
0.98
0.98
0.999
Gradient Clipping
1.0
1.0
1.0
1.0
1.0
Table 11:
for fine-tuning DeBERTaV3 on down-streaming tasks.
Syper-paramette
DeBERTaV3ums
DeBERTaV3....
DeBERTaV3....
mDDeBERiaV....
Dropout of task layer
10.0.15.0.3
1..0.100.15)
(0.0.1.0.15]
(0.0.1.0.15)
Warmup Steps
(50, (00_500_1000)
[50,100 150,1005500,10000 1000)
(50, 150,100,500,100 1000)
150,100,500,1000
Learning Rates
(5e-6,
9e-6, le-5)
(1.5e-5.2e-5,
2.5e-5, 3e-5]
(,25.,,
4e-5)
(1.5e-5,2e-5.
2.5e-5, 3e-5)
Batch Size
(16,32,64)
(16,32,48,64)
(16.32,48,64)
(16,32,48.64]
Weight Decay
0.01
0.01
0.01
0.01
Maximun Trannnn Epochs
10
10
10
10
Learning Rate Decay
Linear
Linear
Linear
Linear
Adam
Ic-6
Ic-6
lc-6
Ic-6
Adam 8
0.9
0.9
0.9
0.9
0.999
0.999
0.999
Gradient Clipping
1.0
1.0
1.0
1.0
16


Reviews:
[{"review_id": "vK9T_63vHNu", "reviewer": "Reviewer_9Bb4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper trained a new DeBERTa model (named NewModel) which incorporates two additional ingredients: 1) The ELECTRA-style training; 2) The token embedding sharing mechanisms, where the authors proposed gradient-disentangled embedding sharing. The author empirically demonstrated that the existing token sharing is not efficient and hurts model performance, and then show that by not propagating RTD loss to generator, the training can be much better. 3)The author also pre-trained a multi-lingual model and observe larger improvement over strong baselines compared to English models.", "summary_of_the_review": "I recommend acceptance of this paper, for the following reasons:\n1) The gradient-disentangled embedding sharing idea is novel and well supported by three aspects of evidences.\n2) The improvement of the model performance is solid and significant, both from the English model and multilingual model.\n3) The paper has excellent quality and clarity.\n4) While ELECTRA is a known idea, the paper does not merely using the method as it is.", "strength_and_weaknesses": "The strength of the paper is the following:\n1) The goal of the paper is clear. The authors wanted to improve over existing DeBERTa model, with new training approach.\n2) The idea of \"gradient-disentangle\" is solid and novel. The author drew the conclusion based on 3 evidences: the average cosine similarity of token embeddings, the convergence of MLM loss, and the downstream task comparison.\n3) The experiment results demonstrating the effectiveness on NLU tasks are comprehensive and solid.\n\nThe weakness of the paper:\n1) The name \"NewModel\" could be improved -- this name appears to have nothing to do the DeBERTa, and does not accommodate model backbone at all, so it will be confusing to keep track of.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "qII838XrJQ", "reviewer": "Reviewer_2Wnk", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to enhance the DeBERTa model with the more sample-efficient replaced token detection (RTD) pre-training task. \nTo further improve the training efficiency and performance of the vanilla embedding-sharing strategy (ELECTRA) for the DeBERTa + RTD setting, a new gradient-entangled embedding-sharing (GDES) alternative is leveraged. Experiments across different model sizes and downstream tasks demonstrate the effectiveness of the combination of DeBERTa and RTD and the newly introduced GDES strategy.", "summary_of_the_review": "The newly proposed pre-trained language model achieves very competitive performance with the existing RTD pre-training task and a newly presented embedding-sharing strategy.", "strength_and_weaknesses": "Strengths:\n1. A new pre-trained language model is proposed, which can combine the merits of DeBERTa and ELECTRA by enhancing DeBERTa with the RTD pre-training task and an extra GDES strategy to improve the vanilla embedding-sharing strategy from ELECTRA.\n2. A few strong pre-trained backbone alternatives will be released that can achieve very competitive performance across different model sizes, downstream tasks, and settings.\n\nWeaknesses: \n1. I am not convinced that the newly introduced GDES can improve the training efficiency of the pre-trained models. It has been well-proven that RTD is a sample-efficient pre-training task with thorough comparisons of pre-train FLOPs to other pre-trained language models. However, this paper only utilizes the MLM loss curve of the generator to demonstrate the efficiency of the whole pre-training procedure of both the generator and the discriminator. In real applications, we mainly utilize the larger discriminator for downstream task fine-tuning.\n2. The technical contribution is limited. RTD is borrowed from ELECTRA, and I am not sure whether the GDES strategy can generalize well to other MLM + RTD setting (e.g., ELECTRA), except for DeBERTa + RTD.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "egqcbbBSySN", "reviewer": "Reviewer_7CEr", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The current research piece introduce combines existing pretrained language models as DeBERTa and ELECTRA into a single model.\nIn addition, authors introduce GDES, a technique that the shared embeddings from the generator and discriminative models interfere with each other by avoiding propagating the gradients due to the discriminative loss. \nThe model is evaluated in a series of standard NLU dataset, particularly GLUE benchmark, plus other dataset for specific tasks such as QA, NLI and NER varying models sizes showing the effectiveness of their proposed method. \n\n\n", "summary_of_the_review": "Despite the jump in performance shown by the experimentation the paper do not provide enough novelty.. \n", "strength_and_weaknesses": "Strength \nThe paper is sound and make sense.\nDeBERTa and ELECTRA are both top level PLMs, but with significantly different training regimes so its combination makes sense to lead to further improvements.\n\nWeaknesses\nNovelty is weak. \nCombining (any) models might the actual contribution here. Picking two SOTA models and combining them is not clear to have enough novelty.\nThe GDES idea is interesting, but doing a thoughtful ablation study with any pairs of PLMs instead will prove authors point. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "GTzvLZUNpZ", "reviewer": "Reviewer_62yM", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper shows that embedding sharing in ELECTRA hurts the training dynamics and proposes a gradient-disentangled embedding sharing method. This method achieves better results in downstream tasks than both embedding sharing and separate embeddings. The paper then uses this ELECTRA objective on a DeBERTa model architecture, showing best in class performance for models of similar size and often surpassing bigger models across many NLU tasks. They also show that this outperformance extends to smaller model settings and to multilingual models.\n\nThe main technical contribution of the paper is in identifying the ELECTRA gradient-sharing issue and in proposing an effective scheme to replace it. While the tests of the combination of ELECTRA and DeBERTa done in the rest of the paper presents less novelty, the high performance of these models for these size means they could be useful to the wider community and so this empirical contribution should not be disregarded.\n", "summary_of_the_review": "The GDES contribution is useful and the final model in this paper achieves very good size/performance tradeoffs which will be useful for the wider community. Despite limited novelty and some minor issues highlighted in the Cons, I think it is well executed and slightly above the threshold.", "strength_and_weaknesses": "Pros:\n- The embedding sharing issue and its fix is discussed clearly and well supported by the author's experiments.\n- The results of \"NewModel\" are empirically quite strong.\n- The paper's writing and structure is quite solid. See below for minor typos/comments. \n- The authors have done a good job at providing experimental details.\n\nCons:\n- The name \"NewModel\" across the paper is really not ideal. There are three possibilities: 1) The authors really do intend to adopt this name 2) a macro failed 3) A name was not found in time for the deadline and the authors feared that the DeBERTaV3 name would be more likely to raise comments about the lack of novelty than \"NewModel\". The \"NewModel\" will then be replaced upon acceptance/finding a new name. I am not very comfortable with the latter as I feel like it games the system. I will give the benefits of the doubt in this instance.\n- I am not a fan of the comparison to larger Megatron models. While it is a larger model, its performance is abnormally low for a model of this size. I think the other comparisons are enough to make your performance clear.\n- Minor: Can you confirm that the step time for GDES is only marginally different than ES? It would be great to have this in the paper as further context for Fig 2.\n\n\nMinor writing issues:\n* where the generator’s and the discriminator’s objectives *interference* with each other and slow down the training convergence -> interfere\n* section 4.1 first sentence: you likely mean GDES not DES. Also I would spell out DA as disentangled attention as all readers might not remember what it stands. Alternatively, use the abbreviation in Section 2.2.\n* page 1 remove space \"(Clark et al., 2020) .\"\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster