ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators
Authors: Kevin Clark
Minh-Thang Luong
Quoc V. Le
Christopher D. Manning
Venue:  (2020)
Keywords: Natural Language Processing
Representation Learning

Abstract: Masked language modeling (MLM) pre-training methods such as BERT corrupt the input by replacing some tokens with [MASK] and then train a model to reconstruct the original tokens. While they produce good results when transferred to downstream NLP tasks, they generally require large amounts of compute to be effective. As an alternative, we propose a more sample-efficient pre-training task called replaced token detection. Instead of masking the input, our approach corrupts it by replacing some tokens with plausible alternatives sampled from a small generator network. Then, instead of training a model that predicts the original identities of the corrupted tokens, we train a discriminative model that predicts whether each token in the corrupted input was replaced by a generator sample or not. Thorough experiments demonstrate this new pre-training task is more efficient than MLM because the task is defined over all input tokens rather than just the small subset that was masked out. As a result, the contextual representations learned by our approach substantially outperform the ones learned by BERT given the same model size, data, and compute. The gains are particularly strong for small models; for example, we train a model on one GPU for 4 days that outperforms GPT (trained using 30x more compute) on the GLUE natural language understanding benchmark. Our approach also works well at scale, where it performs comparably to RoBERTa and XLNet while using less than 1/4 of their compute and outperforms them when using the same amount of compute.


Published as a conference paper at ICLR 2020
ELECTRA: PRE-TRAINING TEXT ENCODERS AS DISCRIMINATORS RATHER THAN GENERATORS
Kevin Clark
Minh-Thang Luong
Quoc V. Le
Stanford University
Google Brain
Google Brain qvl gyllgoogles com
kevclarkēcs:
stanford. sas.lakkkccssssstfrd......
thangl uonge goog 1 com
Christopher D. Manning
Stanford University & CIFAR Fellow manningecs. stanford.edu
ABSTRACT
Masked language modeling (MLM) pre-training methods such as BERT corrupt the input by replacing
tokens with [MASK] and then train model to re- construct the original tokens. While they produce good results when transferred downstream NL. tasks, they generally require large amounts of compute be effective. As an alternative, we propose a more sample-efficient pre-training task called replaced token detection. Instead of masking the input, our approach cor- rupts it by replacing some tokens with plausible alternatives sampled from small generator network. Then, instead of training model that predicts the original identities of the corrupted tokens, we train discriminative model that predicts whether each token in the corrupted input was replaced by generator sample not. Thorough experiments demonstrate this new pre-training task is more ef- ficient than MLM because the task is defined over all input tokens rather than just the small subset that masked out. As result, the contextual representa- tions learned by our approach substantially outperform the ones learned by BERT given the same model size, data, and compute. The gains are particularly strong for small models; for example, we train a model on one GPU for 4 days that outperforms GPT (trained using 30x more compute) guage understanding benchmark. Our approach also works well at scale, where it performs comparably to RoBERTa and XLNet while using less than 1/4 of their compute and outperforms them when using the same amount of compute.
the GLUE natural lan-
INTRODUCTION
Current state-of-the-ar representation learning methods for language can be viewed as learning denoising autoencoders (Vincent al. 2008). They select small subset of the unlabeled input sequence (typically 15%), mask the identities of those tokens (e.g., BERT; Devlin et al. (2019) or attention to those tokens (e.g., XLNet: Yannggg/ann&aaa et (2019)) and then train the network to recover the original input. While more effective than conventional language-moded pre-training due to learning bidirectional representations, these masked language modeling (MLM) approaches incur a substan- tial compute cost because the network only learns from 15% of the tokens per example.
As an alternative, we propose replaced token detection, a pre-training task in which the model learns to distinguish real input tokens from plausible but synthetically generated replacements. Instead of masking, our method corrupts the input by replacing some tokens with samples from proposal distribution, which is typically the output of a small masked language model. This corruption proce- dure solves a mismatch in BERT (although not in XLNet) where the network sees artificial [MASK] tokens during pre-training but not when being fine-tuned on downstream tasks. We then pre-train the network as a discriminator that predicts for every token whether is an original or a replacement. In contrast, MLM trains the network as generator that predicts the original identities of the corrupted tokens. key advantage of our discriminative task is that the model learns from all input tokens instead of just the small masked-out subset, making it more computationally efficient. Although our

---

Published as a conference paper at ICLR 2020
90
XLNet
stops
300k steps
400k steps
90
ELECTRA-Laage
100k steps
RoBERTa steps
300k
RoBERTa
500k steps
RoBERTa XLNet
85
VELECTRA Base
85
BERge
BEERTBBase
ELECTRA Small
80
80
GPT
GLUE
BERT-Small
75
eEMM
GloVe
Replaced Token Detection Pre training
Masked Language Model Pre-training
Pre-train
1e20
Pre-train
1c21
Figure 1: Replaced token detection pre-training consistently outperforms masked language model pre-training given the same compute budget. The left figure is a zoomed-in view of the dashed box.
approach is reminiscent of training the discriminator of GAN, our method is not adversarial that the generator producing corrupted tokens is trained with maximum likelihood due the difficulty of applying GANs to text (Caccia et al. 2018).
We call approach ELECTRA" for "Efficiently Learning an Encoder that Classifies Token Re- placements Accurately." As in prior work, we apply it to pre-train Transformer text encoders Vaasaani al. 2017) that can be fine-tuned on downstream tasks. Through series ablations, we show that learning from all input positions causes ELECTRA to train much faster than BERT. We also show ELECTRA achieves higher accuracy on downstream tasks when fully trained.
Most current pre-training methods require large amounts of compute to be effective, raising con- cerns about their cost and accessibility. Since pre-training with more compute almost always re- sults in better downstream accuracies, we argue important consideration for pre-training methods should compute efficiency as well as absolute downstream performance. From this viewpoint, train ELECTRA models of various sizes and evaluate their downstream performance vs. their compute requirement. In particular, we run experiments on the GLUE natural language understand- ing benchmark (Wang et al.) 2019) and SQuAD question answering benchmark (Rajpurkar et al.) 2016). ELECTRA substantially outperforms MI.M-ased methods such as BERT and XI.Net given the same model size, data, and compute (see Figure T). For example, build an ELECTRA-SSme model that can be trained on GPU in 4 days
2 ELECTRA--Sall outperforms a comparably small BERT model by points on GLUE, and even outperforms the much larger GPT model (Radford et al,, 2018). Our approach also works well at large scale, where we train an ELECTRA: - model that performs comparably to RoBERTa (Liu et al. 2019) and XLNet (Yang spite having fewer parameters and using 1/4 of the compute for training. Training ELECTRA TRA- further results in an even stronger model that outperforms ALBERT (Lan et al., 2019) on GLUE and sets new ttate-of-the-art for SQuAD 2.0. Taken together, our results indicate that the discrim- inative task of distinguishing real data from challenging negative samples more compute-efficien and parameter-efficient than existing generative approaches for language representation learning.
2019), de-
METHOD
We first describe the replaced token detection pre-training task; Figure for overview. We suggest and evaluate several modeling improvements for this method in Section 3.2
"Code and pre-trained weights will released at
elect ra
II has 1/20th the parameters and requires 1/135th the pre-training compute of BERT-Large.
2

---

Published as a conference paper at ICLR 2020
sample
the [MASK]
the
original
chef chef
chef
original
Generator (typically a small MLM)
Discriminator (ELECTRA)
cooked [MASK]
ate
replaced
the
the
the
original
meal
meal
meal
original
Figure 2: An overview of replaced token detection. The generator can be any model that produces an output distribution over tokens, but we usually use small masked language model that trained jointly with the discriminator. Although the models are structured like in a GAN, we train the generator with maximum likelihood rather than adversarially due the difficulty of applying GANs to text. After pre-training, we throw out the generator and only fine-tune the discriminator (the ELECTRA model) on downstream tasks.
Our approach trains two neural networks, generator G and discriminator D. Each one primarily consists of encoder (e-g., Transformer network) that maps sequence input tokens [æ],..., Fn. into sequence of contextualized vector representations h(x) = [f_,.... hnJ. For a given position (in our case only positions where z₁ [MASK]), the generator outputs a probability for generating particular token with a softmax layer:
Pa xx) = exp (e(x))"ha<z))
ha(()))
Σ exp (e(x')"
where e denotes token embeddings. For given position 4, the discriminator predicts whether the token Te is "real," i.e., that it comes from the data rather than the generator distribution, with a sigmoid output layer:
D(x, t) sigmoid(""hpp()))
The generator is trained to perform masked language modeling (MLM). Given input [2,, 22, ...,,..,,,,, JnJ. MLM first select a random set of positions (integers between and n) to mask out [m...... mi The tokens in the selected positions replaced with [MASK] token: REPLACE(E; (x, m, [MASK]). The generator then learns to predict the original identities of the masked-out tokens. The discriminator is trained to distinguish tokens in the data from tokens that have been replaced by generator samples. More specifically, we create a corrupted example cormup by replacing the masked-out tokens with generator samples and train the discriminator to predict which tokens in comupi match the original input æ. Formally, model inputs constructed according
we denote this as x masked
~ unif(1 n1 for = 1 to k
T masked
REPLACE(2. (z, m,, [MASK]
c Pa(a: masked) for
corrutt REPLACE(2, m,
and the loss functions are
(LLM(0, 0G) = E Σ
log pa (2. /er masked)
LDise (2x, 0p)
(1 cormupt xze) log comppt commup 4 x2) log(1 compp .t))
Although similar to the training objective GAN, there are several key differences. First, the generator happens to generate the correct token, that token is considered "real" instead of "fake"; we found this formulation to moderately improve results on downstream tasks. More importantly. the generator is trained with maximum likelihood rather than being trained adversarially fool the discriminator. Adversarially training the generator is challenging because it is impossible to back- propagate through sampling from the generator. Although we experimented circumventing this issue
TTypcallyy i [0.15m] i.e., 15% of the tokens masked out.
3

---

Published as a conference paper at ICLR 2020
by using reinforcement learning to train the generator (see Appendix(F). this performed worse than maxximum-likelihon training. Lastly, we do not supply the generator with a noise vector as input, as is typical with GAN.
We minimize the combined loss
min Σ 8G,9n WEX
LLLM((0, 0c) + >Lee 0p)
over a large corpus X' of raw text. We approximate the expectations in the losses with a single sample. We don't back-propagate the discriminator loss through the generator (indeed, we can't t because of the sampling step). After pre-training, we throw out the generator and fine-tune the discriminator on downstream tasks.
3 EXPERIMENTS
3.1 EXPERIMENTAL SETUP
We evaluate the General Language Understanding Evaluation (GLUE) benchmark (Wang al. 2019) and Stanford Question Answering (SQuAD) dataset (Rajpurkar et al. 2016). GLUE contains a variety of tasks covering textual entailment (RTE and MNLI) question-ansser entailment (QNLI), paraphrase (MRPC), question paraphrase (QQP), textual similarity (STS), sentiment (SST), and lin- guistic acceptability (CoLA). See Appendix f more details on the GLUE tasks. Our evaluation metrics are Spearman correlation for STS, Matthews correlation for CoLA, and accuracy for the other GLUE tasks; we generally report the average score over all tasks. For SQuAD, we evaluate on versions 1.1, in which models select the span of text answering a question, and 2.0, in which some questions are unanswerable by the passage. We use the standard evaluation metrics of Exact-Match (EM) and Fl scores. For most experiments we pre-trate mre-train the same data as BERT, which consists of 3.3 Billion tokens from Wikipedia and BooksCorpus (Zhu et al. 2015). However, for our Large model pre-trained dataset to 33B tokens by including data from Clue Web (Callan et al. 2009), CommonCrawl, and Gigaword (Parker al. (011). All of the pre-training and evaluation is English data, although we think it would be interesting to apply our methods to multilingual data in the future.
the data used for XLNet (Yang al.] 2019), which extends the BERT
Our model architecture and most hyperparameters are the same as BERT's. For fine-tuning on GLUE, we add simple linear classifiers on top of ELECTRA. For SQuAD, we add the question- answering module from XLNet on top of ELECTRA, which is slightly more sophisticated than BERT's in that it jointly rather than independently predicts the start and end positions and has "answerability" classifier added for SQuAD 2.0. Some of our evaluation datasets are small, which means accuracies of fine-tuned models can vary substantially depending therefore report the median of 10 fine-tuning runs from the same pre-trained checkpoint for each result. Unless stated otherwise, results are the dev set. See the appendix for further training details and hyperparameter values.
the random seed. We
3.2 MODEL EXTENSIONS
We improve our method by proposing and evaluating several extensions to the model. Unless stated otherwise, these experiments use the same model size and training data as BERT-Base
Weight Sharing We propose improving the efficiency of the pre-training by sharing weights be- tween the generator and discriminator. If the generator and discriminator are the same size, all of the transformer weights can be tied. However, we found to be more efficient to have a small genera- tor, in which case we only share the embeddings (both the token and positional embeddings) of the generator and discriminator. In this case we use embeddings the size of the discriminator 's hidden states f The "input" and "output" token embeddings of the generator are always tied as in BERT.
We compare the weight tying strategies when the generator is the same size as the discriminator. We train these models for 500k steps. GLUE scores are 83.6 for no weight tying, 84.3 for tying token embeddings, and 84.4 for tying all weights. We hypothesize that ELECTRA benefits from
We add linear layers the generator to project the embeddings into generator-hiddde---izee repeessttt
4

---

Published as a conference paper at ICLR 2020
Which generator size works best?
Comparison Training Algorithrm
85
84
84
83
GLUE Score
Switch
Discriminator
81
70
Discriminator Size
ELECTRA
6-0
768
Adversanal ELECTRA
512
Two Stage ELECTRA
256
BEAT
unigram 32
64
126
256
512
7681024
Generator
Pre-Train FLOPs
1e19
Figure 3: Left: GLUE scores for different generator/ddisrriinaaoo sizes (number of hidden units). Interestingly, having a generator smaller than the discriminator improves results. Right: Comparison of different training algorithms. As our focus on efficiency, the x-axis shows FLOPs rather than train steps (e.g., ELECTRA is trained for fewer steps than BERT because it includes the generator).
tied token embeddings because masked language modeling is particularly effective at learning these representations: while the discriminator only updates tokens that present in the input sampled by the generator, the generator's softmax over the vocabulary densely updates all token embeddings. On the other hand, tying all encoder weights caused little improvement while incurring the significant disadvantage of requiring the generator and discriminator to be the same size, Based on these findings, we use tied embeddings for further experiments in this paper.
Smaller Generators If the generator and discriminator are the same size, training ELECTRA would take around twice as much compute per step as training only with masked language mod- eling. We suggest using smaller generator to reduce this factor. Specifically, we make models smaller by decreasing the layer sizes while keeping the other hyperparameters constant. We also explore using extremely simple "unigram" generator that samples fake tokens according their frequency in the train corpus. GLUE scores for differently-sized generators and discriminators are shown in the left of Figure All models are trained for 500k steps, which puts the smaller gen- erators at a disadvantage in terms compute because they require less compute per training step. Nevertheless, find that models work best with generators 1/4-1/2 the size of the discriminator. We speculate that having too strong of generator may pose too-challenging task for the discriminator, preventing it from learning effectively. In partieular, the discriminator may have its parameters modeling the generator rather than the actual data distribution. Further experiments in this paper use the best generator size found for the given discriminator size.
many of
Training Algorithms Lastly, we explore other training algorithms for ELECTRA, although these did not end up improving results. The proposed training objective jointly trains the generator and discriminator. We experiment with instead using the following two-stage training procedure:
1. Train only the generator with LMLM for steps.
2. Initialize the weights of the discriminator with the weights of the generator. Then train the discriminator with Liss for steps, keeping the generator's weights frozen.
Note that the weight initialization in this procedure requires having the same size for the generator and discriminator. We found that without the weight initialization the discriminator would some- times fail to learn at all beyond the majority class, perhaps because the generator started so far ahead of the discriminator. Joint training on the other hand naturally provides a curriculum for the dis- criminator where the generator starts off weak but gets better throughout training. We also explored training the generator adversarially as in a GAN, using reinforcement learning to accommodate the discrete operations of sampling from the generator. See Appendix F for details.
Results are shown the right of Figure 3 During two-stage training, downstream task performance notably improves after the switch from the generative the discriminative objective, but does not end up outscoring joint training. Although still outperforming BERT, we found adversarial training underperform maximum-lin likelloooiel training. Further analysis suggests the gap is caused by two
5

---

Published as a conference paper at ICLR 2020
Model
Train/ / Infer FLOPs
Speedup
Params
Train Time + Hardware
GLUE
ELMo
3.3c18 2.6c10
19x 1.2x
96M
14d on GTX 1080 GPUs
71.2
GPT
4.0e19 3,0e10
1.6x 0.97x
117M
25d on 8 P6000 GPUs
78.8
BERT-Small
1.4e18 3.7e9
45x / 8x
14M
4d on I V100 GPU
75.1
BERT-Base
6.4e19 2.9e10
1x 1x
110M
4d 16 TPUv3s
82.2
GLECTRA-Smal
1.c18/3.788
45x / 8x
14M
4d on 1 V100 GPU
79.9
50% trained
7.1c17 7..767
90x 8x
14M
2d on V100 GPU
79.0
25% trained
3.6e17 / 3.77e9
181x 8x
14M
on V100 GPU
77.7
12.5% trained
1.8e17 3.7e9
361x / 8x
14M
12h on V100 GPU
76.0
6.25% trained
8.9e16 22216/3.3 3.7e9
722x / 88
14M
6h on 1 V100 GPU
74.1
ELECTRA Base
6.4e19 2.9e10
1x/ Ix
110M
4d on 16 TPUv3s
85.1
implemen- Table I: Comparison of small models on the GLUE dev set. BERT- Small/Base tation and use the same hyperparameters as ELLECTRA-Smal/BBe Length-128 input. Training times should be taken with grain of salt they are for different hard- ware and with sometimes un-optimized code. ELECTRA performs well even when trained on a single GPU, scoring 5 GLUE points higher than comparable BERT model and even outscoring the much larger GPT model.
Infer FLOPs assumes single
problems with adversarial training. First, the adversarial generator is simply worse masked lan- guage modeling; achieves 58% accuracy at masked language modeling compared to 65% accuracy for ML---rained one. We believe the worse accuracy is mainly due the poor sample efficiency of reinforcement learning when working in the large action space of generating text. Secondly, the adversarially trained generator produces low-entropy output distribution where most of the proba- bility mass is on single token, which means there is not much diversity in the generator samples. Both of these problems have been observed in GANs for text in prior work (Caccia al. 2018
3.3 SMALL MODELS
As goal of this work is improve the efficiency of trainngin develop small model that be quickly trained on a single GPU. Starting with the BERT- BERT-Base hyperparameters, we shortened the sequence length (from 512 128), reduced the batch size (from 256 128), reduced the model's hidden dimension size (from 768 to 256), and used smaller token embeddings (from 768 to 128). To provide fair comparison, train BERT-Small for 1.5M steps, so it uses the same training FLOPs as ELECTRA-Sman was trained for 1M steps In addition to BERT, we compare against
also train BERT-Small model using the same hyperparameters. We
which less resource-intensive pre-training methods based on language modeling: ELMo (Peters et al. 2018) and GPT (Radford al. 2018 We also show results for a base-sized ELECTRA model comparable to BERT-Base.
Results are shown in T See uppendix[]joo additional results, including stronger small-sized and base-sized models trained with more compute. ELECTRA-Smin performs remarkably well given its size, achieving higher GLUE score than other methods using substantially more compute and parameters. For example, it scores 5 points higher than a comparable BERT-Small model and outperforms the much larger GPT model. ELECTRA-Small is trained mostly to convergence, with models trained for even less time (as little as 6 hours) still achieving reasonable performance. While small models distilled from larger pre-trained transformers scores (Sun et al. [2196] Diao et al.] all, 2019). these models require first expending substantial compute pre-train the larger teacher model. The results also demonstrate the strength of ELECTRA at moderate size; our base-sized ELECTRA model substantially outperforms BERT-Base and even outperforms BERT-Large (which gets 84.0 GLUE score). We hope ELECTRA's ability achieve strong results with relatively little compute will broaden the accessibility of developing and applying pre-trained models NLP.
also achieve good GLUE
"ELECTRA requires more FL.PPs per step because it consists of the generator as well as the discriminator. 'GPT is similar in size BERT-Base, hut is trained for fewer steps.

---

Published as a conference paper at ICLR 2020
Model
Train FLOPs
Params CoLA
SST MRPC
STS
QQP
MNLI
QNLI
RTE
Avg.
BERT
1.9c20 (0.27x)
335M
60.6
93.2
88.0
90.0
91.3
86.6
92.3
70.4
84.0
RoBERTa-1000
6,4e20 (0,90x)
356M
66.1
95.6
91.4
92.2 92.0
89.3
94.0
82.7
87.9
RoBERTa-500K
3.2e21 (4.5x)
356M
68.0
96.4
90.9
92.1
92.2
90.2
94.7
86.6
88.9
XLNet
3.9e21 (5.4x)
360M
69.0
97.0
90.8
92.2 92.3
90.8
94.9
85.9
89.1
BERT (ours)
7.1c20 (1x)
335M
67.0
95.9
89.1
91.2 91.5
89.6
93.5
79.5
87.2
ELECTRA-400K
7.1c20 (1x)
335M
69.3
96.0
90.6
92.1
92.4
90.5
94.5
86.8
89.0
ELECTRA-L.75M
3.1e21 (4.4x)
335M
69.1
96.9
90.8
92.6
92.4
90.9
95.0
88.0
89.5
Table 2: Comparison of large models on the GLUE dev set. ELECTRA and RoBERTa are shown for different numbers of pre-training steps, indicated by the numbers after the dashes. ELECTRA performs comparably to XeN and RoBERTa when using less than 1/4 of their pre-training compute and outperforms them when given similar amount of pre-training compute. BERT dev results are from Clark et al. (019))
Model
Train FLOPs
CoLA
SST
MRPC
STS
QQP
MNLI
QNLI
RTE
WNLI
Avg.
Score
BERT
1.9e20 (0.06x)
60.5
94.9
85.4
86.5
89.3
86.7
92.7
70.1
65.1
79.8
80.5
RoBERTa
3,2c21 (1.02x)
67.8
96.7
89.8
91.9
90.2
90.8
95.4
88.2
89.0
88.1
88.1
ALBERT
3,1e22 (10x)
69.1
97.1
91.2
92.0
90.5
91.3
89.2
91.8
89.0
XLNet
3.9c21 (1.26x)
70.2
97.1
90.5
92.6
90.4
90.9
88.5
92.5
89.1
ELECTRA 3.1e21 (Ix)
71.7
97.1
90.7
92.5
90.8
91.3
95.8
89.8
92.5
89.5
89.4
Table 3: GLUE test-set results for large models. Models in this table incorporate additional tricks such as ensembling to improve scores (see Appendix B for details). Some models do not have QNLI scores because they treat QNLI ranking task. which has recently been disallowed by the GLUE benchmark. To compare against these models, we report the average score excluding QNLI (Avg.* *) in addition to the GLUE leaderboard score (Score). "ELECTRA" and "RoBERTa" refer to the fully-trained ELECTRA-1.72 1.75M and RoBERTa-5000 models.
3.4 LARGE MODELS
We train big ELECTRA models to measure the effectiveness of the replaced token detection pre- training task the large scale of current state-of-the-an pre-trained Transformers. Our ELECTRA- Large models are the same size as BERT-Large but are trained for much longer. In particular, we train a model for 400k steps BELECRAA-400% roughly 1/4 the pre-training compute of RoBERTa) and one for 1.75M steps TELECTRA-L.75M, similar compute to RoBERTa). We use batch size 2048 and the XLNet pre-training data. We note that although the XLNet data is similar to the data used to train RoBERTa, the comparison is not entirely direct. As baseline, we trained our own BERT-Large model using the same hyperparameters and training time as ELECTRA-400K.
Results on the GLUE dev set are shown in Table 2 ELECTRA-400K performs comparably to RoBERTa and XLNet. However, it took less than I/4 of the compute to train ELECTRA-400K as it did to train RoBERTa and XLNet, demonstrating that ELECTRA's sample-efficiency gains hold at large scale. Training ELECTRA for longer (ELECTRA-1.75M) results in a model that outscores them most GLUE tasks while still requiring less pre-training compute. Surprisingly, our baseline BERT model scores notably worse than ROBERTa-1008 100K, suggesting our models may benefit from more hyperparameter tuning using the RoBERTa training data. ELECTRA's gains hold on the GLUE test set (see Table 3). although these comparisons are less apples-to-apple due to the additional tricks employed by the models (see Appendix B).
Results on SQuAD are shown in Table Consistent, with the GLUE results, ELECTRA scores better than masked-languaee-modeee outperforms RoBERTa- 100k and our BERT baseline, which use similar
based methods given the same compute resources. For ex-
ample, ELECTRA-4000 amounts of pre- tre-training compute. ELECTRA-400K also performs comparably to RoBERTa-500k despite using less than 1/4th of the compute. Unsurprisingly, training ELECTRA longer improves results further: ELECTRA-11. A-1,75M scores higher than previous models the SQuAD 2.0 bench-
7

---

Published as a conference paper at ICLR 2020
SQuAD 1.1 dev
SQuAD
2.0 dev
SQuAD
2.0 test
Model
Train FLOPs
Params
EM
F1
EM
FI
EM
F1
BERT-Base
6,4e19 (0,09x)
110M
80.8
88.5
BERT
1.9e20 (0.27x)
335M
84.1
90.9
79.0
81.8
80.0
83.0
SpanBERT
7.1e20 (Ix)
335M
88.8
94.6
85.7
88.7
85.7
88.7
XLNet-Base
6,6e19 (0.09x)
117M
81.3
78.5
XLNet
3,9c21 (5,4x)
360M
89.7
95.1
87.9
90.6
87.9
90.7
RoBERTa- 100K
6,4c20 (0,90x)
356M
94.0
87.7
RoBERTa-5000
3.2c21 (4.5x)
356M
88.9
94.6
86.5
89.4
86.8
89.8
ALBERT
3.1e22 (44x)
235M
89.3
94.8
87.4
90.2
88.1
90.9
BERT (ours)
7.1e20 (Ix)
335M
88.0
93.7
84.7
87.5
ELECTRA-
6.4e19 (0.09x)
110M
84.5
90.8
80.5
83.3
ELECTRA-400K
7.1e20 (1x)
335M
88.7
94.2
86.9
89.6
ELECTRA-1.75M
3.1c21 (4.4x)
335M
89.7
94.9
88.0
90.6
88.7
91.4
Table 4: Results on the SQuAD for non-ensemble models.
mark. ELECTRA-Base also yields strong results, scoring substantially better than BERT-Base and XLNet-Base, and even surpassing BERT-Large according most metrics. ELECTRA generally performs better at SQuAD 2.0 than 1.1. Perhaps replaced token detection, in which the model distinguishes real tokens from plausible fakes, is particularly transferable to the answerability clas- sification of SQuAD 2.0, in which the model must distinguish answerable questions from fake unan- swerable questions.
3.5 EFFICIENCY ANALYSIS
We have suggested that posing the training objective over small subset of tokens makes masked language modeling inefficient. However, it isn't 't entirely obvious that this is the case. After all, the model still receives large number of input tokens even though it predicts only small number of masked tokens. To better understand where the gains from ELECTRA are coming from, we compare series of other pre-training objectives that designed to be of "stepping stones" between BERT and ELECTRA.
ELECTRA 15%: This model is identical to ELECTRA except the discriminator loss only comes from the 15% the tokens that were masked out of the input. In other words, the sum in the discriminator loss Dise is over E instead of from to n
Replace MLM: This objective is the same as masked language modeling except instead of replacing masked-out tokens with [MASK] they are replaced with tokens from a generator model. This objective tests to what extent ELECTRA's gains come from solving the dis- crepancy of exposing the model to [MASK] tokens during pre-training but not fine-tuning.
All-Tokens MIM: Like in Replace MLM, masked tokens are replaced with generator sam- ples. Furthermore, the model predicts the identity of all tokens in the input, not just ones that were masked out. We found it improved results train this model with an explicit copy mechanism that outputs a copy probability D for each token using sigmoid layer. The model's output distribution puts weight on the input token plus D times the output of the MLM softmax. This model is essentially combination of BERT and ELEC- TRA. Note that without generator replacements, the model would trivially learn to make predictions from the vocabulary for [MASK] tokens and copy the input for other ones.
Results are shown in Table First, we find that ELECTRA is greatly benefiting from having a loss defined over all input tokens rather than just subset: ELECTRA 15% performs much worse than ELECTRA. Secondly, we find that BERT performance is being slightly harmed from the pre-train fine-tune mismatch from [MASK] tokens, as Replace MLM slightly outperforms BERT. We note that BERT (including our implementation) already includes a trick to help with the pre- mre-rainnfiie- tune discrepancy: masked tokens replaced with random token 10% of the time and kept the
We also trained a discriminator that learns from a random 15% of the input tokens distinct from the subset that was originally masked out; this model performed slightly worse.
8

---

Published as a conference paper at ICLR 2020
Model
ELECTRA
All- Tokens MLM
Replace MLM
ELECTRA 15%
BERT
GLUE score
85.0
84.3
82.4
82.4
82.2
Table 5: Compute-cfficieng experiments (see text for details).
ELECTRA improvementt over BERT 5
80
Score
80
GLUE Score
75-
76
70-
74
72
ELECTRA BERT
70
65
BERT-256
68
128
2
Hidden State Size
Hidden State Size
Pre-Train FLOPs 1e18
Figure 4: Left and Center: Comparison of BERT and ELECTRA for different model sizes. Right: A small ELECTRA model converges to higher downstream accuracy than BERT, showing the im- provement comes from more than just faster training.
same 10% of the time. However, our results suggest these simple heuristics are insufficient to fully solve the issue. Lastly, we find that All-Tokens MLM, the generative model that makes predictions over all tokens instead of subset, closes most of the gap between BERT and ELECTRA. total, these results suggest large amount of ELECTRA's improvement can be attributed to learning from all tokens and a smaller amount can be attributed alleviating the pre-train fine-tune mismatch.
The improvement of ELECTRA over All-Tokens MLM suggests that the ELECTRA's gains come from more than just faster training. We study this further by comparing BERT to ELECTRA for various model sizes (see Figure 4 left). We find that the gains from ELECTRA grow larger the models get smaller. The small models are trained fully to convergence (see Figure right), showing that ELECTRA achieves higher downstream accuracy than BERT when fully trained. We speculate that ELECTRA is more parameter-efffcient than BERT because it does not have to model the full distribution of possible tokens each position, but believe more analysis needed to completely explain ELECTRA'S parameter efficiency.
4 RELATED WORK
Self-Supervised Pre-training for NLP Self-supervised learning has been used learn word rep- resentations (Collobert et al. 2 Pennington et al.) 2014) and more recently contextual represen- tations of words though objectives such as language modeling (Dai & Le 2013 Peters et al. 2018 Howard & Ruder 2018). BERT (Devlin et al.) 2019) pre-trains a large Transformer (Vaswani al. 2017) at the masked-language modeling task. There have been numerous extensions to BERT. For example, MASS (Song et al. 2019) and UniLM (Dong et al. 2019) extend BERT to generation tasks by adding auto-regressive generative training objectives. ERNIE (Sun et al. 201 and BERn (Joshi et al., 2019) mask contiguous sequences of token for improved span representations. This idea may be complementary to ELECTRA; we think it would be interesting to make ELECTRA's generator auto-regressive and add "replaced span detection" task. Instead of masking out input tokens, XLNet (Yang et al.) 2019) masks attention weights such that the input sequence is auto- regressively generated in a random order. However, this method suffers from the same inefficiencies as BERT because XLNNt only generates 15% of the input tokens in this way. Like ELECTRA, XL- Net may alleviate BERT's pretrain- finnttunn- discrepancy by not requiring [MASK] tokens, although this isn't entirely clear because XLNet uses two "streams" of attention during pre-training but only one for fine-tuning. Recently, models such as TinyBERT (Jiao et al. 2019) and MobileBERT (Sun) et al. 2019b) show that BERT effectively be distilled down smaller model. In contrast, focus more on pre-traine pre-training speed rather than inference speed, so we train ELECTRA- Amall from scratch.

---

Published as a conference paper at ICLR 2020
Generative Adversarial Networks GANs (Goodfellow et al. 2014 are effective generating high-quality synthetic data. Radford et al. (016) propose using the discriminator of a GAN in downstream tasks, which is similar to method. GANs have been applied to text data (Yu et al. 2017 Zhang et al., 2017), although state-of-the-ar approaches still lag behind standard maximum- likelihood training (Caccia et al. 2018) [eveet et al., 2018). Although we do not use adversarial learning, our generator is particularly reminiscent of MaskGAN (Fedus et al. 2018), which trains the generator fill in tokens deleted from the input.
Contrastive Learning Broadly, contrastive learning methods distinguish observed data points from fictitious negative samples. They have been applied many modalities including text (Smith & Eisner 2005), images (Chopra et al.) 20055, and video (Wang & Gupta [015 Serrmantt et al.) 2017)
data. Common approaches learn embedding spaces where related data points are similar (Saunshi
et al.
2019) or models that rank real data points over negative samples (Collobert et al. [201] [Bordes
ct al.
2013). ELECTRA is particularly related to Noise Contrastive Estimation (NCE) (Gutmann &
Hyvärinen 2010), which also trains a binary classifier to distinguish real and fake data points.
Word2Vec Mikoloov et al. 2013), one the earliest pre-training methods for NLP, uses contrastive learning. In fact, ELECTRA can be viewed as massively scaled-up version of Continuous Bag- of Words (CBOW) with Negative Sampling. CBOW also predicts an input token given surrounding context and negative sampling rephrases the learning task as binary classification task whether the input token comes from the data or proposal distribution. However, CBOW uses a bag-of- vectors encoder rather than transformer and simple proposal distribution derived from unigram token frequencies instead of a learned generator.
CONCLUSION
We have proposed replaced token detection, a new self-supervised task for language representation learning. The key idea is training a text encoder to distinguish input tokens from high-quality nega- tive samples produced by an small generator network. Compared masked language modeling, our pre-training objective is more compute-efficient and results in better performance on downstream tasks. works well even when using relatively small amounts of compute, which we hope will make developing and applying pre-trained text encoders more accessible to researchers and practi- tioners with less access to computing resources. We also hope more future work on NLP pre-training will consider efficiency well absolute performance, and follow effort in reporting compute usage and parameter counts along with evaluation metrics.
ACKNOWLEDGEMENT
We thank Allen Nie, Prajit Ramachandran, audiences at the CIFAR LMB meeting and U. de Montréal, and the anonymous reviewers for their thoughtful comments and suggestions. We thank Matt Peters for answering our questions about ELMo, Alec Radford for answers about GPT, Naman Goyal and Myle Ott for answers about RoBERTa, Zihang Dai for answers about XLNet, Zhenzhong Lan for answers about ALBERT, and Danqi Chen and Mandar Joshi for answers about SpanBERT. Kevin is supported by a Google PhD) Fellowship.
REFERENCES
Antoine Bordes, Nicolas Usunier, Alberto García-Durán, Jason Weston, and Oksana Yakhnenko. Translating embeddings for modeling multi-relational data. In NeurlPS, 2013.
Avishek Joey Bose, Huan Ling, and Yanshuai Cao. Adversarial contrastive estimation. In ACL, 2018.
Massimo Caccia, Lucas Caccia, William Fedus, Hugo Larochelle, Joelle Pineau, and Laurent Char- lin. Language GANs falling short. arXiv preprint 10Xiv:1811.02544 2018
Jamie Callan, Mark Hoy, Changkuk Yoo, and Le Zhao. Clueweb09 data set, 2009. URL https: 7/1emurprojectt wtt/clueweb//bo.ooooooo
10

---

Published as a conference paper at ICLR 2020
Daniel M. Cer, Mona T. Diab, Eneko Agine, lñigo Lopez-Gazpin and Lucia Specia. Semeval- 2017 task I: Semantic textual similarity multilingual and crosslingual focused evaluation. In SemEval@ACL, 2017.
Sumit Chopra, Raia Hadsell, and Yann LeCun. Learning similarity metric discriminatively, with application to face verification. CVPR, 2005.
Kevin Clark, Minh-Thang Luong, Urvashi Khandelwal, Christopher D. Manning, and Quoe V. Le. BAM! Born-again multi-task networks for natural language understanding. In ACL, 2019.
Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel P. Kuksa. Natural language processing (almost) from scratch. JMLR, 2011.
Andrew M Dai and Quoc V Le. Semi-supervised sequence learning. In NeurIPS, 2015.
Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep
bidirectional transformers for language understanding. In NAACL-HLT, 2019.
William B. Dolan and Chris Brockett. Automatically constructing corpus sentential paraphrases. In @WP@ IJCNLP, 2005.
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao- Wuen Hon. Unified language model pre-training for natural language understanding and generation. In NeurlPS, 2019.
William Fedus, Ian J. Goodfellow, and Andrew M. Dai. MaskGAN; Better text generation via filling in the In ICLR, 2018.
Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and William B. Dolan. The third pascal recog- nizing textual entailment challenge. In ACL-PASCAL @ ACL, 2007.
lan J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. In NeurIPS, 2014.
Michael Gutmann and Aapo Hyvärinen. conttscontre estimation: A new estimation principle for unnormalized statistical models. In AISTATS, 2010.
Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. In ACL, 2018.
Shankar Iyer, Nikhil Dandekar, and Korl Csernai. lease: Question pairs,
First Quora dataset re-
2017.
URL https:://ata.qqu quora com/
First Quora Mestaset-Relest
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling bert for natural language understanding. arXiv preprint arXiv:1909. 1001:1909.10351, 2019.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S Weld, Luke Zettlemoyer, and Omer Levy. SpanBERT: Improving pre-training by representing and predicting spans. arXiv preprint 2019.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Sori- cut. ALBERT: lite bert for self-supervised learning of language representations. arXiv preprint arXiv: 1994999999999...... 2019,
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: robustly optimized BERT pre- training approach. arXiv preprint 1rXiv:1907.11692 2019.
Tomas Mikolov, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. In ICLR Workshop Papers, 2013.
Robert Parker, David Graff, Junbo Kong, Ke Chen, and Kazuaki Maeda. English gigaword, fifth edition. Technical report, Linguistic Data Consortium, Philadelphia, 2011.

---

Published as a conference paper at ICLR 2020
Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In EMNLP, 2014.
Matthew Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In NAACL-HLT, 2018.
Jason Phang, Thibault Févry, and Samuel R Bowman. Sentence encoders on STILTs: Supplemen- tary training on intermediate labeled-data tasks. arXiv preprint urXiv: 2018.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks, In ICLR, 2016.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing by generative pre-training. https://blog. wwwpggg.pppaaiicicooo seneenniccoommmggggggoeemusssere 2018.
https://boogoopeeaiccomm
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy S. Liang. Squad: 100, 000+ questions for machine comprehension of text. In EMNLP, 2016.
Nikunj Saunshi, Orestis Plevrakis, Sanjeev Arora, Mikhail Khodak, and Hrishikesh Khandeparkar. A theoretical analysis of contrastive unsupervised representation learning. In ICML, 2019.
Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, and Sergey Levine. Time-contrastive networks: Self-supervised learning from video. ICRA, 2017.
Noah A. Smith and Jason Eisner. Contrastive estimation: Training log-linear models on unlabeled data. In ACL, 2005.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP, 2013.
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tin-Yan Liu. MASS: Masked sequence to sequence pre-training for language generation. ICML, 2019.
Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian, and Hua Wu. Emne: Enhanced representation through knowledge integration. arXiv preprint 190.:0990,,, 2019a.
Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. Mobile- BERT: Task-agnostic compression of bert for resource limited devices, 2019b. URL https:
/ /oopnzevieww wttp:/wwwww..eeeteeee.eeeece...e VaNKK
Guy Tevet, Gavriel Habib, Vered Shwartz, and Jonathan Berant. Evaluating text gans as language models. In NAACL-HLT, 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017.
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In ICMI., 2008
Alex Wang, Amapreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In ICLR, 2019.
Xiaolong Wang and Abhinav Gupta. Unsupervised learning of visual representations using videos. ICCV, 2015.
Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. Neural network acceptability judgments. arXiv preprint wxXiv:1005.144777 2018
Adina Williams, Nikita Nangia, and Samuel R. Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In NAACL-HILT, 2018.
12

---

Published as a conference paper at ICLR 2020
Ronald J. Williams. Simple statistical
algorithms for connectionisttreifforeemee
learning. Machine Learning. 8(3-4):229-255 1992.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. XLNet: Generalized autoregressive pretraining for language understanding, In NeurIPS, 2019.
Lantao Yu, Weinan Zhang, Jun Wang. and Yingrui Yu. SeqGAN: Sequence generative adversarial nets with policy gradient. In AAAI, 2017.
Yizhe Zhang. Zhe Gan, Kai Fan, Zhi Chen, Ricardo Henao, Dinghan Shen, and Lawrence Carin. Adversarial feature matching for text generation. In ICMI, 2017.
Yukun Zhu, Ryan Kiros, Richard S. Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Tor- ralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. ICCV, 2015.
A PRE-TRAINING DETAILS
The following details apply to both our ELECTRA models and BERT baselines. We mostly use the same hyperparameters as BERT. We set the weight for the discriminator objective in the loss to 5 We use dynamic token masking with the masked positions decided on-the-fly instead of during preprocessing. Also, we did not use the next sentence prediction objective proposed the original BERT paper, recent work has suggested it does not improve scores (Yang et al. [019 Liu et al.) 2019) For our ELECTRA-Large model, we used a higher mask percent (25 instead of 15) because noticed the generator was achieving high accuracy with 15% masking, resulting in very few replaced tokens. We searched for the best learning rate for the Base and Small models out of [1c-4, 2e-4, 3e-4, 5e-4] and selected Ả out of [1, 10, 20, 50, 100] in early experiments. Otherwise did no hyperparameter tuning beyond the experiments in Section 3.2 The full set of hyperparameters listed in Table 6
B FINE-TUNING DETAILS
For Large-sized models, used the hyperparameters from Clark al. (2019) for the most part. However, after noticing that RoBERTa (Liu et al., (019)) uses more training epochs (up to 10 rather than 3) searched for the best number of train epochs of [10, 3] for each task. For SQuAD, we decreased the number of train epochs to 2 to be consistent with BERT and RoBERTa. For Base- sized models searched for learning rate out of [3e-5, 5e-5, le-4, 1.5e-4] and the layer-wise leamning-rate decay out of [0.9, 0.8, 0.7], but otherwise used the same hyperparameters as for Large models. We found the small models benefit from larger learning rate and searched for the best one out of [1e-4, 2e-4, 3e-4, 5e-3]. With the exception of number of train epochs, we used the same hyperparameters for all tasks. contrast, previous research on GLUE such as BERT, XLNet, and RoBERTa separately searched for the best hyperparameters for each task. We expect our results would improve slightly if we performed the same sort additional hyperparameter search. The full set of hyperparameters is listed in Table 7
Following BERT, we do not show results on the WNLI GLUE task for the dev set results, as it is difficult to beat even the majority classifier using standard fine-tuning- as-classifier approach. For the GLUE test set results, we apply the standard tricks used by many of the GLUE leaderboard submissions including RoBERTa (Liu al. 2019). XLNet (Yang al. 2019). and ALBERT (Lan et al. 2019). Specifically:
For RTE and STS we use intermediate task training (Phang et al. 2018), starting from an ELECTRA checkpoint that has been fine-tuned on MNLI. For RTE, found it helpful to combine this with lower learning rate of 2e-5.
As binary classification task instead of the 30,000-way classification task in ML.,, the discriminator's loss was typically much lower than the generator's.
13

---

Published as a conference paper at ICLR 2020
Hyperparameter
Small
Base
Large
Number of layers
12
12
24
Hidden Size
256
768
1024
FFN inner hidden size
1024
3072
4096
Attention heads
4
12
16
Attention head size
64
64
Embedding Size
128
768
1024
Generator Size (multiplier for hidden-size,
1/4
1/3
1/4
FFN-size, and nun-attention-heade
Mask percent
15
15
25
Learning Rate Decay
Linear
Linear
Linear
Warmup steps
10000
10000
10000
Learning Rate
5e-4
2e-4
2e-4
Adam e
le-6
le-6
lc-6
Adam 31
0.9
0.9
0.9
Adam &2
0.999
0.999
0.999
Attention Dropout
0.1
0.1
0.1
Dropout
0.1
0.1
0.1
Weight Decay
0.01
0.01
0.01
Batch Size
128
256
2048
Train Steps (BERT/ELECTRA)
1.45M/IM
1M/766K
464K/400K
Table 6: Pre-train hyperparameters. We also train an ELECTRA-LLarge model for 1.75M steps (other hyperparameters identical).
Hyperparameter
GLUE Value
Learning Rate
3e-4 for Small, le-4 for Base, 5e-5 for Large
Adam
le-6
Adam 31
0.9
Adam B2
0.999
Layerwise LR decay
0.8 for Base/Small, 0.9 for Large
Learning rate decay
Linear
Warmup fraction
0.1
Attention Dropout
0.1
Dropout
0.1
Weight Decay
Batch Size
32
Train Epochs
10 for RTE and STS, 2 for SQuAD, 3 for other tasks
Table Fine-tune hyperparameters
For WNLI, follow the trick described in Liu et al. (019) where extract candidate antecedents for the pronoun using rules and train a model to score the correct antecedent highly. However, different from|Liu al. (2019). the scoring function is not based MLM probabilities. Instead, we fine-tune ELECTRA'S discriminator so it assigns high scores to the tokens of the correct antecedent when the correct antecedent replaces the pronoun. For example, if the Winograd schema is "the trophy could not fit in the suitcase because it was too big," we train the discriminator so it gives a high score to "trophy" in "the trophy could not fit in the suitcase because the trophy was too big" but low score to "suitcase" in "the trophy could not in the suitcase because the suitcase was too big."
For each task we ensemble the best of 30 models fine-tuned with different random seeds but initialized from the same pre-trained checkpoint.
While these tricks do improve scores, they make having clear scientific comparisons more difficult because they require extra work to implement, require lots of compute, and make results less apples-
14

---

Published as a conference paper at ICLR 2020
to-apples because different papers implement the tricks differently. We therefore also report results for ELECTRA- with the only trick being dev-set model selection (best of 10 models), which is the setting BERT used to report results, in Table g
For our SQuAD 2.0 test set submission, we fine-tuned 20 models from the same pre-trained check- point and submitted the one with the best dev score.
C DETAILS ABOUT GLUE
We provide further details about the GLUE benchmark tasks below
CoLA:
Corpus of Linguistic Acceptability (Warstadt et al. 2018). The task to determine
whether a given sentence is grammatical or not. The dataset contains 8.5k train examples from books and journal articles on linguistic theory.
Stanford Sentiment Treebank (Socher et al., 2013). The tasks is determine the sentence is positive or negative in sentiment. The dataset contains 67k train examples from movie reviews.
SST:
MRPC: Microsoft Research Paraphrase Corpus (Dolan & Brocketi 2005). The task is to predict whether two sentences are semantically equivalent or not. The dataset contains 3.7k train examples from online news sources.
STS: Semantic Textual Similarity (Cer et al. 2017). The tasks is to predict how seman- tically similar two sentences are on a 1-5 scale. The dataset contains 5.8k train examples drawn from new headlines, video and image captions, and natural language inference data.
QQP: Quora Question Pairs (lyer et al. 2017). The task is to determine whether pair of questions are semantically equivalent. The dataset contains 364k train examples from the community question-answere
website Quora.
MNLI:
Multi-gena gelti Natural Language Inference (Williams et al. 2018). Given a premise sentence and hypothesis sentence, the task is predict whether the premise entails the hypothesis, contradicts the hypothesis, or neither. The dataset contains 393k train examples drawn from ten different sources.
Question Natural Language Inference; constructed from SQuAD (Rajpurkar al. 2016 The task to predict whether a context sentence contains the answer to a question sentence. The dataset contains 108k train examples from Wikipedia.
QNLI:
RTE: Recognizing Textual Entailment (Giampiccolo et al. 2007). Given a premise sen- tence and a hypothesis sentence, the task to predict whether the premise entails the hy- pothesis or not. The dataset contains 2.5k train examples from series of annual textual entailment challenges.
D FURTHER RESULTS ON GLUE
We report results for ELECTRA-Bass and ELECTRA-Small on the GLUE test set in Table S Furthermore, we push the limits of base-sized and siall-sized models by training them on the XLNet data instead of wikibooks and for much longer (4e6 train steps); these models are called ELECTRA LLECTRA-BBse++ and SECTRA-Small++ in the table. For ELECTRA-Smalee
we also in- creased the sequence length to 512; otherwise the hyperparameters are the same as the ones listed in Table 6 Lastly, the table contains results for I.EETRA-1175M without the tricks described in Appendix B Consistent with dev-set results in the paper, ELECTRA- outperforms BERT-Large while ELECTRA-S
outperforms GPT in terms of average score. Unsurprisingly, the models perform even better. The small model scores are even close to TinyBERT (Tio et al. 2019) and Mo- bileBERT (Sun et al. 2019b). These models learn from BERT Base using sophisticated distillation procedures. Our ELECTRA models, on the other hand, are trained from scratch. Given the success of distilling BERT, we believe would be possible build even stronger small pre-trained models by distilling ELECTRA. ELECTRA appears to be particularly effective at CoLA. In CoLA the goal to distinguish linguistically acceptable sentences from ungrammatical ones, which fairly closely matches ELECTRA's pre-training task of identifying fake tokens, perhaps explaining ELÉCTRA'S strength at the task.
15

---

Published as a conference paper at ICLR 2020
Model
Train FLOPs
Params CoLA
SST
MRPC STS QQP
MNLI
QNLI
RTE
Avg.
TinyBERT
6.4c19+ (45x+)
14.5M
51.1
93.1
82.6
83.7 89.1
84.6
90.4
70.0
80.6
MobileBERT
6.4e19+ (45x+)
25.3M
51.1
92.6
84.5
84.8 88.3
84.3
91.6
70.4
81.0
GPT
4.0e19 (29x)
117M
45.4
91.3
75.7
80.0 88.5
82.1
88.1
56.0
75.9
BERT-Base
6.4e19 (45x)
110M
52.1
93.5 84.8
85.8 89.2
84.6
90.5
66.4
80.9
BERT-Large
1.9e20 (135x)
335M
60.5
94.9 85.4
86.5 89.3
86.7
92.7
70.1
83.3
SpanBERT
7.1c20 (507x)
335M
64.3
87.9 94.8
89.9 89.5
87.7
94.3
79.0
85.9
ELECTRA-Small
1.4e18 (Ix)
14M
54.6
83.7
80.3 88.0
79.7
87.7
60.8
78.0
ELECTRA-Small++
3.3e19 (18x)
14M
55.6
91.1
84.9
84.6 88.0
81.6
88.3
63.6
79.7
ELECTRA-Base
6.4e19 (45x)
110M
59.7
93.4
86.7
87.7 89.1
85.8
92.7
73.1
83.5
SECCRRA-Bas++
3.3e20 (182x)
110M
64.6
96.0
88.1
90.2 89.5
88.5
93.1
75.2
85.7
ELECTRA-I.75M
3.1e21 (2200x)
330M
68.1
89.2
91.7
90.7
95.5
86.1
88.6
Table 8: Results for models on the GLUE test set. Only models with single-task finetuning (no ensembling, task-specific tricks, etc.) are shown.
E COUNTING FLOPS
We chose measure compute usage in terms of floating point operations (FLOPs) because it is measure agnostic to the particular hardware, low-level optimizations, etc. However, it is worth not- ing that in some cases abstracting away hardware details is drawback because hardware-centered optimizations can be key parts of model's design, such as the speedup ALBERT (Lan et al.) 2019) gets by tying weights and thus reducing communication overhead between TPU workers. We used TensorFlow's 's FLOP-counting capabilities and checked the results with by-hand computation. We made the following assumptions:
An "operation" is mathematical operation, not machine instruction. For example, an exp is one op like an a add, even though practice the exp might be slower. We believe this assumption does dominate the compute for most models. Similarly, we count matrix-multiplies as 2 m n FLOPs instead of m n as one might if considering fused multiply-add operations.
substantially change compute estimates because matrix-multipliss
The backwards pass takes the same number of FLOPs the forward pass. This assumption is not exactly right (e.g.. for softmax cross entropy loss the backward pass is faster), but importantly, the forward/backward pass FLOPs really are the same for matrix-multiplies, which is most of the compute anyway.
We assume "dense" embedding lookups (i.e., multiplication by a one-hot vector). prac- tice, sparse embedding lookups much slower than constant time; some hardware accelerators dense operations are actually faster than sparse lookups.
F ADVERSARIAL TRAINING
Here we detail attempts to adversarially train the generator instead using maximum likelihood. In particular train the generator G' to maximize the discriminator loss L As discriminator isn't 't precisely the same as the discriminator of GAN (see the discussion Section ( this method really an instance of Adversarial Contrastive Estimation (Bose et al., 2018) rather than Generative Adversarial Training.
is not possible to adversarially train the generator by back-propagating through the discriminator (c.g., as in GAN trained on images) due the discrete sampling from the generator, so we use reinforcement learning instead.
Our generator is different from most text generation models in that it is non-autogregressivv: predic- tions are made independently. In other words, rather than taking a sequence of actions where each action generates a token, the generator takes single giant action of generating all tokens simulta- neously, where the probability for the action factorizes as the product of generator probabilities for each token. deal with this enormous action space, we make the following simplifying assumption: that the discriminator's prediction D commpt t) depends only on the token H; and the non-replaced
See https::/www.tett https//www....oooooooooooooooo//
16

---

Published as a conference paper at ICLR 2020
tokens (x, m). i.e., it does not depend other generated tokens {2, E m A t). This isn't 't too bad of an assumption because a relatively small number tokens are replaced, and it greatly simplifies credit assignment when using reinforcement learning. Notationally, this assumption by (in a slight abuse of notation) by writing D a: for the discriminator predicting whether the generated token it equals the original token 2t given the masked context x masked A useful consequence of this assumption is that the discriminator score for non-replaced tokens (D m for t & m) independent of PG because we are assuming it does depend on any replaced token. Therefore these tokens can be ignored when training G to maximize Lpise- During training we seek find
show
arg max Dise arg 8G
Σ 1=1
1 comupt Iz) log (x t)-
Pc
a.m.
1 12, ( * xz) log(1 Do t))
Using the simplifying assumption, we approximate the above by finding the argmax of
I
Σ 1(2 = Zog ma
1(x; * 21) log(1 D / masked))
0t,,,
R())
tEm
= - log D ( ax: masked)
if i4 = xz otherwise
where R(i,, x)
log(1 1 ( misked))
In short, the simplifying assumption allows us to decompose the loss over the individual generated tokens. We cannot directly find arg maxes using gradient ascent because it is impossible to back- propagate through discrete sampling of ă. Instead, we use policy gradient reinforcement learning (Williams 1992). particular, we use the REINFORCE gradient
Vea Ciis Rd
E
E 2e-pe
Vo, log (āt masked [R((tr, b(x masked t)]
TEm
Where 6 is learned baseline implemented as max t) log sigmoid (w" ha (asked) ).)
where la maskee)) are the outputs the generator's Transformer encoder. The baseline trained with cross-entropy loss to match the reward for the corresponding position. We approximate the expectations with a single sample and learn 6G with gradient ascent. Despite receiving no explicit feedback about which generated tokens correct, found the adversarial training resulted in a fairly accurate generator (for a 256-hidden-size generator, the adversarially trained one achieves 58% accuracy masked language modeling while the same sized MIE generator gets 65%).. How- ever, using this generator did not improve over the MLE-trained one on downstream tasks (see the right of Figure in the main paper).
G EVALUATING ELECTRA AS A MASKED LANGUAGE MODEL
This sections details some initial experiments in evaluating ELECTRA as masked language model. Using slightly different notation from the main paper, given a context consisting of text sequence with one token z masked-out, the discriminator loss can be written as
LDisc
Σ
((1 ((1- Proas Paama ) (x/cc) log D(x, c) +
//unmasked token
2 zEvocab
ProoxPaana (2/c)) log D(x, c) +
//generator samples correct token
Pmnas
Pdata (z c)) (z()) log(1 - D(x, e)))
//generator samples incorrect token
Finding the critical points of this loss with respect to D shows that for a fixed generator the optimal discriminator
D(a, c) (x) pa (c (xlc) (xlc)
17

---

Published as a conference paper at ICLR 2020
which means
Pamaa (x)c)= D(a, c)pc(clc)((a(1-- D(x, c)) + pa(x(c))
where a (1 - Pmmss)/ Pumsk is the number of unmasked tokens for every masked token. We this expression evaluate ELECTRA argmaxxEvecab (2r, c)p((())) c)/(a(1 - D(x, c)) + pe (x(c)) as the model's prediction for given context. In practice, selecting over the whole vocabulary very expensive, so we instead take the argmax over the top 100 predictions from the generator Tt] Using this method, we compared dataset. We found that BERT slightly outperformed ELECTRA at masked language modeling (77.9% vs 75.5% accuracy). It possible that the assumption of an optimal discriminator, which is certainly far from correct, is harming ELECTRA'S accuracy under this evaluation scheme. However, perhaps it is not too surpris- ing that a model like BERT that is trained specifically for generation performs better at generation while model with discriminative objective like ELECTRA is better at being fine-tumed on dis- criminative tasks. We think comparisons BERT's and ELECTRA's MLM predictions might be an interesting way uncover more about the differences between ELECTRA and BERT encoders in future work.
masked language model by selecting
ELECTRA-Base and BERT-Base on the Wikkieedia-BooksCoppp
H NEGATIVE RESULTS
We briefly describe a few ideas that did not look promising in our initial experiments:
We initially attempted make BERT more efficient by strategically masking-out tokens (e.g., masking our rarer tokens more frequently, or training model to guess which tokens BERT would struggle to predict they were masked out). This resulted fairly minor speedups over regular BERT.
Given that ELECTRA seemed to benefit (up to certain point) from having a weaker gener- ator (see Section 3.2), we we explored raising the temperature of the generator output softmax disallowing the generator from sampling the correct token. Neither of these improved results.
We tried adding a sentence-level contrastive objective. For this task. kept 20% of input sentences unchanged rather than noising them with the generator. We then added predic- tion head to the model that predicted if the entire input was corrupted or not. Surprisingly, this slightly decreased scores on downstream tasks.
1oFor ELECCRRA-Base this means the upper-hound for accuracy is around 95%.
18


Reviews:
[{"review_id": "r1xUDhfRtB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposed a novel sample-efficient pretraining task. One inefficiency of BERT is that only 15% tokens are used for training in each example. The paper introduced a generator+discriminator framework to optimize the utility of training examples. The generator task is the MLM which predicts the masked word. The author adds a discriminator to further learn from the example by classifying each word to be either generated or original. In this way, more words can be used. This method looks as only adding the discrimination task after BERT pretraining task. But, the authors later show that the best GLUE scores can be obtained only when both generator and discriminator are co-trained. Moreover, the adversarial ELECTRA perform worse. All these observations are interesting. It will be helpful if the authors provide more empirical analysis why the adversarial ELECTRA perform worse or failed. Is it because the GAN is hard to train or the adversarial task doesn't fit the pretraining? \n\nOverall, I think this is a good paper. The studied problem is important, the idea is new and the experimental results are positive. Specifically, it shows that ELECTRA can outperforms BERT and match RoBERTa with less training time. But, it will be a big plus if the authors can show ELECTRA can outperform RoBERTa with the same amount of training time. Analysis are also provided to give audience insights in this method.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "SJgKAmShKH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: Authors offer an alternative for masked LM pretraining that's more sample-efficient called replaced token detection. Their method basically replaces certain input tokens with alternatives which are sampled from a generator and train a discriminative model to determine whether its generated or real. The work shows empirical success getting better results than GPT with a fraction of the compute on GLUE and others.\n\nPositives: Idea is simple and makes sense intuitively, but not something one would think immediately would work better with a such a small fraction of the compute. I think the formulations of the experiments and ideas to develop this are adequate.\n\nConcerns & Questions: I'd like to see a little more investigation into Table 3. I don't have intuition over why these results are the way that they are and the text nor the experimentation really gives me an indication. How well does this model work with very very little compute; lets say you have only a couple of gpu hours. Whats the degradation in performance?\n\nOverall I'd like to see more clarity in the overall analysis because I'm still unsure how to interpret your results on the why certain choices/experimental groups get the performance numbers they get.\n\n------------------------------------------------------------------------------------------------------------------------\n\nAfter the author response, I have changed my score to a 6. I think the paper merits acceptance.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BkxelNPJqB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose replaced token detection, a novel self-supervised task, for learning text representations.\n\nThe principle advantage of the approach is that, in contrast with the standard masked language model (MLM) objective used by BERT and derivatives, there is a training signal for all tokens of the input (rather than a small fraction, when 10-20% of the input tokens are masked and then reconstructed under the MLM objective).\n\nA smaller MLE-trained BERT-style generator is used to replace masked words with plausible alternatives, which the ELECTRA discriminator (the part that is retained and finetuned on downstream tasks) must detect (unmodified word slots are also in the objective, and must be detected as such).\n\nIn general the paper reads well, and the authors present ablations to reveal the source of gains. ELECTRA matches the performance of RobBERTa on the popular GLUE NLP task, with just 1/4 of the training compute.\n\nStrengths:\n-Simple but novel self-supervised task for learning text representations, strong results, adequate ablation.\n\nLimitations:\n-The authors limit their investigation of downstream performance to the GLUE set of tasks, which are classification tasks. This is a significant limitation of the current version of the paper, as it may be that replaced token detection is more suitable for these tasks, but inferior to MLM (a higher precision self-supervised task) for more involved tasks like question answering. The latter is arguably of much higher importance to the NLP research community at this point, and some consider the GLUE task to be essentially solved for all practical purposes, given inherent noise levels.\n-In contrast with BERT, there is no mention of any plan to release ELECTRA (big or small versions), which is a disappointment, lowers the significance of the work\n\nOverall:\nAn okay paper. Results on SQUAD or another more elaborate NLP task and/or the release of the ELECTRA models would make the paper much stronger.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper investigates the tasks used to pretrain language models. The paper proposes not using a generative tasks ('filling in' masked tokens), but instead a discriminative tasked (recognising corrupted tokens). The authors empirically show that the proposed method leads to improved performance, especially in the "limited compute" regime. 

Initially, the reviewers had quite split opinions on the paper, but after the rebuttal and discussion phases all reviewers agreed on an "accept" recommendation. I am happy to agree with this recommendation based on the following observations:
- The authors provide strong empirical results including relevant ablations. Reviews initially suggested a limitation to classification tasks and a lack of empirical analysis, but those issues have been addressed in the updated version. 
- The problem of pre-training language model is relevant for the ML and NLP communities, and it should be especially relevant for ICLR. The resulting method significantly outperforms existing methods, especially in the low compute regime. 
- The idea is quite simple, but at the same time it seems to be a quite novel idea. 
Decision: Accept (Poster)