Title: ELECTRA: PRE-TRAINING TEXT ENCODERS AS DISCRIMINATORS RATHER THAN GENERATORS

Abstract: Masked language modeling (MLM) pre-training methods such as BERT corrupt the input by replacing some tokens with [MASK] and then train a model to reconstruct the original tokens. While they produce good results when transferred to downstream NLP tasks, they generally require large amounts of compute to be effective. As an alternative, we propose a more sample-efficient pre-training task called replaced token detection. Instead of masking the input, our approach corrupts it by replacing some tokens with plausible alternatives sampled from a small generator network. Then, instead of training a model that predicts the original identities of the corrupted tokens, we train a discriminative model that predicts whether each token in the corrupted input was replaced by a generator sample or not. Thorough experiments demonstrate this new pre-training task is more efficient than MLM because the task is defined over all input tokens rather than just the small subset that was masked out. As a result, the contextual representations learned by our approach substantially outperform the ones learned by BERT given the same model size, data, and compute. The gains are particularly strong for small models; for example, we train a model on one GPU for 4 days that outperforms GPT (trained using 30x more compute) on the GLUE natural language understanding benchmark. Our approach also works well at scale, where it performs comparably to RoBERTa and XLNet while using less than 1/4 of their compute and outperforms them when using the same amount of compute.

1. INTRODUCTION:

Current state-of-the-art representation learning methods for language can be viewed as learning denoising autoencoders (Vincent et al., 2008) . They select a small subset of the unlabeled input sequence (typically 15%), mask the identities of those tokens (e.g., BERT; Devlin et al. (2019) ) or attention to those tokens (e.g., XLNet; Yang et al. ( 2019)), and then train the network to recover the original input. While more effective than conventional language-model pre-training due to learning bidirectional representations, these masked language modeling (MLM) approaches incur a substantial compute cost because the network only learns from 15% of the tokens per example.

As an alternative, we propose replaced token detection, a pre-training task in which the model learns to distinguish real input tokens from plausible but synthetically generated replacements. Instead of masking, our method corrupts the input by replacing some tokens with samples from a proposal distribution, which is typically the output of a small masked language model. This corruption procedure solves a mismatch in BERT (although not in XLNet) where the network sees artificial [MASK] tokens during pre-training but not when being fine-tuned on downstream tasks. We then pre-train the network as a discriminator that predicts for every token whether it is an original or a replacement. In contrast, MLM trains the network as a generator that predicts the original identities of the corrupted tokens. A key advantage of our discriminative task is that the model learns from all input tokens instead of just the small masked-out subset, making it more computationally efficient. Although our approach is reminiscent of training the discriminator of a GAN, our method is not adversarial in that the generator producing corrupted tokens is trained with maximum likelihood due to the difficulty of applying GANs to text (Caccia et al., 2018) .

We call our approach ELECTRA1 for "Efficiently Learning an Encoder that Classifies Token Replacements Accurately." As in prior work, we apply it to pre-train Transformer text encoders (Vaswani et al., 2017) that can be fine-tuned on downstream tasks. Through a series of ablations, we show that learning from all input positions causes ELECTRA to train much faster than BERT. We also show ELECTRA achieves higher accuracy on downstream tasks when fully trained.

Most current pre-training methods require large amounts of compute to be effective, raising concerns about their cost and accessibility. Since pre-training with more compute almost always results in better downstream accuracies, we argue an important consideration for pre-training methods should be compute efficiency as well as absolute downstream performance. From this viewpoint, we train ELECTRA models of various sizes and evaluate their downstream performance vs. their compute requirement. In particular, we run experiments on the GLUE natural language understanding benchmark (Wang et al., 2019) and SQuAD question answering benchmark (Rajpurkar et al., 2016) . ELECTRA substantially outperforms MLM-based methods such as BERT and XLNet given the same model size, data, and compute (see Figure 1 ). For example, we build an ELECTRA-Small model that can be trained on 1 GPU in 4 days.2 ELECTRA-Small outperforms a comparably small BERT model by 5 points on GLUE, and even outperforms the much larger GPT model (Radford et al., 2018) . Our approach also works well at large scale, where we train an ELECTRA-Large model that performs comparably to RoBERTa (Liu et al., 2019) and XLNet (Yang et al., 2019), despite having fewer parameters and using 1/4 of the compute for training. Training ELECTRA-Large further results in an even stronger model that outperforms ALBERT (Lan et al., 2019) on GLUE and sets a new state-of-the-art for SQuAD 2.0. Taken together, our results indicate that the discriminative task of distinguishing real data from challenging negative samples is more compute-efficient and parameter-efficient than existing generative approaches for language representation learning.

2. METHOD:

We first describe the replaced token detection pre-training task; see Figure 2 for an overview. We suggest and evaluate several modeling improvements for this method in Section 3.2. The generator can be any model that produces an output distribution over tokens, but we usually use a small masked language model that is trained jointly with the discriminator. Although the models are structured like in a GAN, we train the generator with maximum likelihood rather than adversarially due to the difficulty of applying GANs to text. After pre-training, we throw out the generator and only fine-tune the discriminator (the ELECTRA model) on downstream tasks.

Our approach trains two neural networks, a generator G and a discriminator D. Each one primarily consists of an encoder (e.g., a Transformer network) that maps a sequence on input tokens x = [x 1 , ..., x n ] into a sequence of contextualized vector representations h(x) = [h 1 , ..., h n ]. For a given position t, (in our case only positions where x t = [MASK]), the generator outputs a probability for generating a particular token x t with a softmax layer:

p G (x t |x) = exp e(x t ) T h G (x) t / x exp e(x ) T h G (x) t

where e denotes token embeddings. For a given position t, the discriminator predicts whether the token x t is "real," i.e., that it comes from the data rather than the generator distribution, with a sigmoid output layer:

D(x, t) = sigmoid(w T h D (x) t )

The generator is trained to perform masked language modeling (MLM). Given an input x = [x 1 , x 2 , ..., x n ], MLM first select a random set of positions (integers between 1 and n) to mask out m = [m 1 , ..., m k ]. 3 The tokens in the selected positions are replaced with a [MASK] token: we denote this as x masked = REPLACE(x, m, [MASK]). The generator then learns to predict the original identities of the masked-out tokens. The discriminator is trained to distinguish tokens in the data from tokens that have been replaced by generator samples. More specifically, we create a corrupted example x corrupt by replacing the masked-out tokens with generator samples and train the discriminator to predict which tokens in x corrupt match the original input x. Formally, model inputs are constructed according to

m i ∼ unif{1, n} for i = 1 to k x masked = REPLACE(x, m, [MASK]) xi ∼ p G (x i |x masked ) for i ∈ m x corrupt = REPLACE(x, m, x)

and the loss functions are

L MLM (x, θ G ) = E i∈m -log p G (x i |x masked ) L Disc (x, θ D ) = E n t=1 -1(x corrupt t = x t ) log D(x corrupt , t) -1(x corrupt t = x t ) log(1 -D(x corrupt , t))

Although similar to the training objective of a GAN, there are several key differences. First, if the generator happens to generate the correct token, that token is considered "real" instead of "fake"; we found this formulation to moderately improve results on downstream tasks. More importantly, the generator is trained with maximum likelihood rather than being trained adversarially to fool the discriminator. Adversarially training the generator is challenging because it is impossible to backpropagate through sampling from the generator. Although we experimented circumventing this issue by using reinforcement learning to train the generator (see Appendix F), this performed worse than maximum-likelihood training. Lastly, we do not supply the generator with a noise vector as input, as is typical with a GAN.

We minimize the combined loss

min θ G ,θ D x∈X L MLM (x, θ G ) + λL Disc (x, θ D )

over a large corpus X of raw text. We approximate the expectations in the losses with a single sample. We don't back-propagate the discriminator loss through the generator (indeed, we can't because of the sampling step). After pre-training, we throw out the generator and fine-tune the discriminator on downstream tasks.

3. EXPERIMENTS:



3.1. EXPERIMENTAL SETUP:

We evaluate on the General Language Understanding Evaluation (GLUE) benchmark (Wang et al., 2019) and Stanford Question Answering (SQuAD) dataset (Rajpurkar et al., 2016) . GLUE contains a variety of tasks covering textual entailment (RTE and MNLI) question-answer entailment (QNLI), paraphrase (MRPC), question paraphrase (QQP), textual similarity (STS), sentiment (SST), and linguistic acceptability (CoLA). See Appendix C for more details on the GLUE tasks. Our evaluation metrics are Spearman correlation for STS, Matthews correlation for CoLA, and accuracy for the other GLUE tasks; we generally report the average score over all tasks. For SQuAD, we evaluate on versions 1.1, in which models select the span of text answering a question, and 2.0, in which some questions are unanswerable by the passage. We use the standard evaluation metrics of Exact-Match (EM) and F1 scores. For most experiments we pre-train on the same data as BERT, which consists of 3.3 Billion tokens from Wikipedia and BooksCorpus (Zhu et al., 2015) . However, for our Large model we pre-trained on the data used for XLNet (Yang et al., 2019) , which extends the BERT dataset to 33B tokens by including data from ClueWeb (Callan et al., 2009) , CommonCrawl, and Gigaword (Parker et al., 2011) . All of the pre-training and evaluation is on English data, although we think it would be interesting to apply our methods to multilingual data in the future.

Our model architecture and most hyperparameters are the same as BERT's. For fine-tuning on GLUE, we add simple linear classifiers on top of ELECTRA. For SQuAD, we add the questionanswering module from XLNet on top of ELECTRA, which is slightly more sophisticated than BERT's in that it jointly rather than independently predicts the start and end positions and has a "answerability" classifier added for SQuAD 2.0. Some of our evaluation datasets are small, which means accuracies of fine-tuned models can vary substantially depending on the random seed. We therefore report the median of 10 fine-tuning runs from the same pre-trained checkpoint for each result. Unless stated otherwise, results are on the dev set. See the appendix for further training details and hyperparameter values.

3.2. MODEL EXTENSIONS:

We improve our method by proposing and evaluating several extensions to the model. Unless stated otherwise, these experiments use the same model size and training data as BERT-Base.

Weight Sharing We propose improving the efficiency of the pre-training by sharing weights between the generator and discriminator. If the generator and discriminator are the same size, all of the transformer weights can be tied. However, we found it to be more efficient to have a small generator, in which case we only share the embeddings (both the token and positional embeddings) of the generator and discriminator. In this case we use embeddings the size of the discriminator's hidden states. 4 The "input" and "output" token embeddings of the generator are always tied as in BERT.

We compare the weight tying strategies when the generator is the same size as the discriminator.

We train these models for 500k steps. GLUE scores are 83.6 for no weight tying, 84.3 for tying token embeddings, and 84.4 for tying all weights. We hypothesize that ELECTRA benefits from tied token embeddings because masked language modeling is particularly effective at learning these representations: while the discriminator only updates tokens that are present in the input or are sampled by the generator, the generator's softmax over the vocabulary densely updates all token embeddings. On the other hand, tying all encoder weights caused little improvement while incurring the significant disadvantage of requiring the generator and discriminator to be the same size. Based on these findings, we use tied embeddings for further experiments in this paper.

Smaller Generators If the generator and discriminator are the same size, training ELECTRA would take around twice as much compute per step as training only with masked language modeling. We suggest using a smaller generator to reduce this factor. Specifically, we make models smaller by decreasing the layer sizes while keeping the other hyperparameters constant. We also explore using an extremely simple "unigram" generator that samples fake tokens according their frequency in the train corpus. GLUE scores for differently-sized generators and discriminators are shown in the left of Figure 3 . All models are trained for 500k steps, which puts the smaller generators at a disadvantage in terms of compute because they require less compute per training step. Nevertheless, we find that models work best with generators 1/4-1/2 the size of the discriminator. We speculate that having too strong of a generator may pose a too-challenging task for the discriminator, preventing it from learning as effectively. In particular, the discriminator may have to use many of its parameters modeling the generator rather than the actual data distribution. Further experiments in this paper use the best generator size found for the given discriminator size.

Training Algorithms Lastly, we explore other training algorithms for ELECTRA, although these did not end up improving results. The proposed training objective jointly trains the generator and discriminator. We experiment with instead using the following two-stage training procedure:

1. Train only the generator with L MLM for n steps.

2. Initialize the weights of the discriminator with the weights of the generator. Then train the discriminator with L Disc for n steps, keeping the generator's weights frozen.

Note that the weight initialization in this procedure requires having the same size for the generator and discriminator. We found that without the weight initialization the discriminator would sometimes fail to learn at all beyond the majority class, perhaps because the generator started so far ahead of the discriminator. Joint training on the other hand naturally provides a curriculum for the discriminator where the generator starts off weak but gets better throughout training. We also explored training the generator adversarially as in a GAN, using reinforcement learning to accommodate the discrete operations of sampling from the generator. See Appendix F for details.

Results are shown in the right of Figure 3 problems with adversarial training. First, the adversarial generator is simply worse at masked language modeling; it achieves 58% accuracy at masked language modeling compared to 65% accuracy for an MLE-trained one. We believe the worse accuracy is mainly due to the poor sample efficiency of reinforcement learning when working in the large action space of generating text. Secondly, the adversarially trained generator produces a low-entropy output distribution where most of the probability mass is on a single token, which means there is not much diversity in the generator samples.

Both of these problems have been observed in GANs for text in prior work (Caccia et al., 2018) .

3.3. SMALL MODELS:

As a goal of this work is to improve the efficiency of pre-training, we develop a small model that can be quickly trained on a single GPU. Starting with the BERT-Base hyperparameters, we shortened the sequence length (from 512 to 128), reduced the batch size (from 256 to 128), reduced the model's hidden dimension size (from 768 to 256), and used smaller token embeddings (from 768 to 128). To provide a fair comparison, we also train a BERT-Small model using the same hyperparameters. We train BERT-Small for 1.5M steps, so it uses the same training FLOPs as ELECTRA-Small, which was trained for 1M steps. 5 In addition to BERT, we compare against two less resource-intensive pre-training methods based on language modeling: ELMo (Peters et al., 2018) and GPT (Radford et al., 2018) . 6 We also show results for a base-sized ELECTRA model comparable to BERT-Base.

Results are shown in Table 1 . See Appendix D for additional results, including stronger small-sized and base-sized models trained with more compute. ELECTRA-Small performs remarkably well given its size, achieving a higher GLUE score than other methods using substantially more compute and parameters. For example, it scores 5 points higher than a comparable BERT-Small model and even outperforms the much larger GPT model. ELECTRA-Small is trained mostly to convergence, with models trained for even less time (as little as 6 hours) still achieving reasonable performance. While small models distilled from larger pre-trained transformers can also achieve good GLUE scores (Sun et al., 2019b; Jiao et al., 2019) , these models require first expending substantial compute to pre-train the larger teacher model. The results also demonstrate the strength of ELECTRA at a moderate size; our base-sized ELECTRA model substantially outperforms BERT-Base and even outperforms BERT-Large (which gets 84.0 GLUE score). We hope ELECTRA's ability to achieve strong results with relatively little compute will broaden the accessibility of developing and applying pre-trained models in NLP. ). Some models do not have QNLI scores because they treat QNLI as a ranking task, which has recently been disallowed by the GLUE benchmark. To compare against these models, we report the average score excluding QNLI (Avg.*) in addition to the GLUE leaderboard score (Score). "ELECTRA" and "RoBERTa" refer to the fully-trained ELECTRA-1.75M and RoBERTa-500K models.

3.4. LARGE MODELS:

We train big ELECTRA models to measure the effectiveness of the replaced token detection pretraining task at the large scale of current state-of-the-art pre-trained Transformers. Our ELECTRA-Large models are the same size as BERT-Large but are trained for much longer. In particular, we train a model for 400k steps (ELECTRA-400K; roughly 1/4 the pre-training compute of RoBERTa) and one for 1.75M steps (ELECTRA-1.75M; similar compute to RoBERTa). We use a batch size 2048 and the XLNet pre-training data. We note that although the XLNet data is similar to the data used to train RoBERTa, the comparison is not entirely direct. As a baseline, we trained our own BERT-Large model using the same hyperparameters and training time as ELECTRA-400K.

Results on the GLUE dev set are shown in Table 2 . ELECTRA-400K performs comparably to RoBERTa and XLNet. However, it took less than 1/4 of the compute to train ELECTRA-400K as it did to train RoBERTa and XLNet, demonstrating that ELECTRA's sample-efficiency gains hold at large scale. Training ELECTRA for longer (ELECTRA-1.75M) results in a model that outscores them on most GLUE tasks while still requiring less pre-training compute. Surprisingly, our baseline BERT model scores notably worse than RoBERTa-100K, suggesting our models may benefit from more hyperparameter tuning or using the RoBERTa training data. ELECTRA's gains hold on the GLUE test set (see Table 3 ), although these comparisons are less apples-to-apples due to the additional tricks employed by the models (see Appendix B).

Results on SQuAD are shown in Table 4 . Consistent, with the GLUE results, ELECTRA scores better than masked-language-modeling-based methods given the same compute resources. For example, ELECTRA-400K outperforms RoBERTa-100k and our BERT baseline, which use similar amounts of pre-training compute. ELECTRA-400K also performs comparably to RoBERTa-500K despite using less than 1/4th of the compute. Unsurprisingly, training ELECTRA longer improves results further: ELECTRA-1.75M scores higher than previous models on the SQuAD 2.0 bench- mark. ELECTRA-Base also yields strong results, scoring substantially better than BERT-Base and XLNet-Base, and even surpassing BERT-Large according to most metrics. ELECTRA generally performs better at SQuAD 2.0 than 1.1. Perhaps replaced token detection, in which the model distinguishes real tokens from plausible fakes, is particularly transferable to the answerability classification of SQuAD 2.0, in which the model must distinguish answerable questions from fake unanswerable questions.

3.5. EFFICIENCY ANALYSIS:

We have suggested that posing the training objective over a small subset of tokens makes masked language modeling inefficient. However, it isn't entirely obvious that this is the case. After all, the model still receives a large number of input tokens even though it predicts only a small number of masked tokens. To better understand where the gains from ELECTRA are coming from, we compare a series of other pre-training objectives that are designed to be a set of "stepping stones" between BERT and ELECTRA.

• ELECTRA 15%: This model is identical to ELECTRA except the discriminator loss only comes from the 15% of the tokens that were masked out of the input. In other words, the sum in the discriminator loss L Disc is over i ∈ m instead of from 1 to n.7 • Replace MLM: This objective is the same as masked language modeling except instead of replacing masked-out tokens with [MASK], they are replaced with tokens from a generator model. This objective tests to what extent ELECTRA's gains come from solving the discrepancy of exposing the model to [MASK] tokens during pre-training but not fine-tuning. • All-Tokens MLM: Like in Replace MLM, masked tokens are replaced with generator samples. Furthermore, the model predicts the identity of all tokens in the input, not just ones that were masked out. We found it improved results to train this model with an explicit copy mechanism that outputs a copy probability D for each token using a sigmoid layer. The model's output distribution puts D weight on the input token plus 1 -D times the output of the MLM softmax. This model is essentially a combination of BERT and ELEC-TRA. Note that without generator replacements, the model would trivially learn to make predictions from the vocabulary for [MASK] tokens and copy the input for other ones.

Results are shown in Table 5 . First, we find that ELECTRA is greatly benefiting from having a loss defined over all input tokens rather than just a subset: ELECTRA 15% performs much worse than ELECTRA. Secondly, we find that BERT performance is being slightly harmed from the pre-train fine-tune mismatch from [MASK] tokens, as Replace MLM slightly outperforms BERT. We note that BERT (including our implementation) already includes a trick to help with the pre-train/finetune discrepancy: masked tokens are replaced with a random token 10% of the time and are kept the same 10% of the time. However, our results suggest these simple heuristics are insufficient to fully solve the issue. Lastly, we find that All-Tokens MLM, the generative model that makes predictions over all tokens instead of a subset, closes most of the gap between BERT and ELECTRA. In total, these results suggest a large amount of ELECTRA's improvement can be attributed to learning from all tokens and a smaller amount can be attributed to alleviating the pre-train fine-tune mismatch.

The improvement of ELECTRA over All-Tokens MLM suggests that the ELECTRA's gains come from more than just faster training. We study this further by comparing BERT to ELECTRA for various model sizes (see Figure 4 , left). We find that the gains from ELECTRA grow larger as the models get smaller. The small models are trained fully to convergence (see Figure 4 , right), showing that ELECTRA achieves higher downstream accuracy than BERT when fully trained. We speculate that ELECTRA is more parameter-efficient than BERT because it does not have to model the full distribution of possible tokens at each position, but we believe more analysis is needed to completely explain ELECTRA's parameter efficiency.

4. RELATED WORK:

Self-Supervised Pre-training for NLP Self-supervised learning has been used to learn word representations (Collobert et al., 2011; Pennington et al., 2014) and more recently contextual representations of words though objectives such as language modeling (Dai & Le, 2015; Peters et al., 2018; Howard & Ruder, 2018) . BERT (Devlin et al., 2019) pre-trains a large Transformer (Vaswani et al., 2017) at the masked-language modeling task. There have been numerous extensions to BERT. For example, MASS (Song et al., 2019) and UniLM (Dong et al., 2019) extend BERT to generation tasks by adding auto-regressive generative training objectives. ERNIE (Sun et al., 2019a) and SpanBERT (Joshi et al., 2019) mask out contiguous sequences of token for improved span representations. This idea may be complementary to ELECTRA; we think it would be interesting to make ELECTRA's generator auto-regressive and add a "replaced span detection" task. Instead of masking out input tokens, XLNet (Yang et al., 2019) masks attention weights such that the input sequence is autoregressively generated in a random order. However, this method suffers from the same inefficiencies as BERT because XLNet only generates 15% of the input tokens in this way. Like ELECTRA, XL-Net may alleviate BERT's pretrain-finetune discrepancy by not requiring [MASK] tokens, although this isn't entirely clear because XLNet uses two "streams" of attention during pre-training but only one for fine-tuning. Recently, models such as TinyBERT (Jiao et al., 2019) and MobileBERT (Sun et al., 2019b) show that BERT can effectively be distilled down to a smaller model. In contrast, we focus more on pre-training speed rather than inference speed, so we train ELECTRA-Small from scratch.

Generative Adversarial Networks GANs (Goodfellow et al., 2014) are effective at generating high-quality synthetic data. Radford et al. (2016) propose using the discriminator of a GAN in downstream tasks, which is similar to our method. GANs have been applied to text data (Yu et al., 2017; Zhang et al., 2017) , although state-of-the-art approaches still lag behind standard maximumlikelihood training (Caccia et al., 2018; Tevet et al., 2018) . Although we do not use adversarial learning, our generator is particularly reminiscent of MaskGAN (Fedus et al., 2018) , which trains the generator to fill in tokens deleted from the input.

Contrastive Learning Broadly, contrastive learning methods distinguish observed data points from fictitious negative samples. They have been applied to many modalities including text (Smith & Eisner, 2005) , images (Chopra et al., 2005) , and video (Wang & Gupta, 2015; Sermanet et al., 2017) data. Common approaches learn embedding spaces where related data points are similar (Saunshi et al., 2019) or models that rank real data points over negative samples (Collobert et al., 2011; Bordes et al., 2013) . ELECTRA is particularly related to Noise-Contrastive Estimation (NCE) (Gutmann & Hyvärinen, 2010) , which also trains a binary classifier to distinguish real and fake data points.

Word2Vec (Mikolov et al., 2013) , one of the earliest pre-training methods for NLP, uses contrastive learning. In fact, ELECTRA can be viewed as a massively scaled-up version of Continuous Bagof-Words (CBOW) with Negative Sampling. CBOW also predicts an input token given surrounding context and negative sampling rephrases the learning task as a binary classification task on whether the input token comes from the data or proposal distribution. However, CBOW uses a bag-ofvectors encoder rather than a transformer and a simple proposal distribution derived from unigram token frequencies instead of a learned generator.

5. CONCLUSION:

We have proposed replaced token detection, a new self-supervised task for language representation learning. The key idea is training a text encoder to distinguish input tokens from high-quality negative samples produced by an small generator network. Compared to masked language modeling, our pre-training objective is more compute-efficient and results in better performance on downstream tasks. It works well even when using relatively small amounts of compute, which we hope will make developing and applying pre-trained text encoders more accessible to researchers and practitioners with less access to computing resources. We also hope more future work on NLP pre-training will consider efficiency as well as absolute performance, and follow our effort in reporting compute usage and parameter counts along with evaluation metrics.

Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8(3-4):229-256, 1992.

Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. XLNet: Generalized autoregressive pretraining for language understanding. In NeurIPS, 2019.

Lantao Yu, Weinan Zhang, Jun Wang, and Yingrui Yu. SeqGAN: Sequence generative adversarial nets with policy gradient. In AAAI, 2017.

Yizhe Zhang, Zhe Gan, Kai Fan, Zhi Chen, Ricardo Henao, Dinghan Shen, and Lawrence Carin. Adversarial feature matching for text generation. In ICML, 2017.

Yukun Zhu, Ryan Kiros, Richard S. Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. ICCV, 2015.

A PRE-TRAINING DETAILS:

The following details apply to both our ELECTRA models and BERT baselines. We mostly use the same hyperparameters as BERT. We set λ, the weight for the discriminator objective in the loss to 50. 8 We use dynamic token masking with the masked positions decided on-the-fly instead of during preprocessing. Also, we did not use the next sentence prediction objective proposed in the original BERT paper, as recent work has suggested it does not improve scores (Yang et al., 2019; Liu et al., 2019) . For our ELECTRA-Large model, we used a higher mask percent (25 instead of 15) because we noticed the generator was achieving high accuracy with 15% masking, resulting in very few replaced tokens. We searched for the best learning rate for the Base and Small models out of [1e-4, 2e-4, 3e-4, 5e-4] and selected λ out of [1, 10, 20, 50, 100] in early experiments. Otherwise we did no hyperparameter tuning beyond the experiments in Section 3.2. The full set of hyperparameters are listed in Table 6 .

B FINE-TUNING DETAILS:

For Large-sized models, we used the hyperparameters from Clark et al. (2019) for the most part. However, after noticing that RoBERTa (Liu et al., 2019) uses more training epochs (up to 10 rather than 3) we searched for the best number of train epochs out of [10, 3] for each task. For SQuAD, we decreased the number of train epochs to 2 to be consistent with BERT and RoBERTa. For Basesized models we searched for a learning rate out of [3e-5, 5e-5, 1e-4, 1.5e-4] and the layer-wise learning-rate decay out of [0.9, 0.8, 0.7], but otherwise used the same hyperparameters as for Large models. We found the small models benefit from a larger learning rate and searched for the best one out of [1e-4, 2e-4, 3e-4, 5e-3] . With the exception of number of train epochs, we used the same hyperparameters for all tasks. In contrast, previous research on GLUE such as BERT, XLNet, and RoBERTa separately searched for the best hyperparameters for each task. We expect our results would improve slightly if we performed the same sort of additional hyperparameter search. The full set of hyperparameters is listed in Table 7 .

Following BERT, we do not show results on the WNLI GLUE task for the dev set results, as it is difficult to beat even the majority classifier using a standard fine-tuning-as-classifier approach. For the GLUE test set results, we apply the standard tricks used by many of the GLUE leaderboard submissions including RoBERTa (Liu et al., 2019 ), XLNet (Yang et al., 2019) , and ALBERT (Lan et al., 2019) . Specifically:

• For RTE and STS we use intermediate task training (Phang et al., 2018) , starting from an ELECTRA checkpoint that has been fine-tuned on MNLI. For RTE, we found it helpful to combine this with a lower learning rate of 2e-5. 

E COUNTING FLOPS:

We chose to measure compute usage in terms of floating point operations (FLOPs) because it is a measure agnostic to the particular hardware, low-level optimizations, etc. However, it is worth noting that in some cases abstracting away hardware details is a drawback because hardware-centered optimizations can be key parts of a model's design, such as the speedup ALBERT (Lan et al., 2019) gets by tying weights and thus reducing communication overhead between TPU workers. We used TensorFlow's FLOP-counting capabilities9 and checked the results with by-hand computation. We made the following assumptions:

• An "operation" is a mathematical operation, not a machine instruction. For example, an exp is one op like an add, even though in practice the exp might be slower. We believe this assumption does not substantially change compute estimates because matrix-multiplies dominate the compute for most models. Similarly, we count matrix-multiplies as 2 * m * n FLOPs instead of m * n as one might if considering fused multiply-add operations. • The backwards pass takes the same number of FLOPs as the forward pass. This assumption is not exactly right (e.g., for softmax cross entropy loss the backward pass is faster), but importantly, the forward/backward pass FLOPs really are the same for matrix-multiplies, which is most of the compute anyway. • We assume "dense" embedding lookups (i.e., multiplication by a one-hot vector). In practice, sparse embedding lookups are much slower than constant time; on some hardware accelerators dense operations are actually faster than sparse lookups.

F ADVERSARIAL TRAINING:

Here we detail attempts to adversarially train the generator instead of using maximum likelihood. In particular we train the generator G to maximize the discriminator loss L Disc . As our discriminator isn't precisely the same as the discriminator of a GAN (see the discussion in Section 2), this method is really an instance of Adversarial Contrastive Estimation (Bose et al., 2018) rather than Generative Adversarial Training. It is not possible to adversarially train the generator by back-propagating through the discriminator (e.g., as in a GAN trained on images) due to the discrete sampling from the generator, so we use reinforcement learning instead.

Our generator is different from most text generation models in that it is non-autogregressive: predictions are made independently. In other words, rather than taking a sequence of actions where each action generates a token, the generator takes a single giant action of generating all tokens simultaneously, where the probability for the action factorizes as the product of generator probabilities for each token. To deal with this enormous action space, we make the following simplifying assumption: that the discriminator's prediction D(x corrupt , t) depends only on the token x t and the non-replaced tokens {x i : i ∈ m}, i.e., it does not depend on other generated tokens {x i : i ∈ m ∧ i = t}. This isn't too bad of an assumption because a relatively small number of tokens are replaced, and it greatly simplifies credit assignment when using reinforcement learning. Notationally, we show this assumption by (in a slight abuse of notation) by writing D(x t |x masked ) for the discriminator predicting whether the generated token xt equals the original token x t given the masked context x masked . A useful consequence of this assumption is that the discriminator score for non-replaced tokens (D(x t |x masked ) for t ∈ m) is independent of p G because we are assuming it does not depend on any replaced token. Therefore these tokens can be ignored when training G to maximize L Disc .

During training we seek to find

arg max θ G L Disc = arg max θ G E x,m, x n t=1 -1(x corrupt t = x t ) log D(x corrupt , t)- 1(x corrupt t = x t ) log(1 -D(x corrupt , t))

Using the simplifying assumption, we approximate the above by finding the argmax of

E x,m, x t∈m -1(x t = x t ) log D(x|x masked ) -1(x t = x t ) log(1 -D(x|x masked )) = E x,m t∈m E xt∼pG R(x t , x) where R(x t , x) = -log D(x t |x masked ) if xt = x t -log(1 -D(x t |x masked )) otherwise

In short, the simplifying assumption allows us to decompose the loss over the individual generated tokens. We cannot directly find arg max θ G using gradient ascent because it is impossible to backpropagate through discrete sampling of x. Instead, we use policy gradient reinforcement learning (Williams, 1992) . In particular, we use the REINFORCE gradient

∇ θ G L Disc ≈ E x,m t∈m E xt∼pG ∇ θg log p G (x t |x masked )[R(x t , x) -b(x masked , t)]

Where b is a learned baseline implemented as b(x masked , t) = -log sigmoid(w T h G (x masked ) t ) where h G (x masked ) are the outputs of the generator's Transformer encoder. The baseline is trained with cross-entropy loss to match the reward for the corresponding position. We approximate the expectations with a single sample and learn θ G with gradient ascent. Despite receiving no explicit feedback about which generated tokens are correct, we found the adversarial training resulted in a fairly accurate generator (for a 256-hidden-size generator, the adversarially trained one achieves 58% accuracy at masked language modeling while the same sized MLE generator gets 65%). However, using this generator did not improve over the MLE-trained one on downstream tasks (see the right of Figure 3 in the main paper).

G EVALUATING ELECTRA AS A MASKED LANGUAGE MODEL:

This sections details some initial experiments in evaluating ELECTRA as a masked language model. Using slightly different notation from the main paper, given a context c consisting of a text sequence with one token x masked-out, the discriminator loss can be written as where a = (1 -p mask )/p mask is the number of unmasked tokens for every masked token. We can use this expression to evaluate ELECTRA as a masked language model by selecting argmax x∈vocab D(x, c)p G (x|c)/(a(1 -D(x, c)) + p G (x|c)) as the model's prediction for a given context. In practice, selecting over the whole vocabulary is very expensive, so we instead take the argmax over the top 100 predictions from the generator.10 Using this method, we compared ELECTRA-Base and BERT-Base on the Wikipedia+BooksCorpus dataset. We found that BERT slightly outperformed ELECTRA at masked language modeling (77.9% vs 75.5% accuracy). It is possible that the assumption of an optimal discriminator, which is certainly far from correct, is harming ELECTRA's accuracy under this evaluation scheme. However, perhaps it is not too surprising that a model like BERT that is trained specifically for generation performs better at generation while a model with a discriminative objective like ELECTRA is better at being fine-tuned on discriminative tasks. We think comparisons of BERT's and ELECTRA's MLM predictions might be an interesting way to uncover more about the differences between ELECTRA and BERT encoders in future work.

L Disc = -

H NEGATIVE RESULTS:

We briefly describe a few ideas that did not look promising in our initial experiments:

• We initially attempted to make BERT more efficient by strategically masking-out tokens (e.g., masking our rarer tokens more frequently, or training a model to guess which tokens BERT would struggle to predict if they were masked out). This resulted in fairly minor speedups over regular BERT.

• Given that ELECTRA seemed to benefit (up to a certain point) from having a weaker generator (see Section 3.2), we explored raising the temperature of the generator's output softmax or disallowing the generator from sampling the correct token. Neither of these improved results.

• We tried adding a sentence-level contrastive objective. For this task, we kept 20% of input sentences unchanged rather than noising them with the generator. We then added a prediction head to the model that predicted if the entire input was corrupted or not. Surprisingly, this slightly decreased scores on downstream tasks.

:

Code and pre-trained weights will be released at https://github.com/google-research/ electra

It has 1/20th the parameters and requires 1/135th the pre-training compute of BERT-Large.

Typically k = 0.15n , i.e., 15% of the tokens are masked out.

We add linear layers to the generator to project the embeddings into generator-hidden-sized representations.

ELECTRA requires more FLOPs per step because it consists of the generator as well as the discriminator.

GPT is similar in size to BERT-Base, but is trained for fewer steps.

We also trained a discriminator that learns from a random 15% of the input tokens distinct from the subset that was originally masked out; this model performed slightly worse.

As a binary classification task instead of the 30,000-way classification task in MLM, the discriminator's loss was typically much lower than the generator's.

See https://www.tensorflow.org/api_docs/python/tf/profiler

For ELECTRA-Base, this means the upper-bound for accuracy is around 95%.

Figures and Tables:

Figure 1: Replaced token detection pre-training consistently outperforms masked language model pre-training given the same compute budget. The left figure is a zoomed-in view of the dashed box.

Figure2: An overview of replaced token detection. The generator can be any model that produces an output distribution over tokens, but we usually use a small masked language model that is trained jointly with the discriminator. Although the models are structured like in a GAN, we train the generator with maximum likelihood rather than adversarially due to the difficulty of applying GANs to text. After pre-training, we throw out the generator and only fine-tune the discriminator (the ELECTRA model) on downstream tasks.

Figure 4: Left and Center: Comparison of BERT and ELECTRA for different model sizes. Right: A small ELECTRA model converges to higher downstream accuracy than BERT, showing the improvement comes from more than just faster training.

-p mask )p data (x|c) log D(x, c) + //unmasked token p mask p data (x|c)p G (x|c) log D(x, c) + //generator samples correct token p mask (1 -p data (x|c))p G (x|c) log(1 -D(x, c)) //generator samples incorrect token Finding the critical points of this loss with respect to D shows that for a fixed generator the optimal discriminator is D(x, c) = p data (x|c)(a + p G (x|c))/(ap data (x|c) + p G (x|c)) which means p data (x|c) = D(x, c)p G (x|c)/(a(1 -D(x, c)) + p G (x|c))

Left: GLUE scores for different generator/discriminator sizes (number of hidden units). Interestingly, having a generator smaller than the discriminator improves results. Right: Comparison of different training algorithms. As our focus is on efficiency, the x-axis shows FLOPs rather than train steps (e.g., ELECTRA is trained for fewer steps than BERT because it includes the generator).

Table content: <table><tr><td/><td>85</td><td/><td/><td colspan="4">Which generator size works best?</td><td/><td/><td colspan="3">Comparison of Training Algorithms</td></tr><tr><td/><td>84</td><td/><td/><td/><td/><td/><td/><td>84</td><td/><td/><td/></tr><tr><td/><td>83</td><td/><td/><td/><td/><td/><td/><td>82</td><td/><td/><td/></tr><tr><td>GLUE Score</td><td>81 82</td><td/><td/><td/><td/><td/><td>GLUE Score</td><td>78 80</td><td/><td/><td colspan="2">Switch to Discriminator Loss</td></tr><tr><td/><td>78 79 80</td><td>unigram</td><td>32</td><td>64</td><td>128 Generator Size</td><td>256</td><td>512 7681024 Discriminator Size 768 512 256</td><td>74 76</td><td>0</td><td>1</td><td>2</td><td>3 Pre-Train FLOPs 4 ELECTRA Adversarial ELECTRA 5 1e19 6 Two-Stage ELECTRA BERT</td></tr><tr><td colspan="3">Figure 3:</td><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr></table>

Comparison of small models on the GLUE dev set. BERT-Small/Base are our implementation and use the same hyperparameters as ELECTRA-Small/Base. Infer FLOPs assumes single length-128 input. Training times should be taken with a grain of salt as they are for different hardware and with sometimes un-optimized code. ELECTRA performs well even when trained on a single GPU, scoring 5 GLUE points higher than a comparable BERT model and even outscoring the much larger GPT model.

Table content: <table><tr><td>Model</td><td colspan="2">Train / Infer FLOPs Speedup</td><td colspan="2">Params Train Time + Hardware</td><td>GLUE</td></tr><tr><td>ELMo</td><td>3.3e18 / 2.6e10</td><td>19x / 1.2x</td><td>96M</td><td colspan="2">14d on 3 GTX 1080 GPUs 71.2</td></tr><tr><td>GPT</td><td>4.0e19 / 3.0e10</td><td colspan="2">1.6x / 0.97x 117M</td><td>25d on 8 P6000 GPUs</td><td>78.8</td></tr><tr><td>BERT-Small</td><td>1.4e18 / 3.7e9</td><td>45x / 8x</td><td>14M</td><td>4d on 1 V100 GPU</td><td>75.1</td></tr><tr><td>BERT-Base</td><td>6.4e19 / 2.9e10</td><td>1x / 1x</td><td>110M</td><td>4d on 16 TPUv3s</td><td>82.2</td></tr><tr><td colspan="2">ELECTRA-Small 1.4e18 / 3.7e9</td><td>45x / 8x</td><td>14M</td><td>4d on 1 V100 GPU</td><td>79.9</td></tr><tr><td>50% trained</td><td>7.1e17 / 3.7e9</td><td>90x / 8x</td><td>14M</td><td>2d on 1 V100 GPU</td><td>79.0</td></tr><tr><td>25% trained</td><td>3.6e17 / 3.7e9</td><td>181x / 8x</td><td>14M</td><td>1d on 1 V100 GPU</td><td>77.7</td></tr><tr><td>12.5% trained</td><td>1.8e17 / 3.7e9</td><td>361x / 8x</td><td>14M</td><td>12h on 1 V100 GPU</td><td>76.0</td></tr><tr><td>6.25% trained</td><td>8.9e16 / 3.7e9</td><td>722x / 8x</td><td>14M</td><td>6h on 1 V100 GPU</td><td>74.1</td></tr><tr><td>ELECTRA-Base</td><td>6.4e19 / 2.9e10</td><td>1x / 1x</td><td>110M</td><td>4d on 16 TPUv3s</td><td>85.1</td></tr></table>

ModelTrain FLOPs Params CoLA SST MRPC STS QQP MNLI QNLI RTE Avg. Comparison of large models on the GLUE dev set. ELECTRA and RoBERTa are shown for different numbers of pre-training steps, indicated by the numbers after the dashes. ELECTRA performs comparably to XLNet and RoBERTa when using less than 1/4 of their pre-training compute and outperforms them when given a similar amount of pre-training compute. BERT dev results are fromClark et al. (2019).

Table content: <table><tr><td>BERT</td><td/><td colspan="4">1.9e20 (0.27x) 335M 60.6 93.2 88.0</td><td>90.0 91.3 86.6</td><td>92.3 70.4 84.0</td></tr><tr><td colspan="2">RoBERTa-100K</td><td colspan="4">6.4e20 (0.90x) 356M 66.1 95.6 91.4</td><td>92.2 92.0 89.3</td><td>94.0 82.7 87.9</td></tr><tr><td colspan="2">RoBERTa-500K</td><td colspan="4">3.2e21 (4.5x) 356M 68.0 96.4 90.9</td><td>92.1 92.2 90.2</td><td>94.7 86.6 88.9</td></tr><tr><td>XLNet</td><td/><td colspan="4">3.9e21 (5.4x) 360M 69.0 97.0 90.8</td><td>92.2 92.3 90.8</td><td>94.9 85.9 89.1</td></tr><tr><td>BERT (ours)</td><td/><td colspan="2">7.1e20 (1x)</td><td colspan="2">335M 67.0 95.9 89.1</td><td>91.2 91.5 89.6</td><td>93.5 79.5 87.2</td></tr><tr><td colspan="4">ELECTRA-400K 7.1e20 (1x)</td><td colspan="2">335M 69.3 96.0 90.6</td><td>92.1 92.4 90.5</td><td>94.5 86.8 89.0</td></tr><tr><td colspan="6">ELECTRA-1.75M 3.1e21 (4.4x) 335M 69.1 96.9 90.8</td><td>92.6 92.4 90.9</td><td>95.0 88.0 89.5</td></tr><tr><td>Model</td><td colspan="6">Train FLOPs CoLA SST MRPC STS QQP MNLI QNLI RTE WNLI Avg.* Score</td></tr><tr><td>BERT</td><td colspan="4">1.9e20 (0.06x) 60.5 94.9 85.4</td><td colspan="2">86.5 89.3 86.7</td><td>92.7 70.1 65.1</td><td>79.8 80.5</td></tr><tr><td colspan="5">RoBERTa 3.2e21 (1.02x) 67.8 96.7 89.8</td><td colspan="2">91.9 90.2 90.8</td><td>95.4 88.2 89.0</td><td>88.1 88.1</td></tr><tr><td colspan="5">ALBERT 3.1e22 (10x) 69.1 97.1 91.2</td><td colspan="2">92.0 90.5 91.3</td><td>-</td><td>89.2 91.8</td><td>89.0 -</td></tr><tr><td>XLNet</td><td colspan="4">3.9e21 (1.26x) 70.2 97.1 90.5</td><td colspan="2">92.6 90.4 90.9</td><td>-</td><td>88.5 92.5</td><td>89.1 -</td></tr><tr><td colspan="3">ELECTRA 3.1e21 (1x)</td><td colspan="2">71.7 97.1 90.7</td><td colspan="2">92.5 90.8 91.3</td><td>95.8 89.8 92.5</td><td>89.5 89.4</td></tr></table>

GLUE test-set results for large models. Models in this table incorporate additional tricks such as ensembling to improve scores (see Appendix B for details

Table content: <table/>

Results on the SQuAD for non-ensemble models.

Table content: <table><tr><td>Model</td><td colspan="2">Train FLOPs Params</td><td colspan="6">SQuAD 1.1 dev SQuAD 2.0 dev SQuAD 2.0 test EM F1 EM F1 EM F1</td></tr><tr><td>BERT-Base</td><td colspan="2">6.4e19 (0.09x) 110M</td><td>80.8</td><td>88.5</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>BERT</td><td colspan="2">1.9e20 (0.27x) 335M</td><td>84.1</td><td>90.9</td><td>79.0</td><td>81.8</td><td>80.0</td><td>83.0</td></tr><tr><td>SpanBERT</td><td>7.1e20 (1x)</td><td>335M</td><td>88.8</td><td>94.6</td><td>85.7</td><td>88.7</td><td>85.7</td><td>88.7</td></tr><tr><td>XLNet-Base</td><td colspan="2">6.6e19 (0.09x) 117M</td><td>81.3</td><td>-</td><td>78.5</td><td>-</td><td>-</td><td>-</td></tr><tr><td>XLNet</td><td>3.9e21 (5.4x)</td><td>360M</td><td>89.7</td><td>95.1</td><td>87.9</td><td>90.6</td><td>87.9</td><td>90.7</td></tr><tr><td>RoBERTa-100K</td><td colspan="2">6.4e20 (0.90x) 356M</td><td>-</td><td>94.0</td><td>-</td><td>87.7</td><td>-</td><td>-</td></tr><tr><td>RoBERTa-500K</td><td>3.2e21 (4.5x)</td><td>356M</td><td>88.9</td><td>94.6</td><td>86.5</td><td>89.4</td><td>86.8</td><td>89.8</td></tr><tr><td>ALBERT</td><td>3.1e22 (44x)</td><td>235M</td><td>89.3</td><td>94.8</td><td>87.4</td><td>90.2</td><td>88.1</td><td>90.9</td></tr><tr><td>BERT (ours)</td><td>7.1e20 (1x)</td><td>335M</td><td>88.0</td><td>93.7</td><td>84.7</td><td>87.5</td><td>-</td><td>-</td></tr><tr><td>ELECTRA-Base</td><td colspan="2">6.4e19 (0.09x) 110M</td><td>84.5</td><td>90.8</td><td>80.5</td><td>83.3</td><td>-</td><td>-</td></tr><tr><td>ELECTRA-400K</td><td>7.1e20 (1x)</td><td>335M</td><td>88.7</td><td>94.2</td><td>86.9</td><td>89.6</td><td>-</td><td>-</td></tr><tr><td>ELECTRA-1.75M</td><td>3.1e21 (4.4x)</td><td>335M</td><td>89.7</td><td>94.9</td><td>88.0</td><td>90.6</td><td>88.7</td><td>91.4</td></tr></table>

Compute-efficiency experiments (see text for details).

Table content: <table><tr><td colspan="2">Model</td><td/><td colspan="4">ELECTRA All-Tokens MLM Replace MLM ELECTRA 15% BERT</td></tr><tr><td colspan="3">GLUE score</td><td>85.0</td><td>84.3</td><td colspan="2">82.4</td><td>82.4</td><td>82.2</td></tr><tr><td/><td>86</td><td/><td/><td/><td/></tr><tr><td/><td>84</td><td/><td/><td/><td/></tr><tr><td/><td>82</td><td/><td/><td/><td/></tr><tr><td>GLUE Score</td><td>74 76 78 80</td><td/><td/><td/><td/></tr><tr><td/><td>70 72 68</td><td colspan="2">128 256 384 512 Hidden State Size ELECTRA 768 BERT</td><td colspan="2">128 256 384 512 Hidden State Size</td><td>768</td></tr></table>

Results for models on the GLUE test set. Only models with single-task finetuning (no ensembling, task-specific tricks, etc.) are shown.

Table content: <table/>

Peer review (iclr2020):
Summary: Authors offer an alternative for masked LM pretraining that*s more sample-efficient called replaced token detection. Their method basically replaces certain input tokens with alternatives which are sampled from a generator and train a discriminative model to determine whether its generated or real. The work shows empirical success getting better results than GPT with a fraction of the compute on GLUE and others. Positives: Idea is simple and makes sense intuitively, but not something one would think immediately would work better with a such a small fraction of the compute. I think the formulations of the experiments and ideas to develop this are adequate. Concerns & Questions: I*d like to see a little more investigation into Table 3. I don*t have intuition over why these results are the way that they are and the text nor the experimentation really gives me an indication. How well does this model work with very very little compute; lets say you have only a couple of gpu hours. Whats the degradation in performance? Overall I*d like to see more clarity in the overall analysis because I*m still unsure how to interpret your results on the why certain choices/experimental groups get the performance numbers they get. ------------------------------------------------------------------------------------------------------------------------ After the author response, I have changed my score to a 6. I think the paper merits acceptance.