ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators

Abstract: Masked language modeling (MLM) pre-training methods such as BERT corrupt the input by replacing some tokens with [MASK] and then train a model to reconstruct the original tokens. While they produce good results when transferred to downstream NLP tasks, they generally require large amounts of compute to be effective. As an alternative, we propose a more sample-efficient pre-training task called replaced token detection. Instead of masking the input, our approach corrupts it by replacing some tokens with plausible alternatives sampled from a small generator network. Then, instead of training a model that predicts the original identities of the corrupted tokens, we train a discriminative model that predicts whether each token in the corrupted input was replaced by a generator sample or not. Thorough experiments demonstrate this new pre-training task is more efficient than MLM because the task is defined over all input tokens rather than just the small subset that was masked out. As a result, the contextual representations learned by our approach substantially outperform the ones learned by BERT given the same model size, data, and compute. The gains are particularly strong for small models; for example, we train a model on one GPU for 4 days that outperforms GPT (trained using 30x more compute) on the GLUE natural language understanding benchmark. Our approach also works well at scale, where it performs comparably to RoBERTa and XLNet while using less than 1/4 of their compute and outperforms them when using the same amount of compute.
      

Reviews and author responses:

The authors propose replaced token detection, a novel self-supervised task, for learning text representations.

The principle advantage of the approach is that, in contrast with the standard masked language model (MLM) objective used by BERT and derivatives, there is a training signal for all tokens of the input (rather than a small fraction, when 10-20% of the input tokens are masked and then reconstructed under the MLM objective).

A smaller MLE-trained BERT-style generator is used to replace masked words with plausible alternatives, which the ELECTRA discriminator (the part that is retained and finetuned on downstream tasks) must detect (unmodified word slots are also in the objective, and must be detected as such).

In general the paper reads well, and the authors present ablations to reveal the source of gains. ELECTRA matches the performance of RobBERTa on the popular GLUE NLP task, with just 1/4 of the training compute.

Strengths:
-Simple but novel self-supervised task for learning text representations, strong results, adequate ablation.

Limitations:
-The authors limit their investigation of downstream performance to the GLUE set of tasks, which are classification tasks. This is a significant limitation of the current version of the paper, as it may be that replaced token detection is more suitable for these tasks, but inferior to MLM (a higher precision self-supervised task) for more involved tasks like question answering. The latter is arguably of much higher importance to the NLP research community at this point, and some consider the GLUE task to be essentially solved for all practical purposes, given inherent noise levels.
-In contrast with BERT, there is no mention of any plan to release ELECTRA (big or small versions), which is a disappointment, lowers the significance of the work

Overall:
An okay paper. Results on SQUAD or another more elaborate NLP task and/or the release of the ELECTRA models would make the paper much stronger.

Summary: Authors offer an alternative for masked LM pretraining that's more sample-efficient called replaced token detection. Their method basically replaces certain input tokens with alternatives which are sampled from a generator and train a discriminative model to determine whether its generated or real. The work shows empirical success getting better results than GPT with a fraction of the compute on GLUE and others.

Positives: Idea is simple and makes sense intuitively, but not something one would think immediately would work better with a such a small fraction of the compute. I think the formulations of the experiments and ideas to develop this are adequate.

Concerns & Questions: I'd like to see a little more investigation into Table 3. I don't have intuition over why these results are the way that they are and the text nor the experimentation really gives me an indication. How well does this model work with very very little compute; lets say you have only a couple of gpu hours. Whats the degradation in performance?

Overall I'd like to see more clarity in the overall analysis because I'm still unsure how to interpret your results on the why certain choices/experimental groups get the performance numbers they get.

------------------------------------------------------------------------------------------------------------------------

After the author response, I have changed my score to a 6. I think the paper merits acceptance.

Thank you for the clarifications. The response provides a lot more information and clarifies a few things. After reading through the recommendations from other reviewers as well as the GLUE scores for various snapshots of Electra-small, I'm inclined to bump up my score to a 6: weak accept if the updated descriptions can be integrated into the paper and tables with these training curves can be provided for your models!

We want to thank the reviewers again for their suggestions! We have updated the paper with the following changes: 
  - Addressing Reviewer 1’s concern, we added results for SQuAD (both 1.1 and 2.0). ELECTRA-Large matches RoBERTa on SQuAD (getting 0.2 exact-match points worse on 1.1 but 0.4 points better on 2.0) despite using less than ¼ of the pre-training compute. ELECTRA-Base scores much higher than BERT-Base/XLNet-Base and even outperforms BERT-Large.
  - Addressing Reviewer 2’s question, we added some discussion and empirical analysis of the problems with adversarial ELECTRA.
  - Addressing Reviewer 3’s question, we added results for ELECTRA-Small with even fewer GPU-hours in Table 1. ELECTRA performs well even when using as little as 6 GPU-hours of pre-training. 
  - We also updated the appendix with descriptions of the GLUE dataset and test-set results of the Base/Small ELECTRA models. A new finding is that ELECTRA-Small outperforms TinyBERT, a model of comparable size that uses a complicated distillation procedure to learn from BERT-Base during both pre-training and fine-tuning, as well as data augmentation (whereas ELECTRA-Small uses no distillation or data augmentation).

Lastly, as suggested by Reviewer 2, we are in the process of training ELECTRA for longer (with compute comparable to RoBERTa), although it is still early along in training. 

Thank you for the comments! We address some of the concerns and questions below:

>“I'd like to see a little more investigation into Table 3. I don't have intuition over why these results are the way that they are and the text nor the experimentation really gives me an indication.”
>“Overall I'd like to see more clarity in the overall analysis because I'm still unsure how to interpret your results...”

The ablations in Table 3 are a series of “stepping stones” between BERT and ELECTRA designed to show where ELECTRA’s improvements come from. To reiterate the discussion in the paper:
  - “Replace MLM” slightly outperforming BERT suggests that a small amount of the gains can be attributed to solving BERT’s pre-train/fine-tune mismatch due to [MASK] tokens, as “Replace MLM” is essentially BERT with this mismatch fixed.
  - “ELECTRA 15%” matching “Replace MLM” suggests that ELECTRA is benefitting a lot from learning from all input tokens. ELECTRA 15% is essentially ELECTRA with this advantage over BERT removed and indeed the gains over BERT mostly go away. 
  - “All-Tokens MLM” outperforming “Replace MLM” further demonstrates the benefit of learning from all input tokens, as we substantially improve BERT’s masked language model objective when incorporating this idea into BERT.
  - Lastly, ELECTRA outperforming “All-Tokens MLM” shows the additional value of our discriminative second stage classifier rather than replacing it with a BERT-style generative model. 

Please let us know if there are other results you find unclear or if you have suggestions for further experiments/discussion that would help clarify the results. 


>“How well does this model work with very very little compute; lets say you have only a couple of gpu hours. Whats the degradation in performance?”

Here are GLUE scores for ELECTRA-Small for various training times, which we will add to the paper. 
4 days: 79.9 (slight improvement over the number in the submission due to some additional hyperparameter tuning)
2 days: 79.0
1 day: 77.7
12 hours: 76.0
6 hours: 74.1
We note that even the full 4 days is already a tiny fraction (~1/50th) of the compute used to train BERT, but these results show that an effective model can be built with even more limited resources. 



Thank you for the comments! We address some of the concerns and questions below:

>“It will be helpful if the authors provide more empirical analysis why the adversarial ELECTRA perform worse or failed.”

Yes, that is a good question! We did not have too much discussion on this in the submission because it is a negative result and we were limited for space. We found two problems with the adversarially trained generator. The main one is that the adversarial generator is simply worse at masked language modeling. For example, a size-256 adversarial generator after 500k training steps achieves 58% accuracy at masked language modeling compared to 65% accuracy for an MLE-trained one. We believe the worse accuracy is mainly due to the poor sample efficiency of reinforcement learning when working in the large action space of generating text. As evidence for this, the adversarial generator's MLM accuracy was still increasing towards the end of training while the MLE generator‘s accuracy vs train step curve had mostly flattened out. The second problem is that the adversarially trained generator produces a “peaky” low-entropy output distribution where most of the probability mass is on a single token, which means there is not much diversity in the generator samples. Both of these problems have been observed in GANs for text in prior work (see “Language GANs Falling Short” from Caccia et al., 2018 and “Evaluating Text GANs as Language Models” from Tevet et al., 2019). We will add this additional discussion to the paper.


>“But, it will be a big plus if the authors can show ELECTRA can outperform RoBERTa with the same amount of training time.”

We are working on training ELECTRA for longer (it just takes lots of compute!). We want to emphasize that our focus is on compute efficiency. However, given the trend of the accuracy vs compute curve in Figure 1, we are confident that ELECTRA will continue to improve and therefore outperform RoBERTa when given the same training time. 

Thank you for the comments! We address some of the concerns and questions below:

>“The authors limit their investigation of downstream performance to the GLUE set of tasks, which are classification tasks. This is a significant limitation of the current version of the paper…”

We definitely agree with the reviewer’s point that evaluating on diverse tasks is useful! We initially focused on GLUE because it contains a variety of tasks and has been the main benchmark for evaluating pre-trained representations from GPT onwards. However, we have since run experiments on SQuAD (both 1.1 and 2.0) and will add them to the paper. Results are consistent with the GLUE ones (e.g., ELECTRA-Base outperforms BERT-Large and ELECTRA-Large matches RoBERTa). ELECTRA appears to be slightly better at SQuAD 2.0 (where it outperforms RoBERTa by 0.4 exact-match points) than 1.1 (where it slightly underperforms RoBERTa by 0.2 exact-match points). We think the best way of evaluating pre-trained encoders is still an open question - more challenging tasks might better distinguish models, but require more sophisticated classifiers on top of the transformer that can complicate the analysis. 


>“In contrast with BERT, there is no mention of any plan to release ELECTRA (big or small versions), which is a disappointment, lowers the significance of the work.”

We absolutely will release the code and pre-trained weights (for all model sizes)! We apologize for not making that clear in the submission. 

You're welcome! 

1. The generator is a small network only used to help train the discriminator. In fact, it could be as simple as a unigram LM, in which case there would be no way to fine-tune it for downstream tasks. At any rate, it’s performance for downstream use is usually much below that of the discriminator.

2. Could you please rephrase the question? We don’t quite understand what you are asking.

The paper proposed a novel sample-efficient pretraining task. One inefficiency of BERT is that only 15% tokens are used for training in each example. The paper introduced a generator+discriminator framework to optimize the utility of training examples. The generator task is the MLM which predicts the masked word. The author adds a discriminator to further learn from the example by classifying each word to be either generated or original. In this way, more words can be used. This method looks as only adding the discrimination task after BERT pretraining task. But, the authors later show that the best GLUE scores can be obtained only when both generator and discriminator are co-trained. Moreover, the adversarial ELECTRA perform worse. All these observations are interesting. It will be helpful if the authors provide more empirical analysis why the adversarial ELECTRA perform worse or failed. Is it because the GAN is hard to train or the adversarial task doesn't fit the pretraining? 

Overall, I think this is a good paper. The studied problem is important, the idea is new and the experimental results are positive. Specifically, it shows that ELECTRA can outperforms BERT and match RoBERTa with less training time. But, it will be a big plus if the authors can show ELECTRA can outperform RoBERTa with the same amount of training time. Analysis are also provided to give audience insights in this method.

Thanks the authors for the small model. Some questions:

1. 'we throw out the generator and only fine-tune the discriminator on downstream tasks', what is the motivation?

2. Can be changed to this: originalSentence --> [HardMaskModel] --> MaskedSentence --> ELECTRA, moreover there can be more discriminator tasks?

Hi! We considered a token "hard to predict" if it had low probability under BERT's output distribution when it was masked out. The model learning which tokens were hard to predict was a small transformer network taking the unmasked text as input. For each token, it predicted (using a sigmoid output layer) how much probability BERT would assign that token if it was masked. We trained it by running BERT as normal on the input and then minimizing the sigmoid cross-entropy loss between the model’s output and the probability BERT gave to the correct token for the 15% of tokens that were masked. It’s hard to come up with an easily interpretable evaluation metric for this model, but the per-token KL divergences were ~0.25 for our model versus ~0.5 for the baseline of saying all tokens are equally hard to predict.

Hello, I want to thank the authors for a very interesting and useful paper.

In the paper, in the negative results section, the first bullet says :

"We initially attempted to make BERT more efficient by strategically masking-out tokens
(e.g., masking our rarer tokens more frequently, or training a model to guess which tokens
BERT would struggle to predict if they were masked out). This resulted in fairly minor
speedups over regular BERT"

I was wondering if you could mention which model they used to "training a model to guess which tokens
BERT would struggle to predict if they were masked out", and some measure of how well the model it could do such a task.

Thanks.

Meta-review: This paper investigates the tasks used to pretrain language models. The paper proposes not using a generative tasks ('filling in' masked tokens), but instead a discriminative tasked (recognising corrupted tokens). The authors empirically show that the proposed method leads to improved performance, especially in the "limited compute" regime. 

Initially, the reviewers had quite split opinions on the paper, but after the rebuttal and discussion phases all reviewers agreed on an "accept" recommendation. I am happy to agree with this recommendation based on the following observations:
- The authors provide strong empirical results including relevant ablations. Reviews initially suggested a limitation to classification tasks and a lack of empirical analysis, but those issues have been addressed in the updated version. 
- The problem of pre-training language model is relevant for the ML and NLP communities, and it should be especially relevant for ICLR. The resulting method significantly outperforms existing methods, especially in the low compute regime. 
- The idea is quite simple, but at the same time it seems to be a quite novel idea. 

Decision: accept-poster