Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: The authors propose replaced token detection, a novel self-supervised task, for learning text representations.

The principle advantage of the approach is that, in contrast with the standard masked language model (MLM) objective used by BERT and derivatives, there is a training signal for all tokens of the input (rather than a small fraction, when 10-20% of the input tokens are masked and then reconstructed under the MLM objective).

A smaller MLE-trained BERT-style generator is used to replace masked words with plausible alternatives, which the ELECTRA discriminator (the part that is retained and finetuned on downstream tasks) must detect (unmodified word slots are also in the objective, and must be detected as such).

In general the paper reads well, and the authors present ablations to reveal the source of gains. ELECTRA matches the performance of RobBERTa on the popular GLUE NLP task, with just 1/4 of the training compute.

Strengths:
-Simple but novel self-supervised task for learning text representations, strong results, adequate ablation.

Limitations:
-The authors limit their investigation of downstream performance to the GLUE set of tasks, which are classification tasks. This is a significant limitation of the current version of the paper, as it may be that replaced token detection is more suitable for these tasks, but inferior to MLM (a higher precision self-supervised task) for more involved tasks like question answering. The latter is arguably of much higher importance to the NLP research community at this point, and some consider the GLUE task to be essentially solved for all practical purposes, given inherent noise levels.
-In contrast with BERT, there is no mention of any plan to release ELECTRA (big or small versions), which is a disappointment, lowers the significance of the work

Overall:
An okay paper. Results on SQUAD or another more elaborate NLP task and/or the release of the ELECTRA models would make the paper much stronger.
review_title: Official Blind Review #1

final_score: rating: 8: Accept
review_content: The paper proposed a novel sample-efficient pretraining task. One inefficiency of BERT is that only 15% tokens are used for training in each example. The paper introduced a generator+discriminator framework to optimize the utility of training examples. The generator task is the MLM which predicts the masked word. The author adds a discriminator to further learn from the example by classifying each word to be either generated or original. In this way, more words can be used. This method looks as only adding the discrimination task after BERT pretraining task. But, the authors later show that the best GLUE scores can be obtained only when both generator and discriminator are co-trained. Moreover, the adversarial ELECTRA perform worse. All these observations are interesting. It will be helpful if the authors provide more empirical analysis why the adversarial ELECTRA perform worse or failed. Is it because the GAN is hard to train or the adversarial task doesn't fit the pretraining? 

Overall, I think this is a good paper. The studied problem is important, the idea is new and the experimental results are positive. Specifically, it shows that ELECTRA can outperforms BERT and match RoBERTa with less training time. But, it will be a big plus if the authors can show ELECTRA can outperform RoBERTa with the same amount of training time. Analysis are also provided to give audience insights in this method.
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: Summary: Authors offer an alternative for masked LM pretraining that's more sample-efficient called replaced token detection. Their method basically replaces certain input tokens with alternatives which are sampled from a generator and train a discriminative model to determine whether its generated or real. The work shows empirical success getting better results than GPT with a fraction of the compute on GLUE and others.

Positives: Idea is simple and makes sense intuitively, but not something one would think immediately would work better with a such a small fraction of the compute. I think the formulations of the experiments and ideas to develop this are adequate.

Concerns & Questions: I'd like to see a little more investigation into Table 3. I don't have intuition over why these results are the way that they are and the text nor the experimentation really gives me an indication. How well does this model work with very very little compute; lets say you have only a couple of gpu hours. Whats the degradation in performance?

Overall I'd like to see more clarity in the overall analysis because I'm still unsure how to interpret your results on the why certain choices/experimental groups get the performance numbers they get.

------------------------------------------------------------------------------------------------------------------------

After the author response, I have changed my score to a 6. I think the paper merits acceptance.
review_title: Official Blind Review #3

Meta-review: This paper investigates the tasks used to pretrain language models. The paper proposes not using a generative tasks ('filling in' masked tokens), but instead a discriminative tasked (recognising corrupted tokens). The authors empirically show that the proposed method leads to improved performance, especially in the "limited compute" regime. 

Initially, the reviewers had quite split opinions on the paper, but after the rebuttal and discussion phases all reviewers agreed on an "accept" recommendation. I am happy to agree with this recommendation based on the following observations:
- The authors provide strong empirical results including relevant ablations. Reviews initially suggested a limitation to classification tasks and a lack of empirical analysis, but those issues have been addressed in the updated version. 
- The problem of pre-training language model is relevant for the ML and NLP communities, and it should be especially relevant for ICLR. The resulting method significantly outperforms existing methods, especially in the low compute regime. 
- The idea is quite simple, but at the same time it seems to be a quite novel idea. 
Decision: Accept (Poster)