Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper presents a replication study of BERT pretraining and carefully measures the impact of many key hyperparameters and training data size. It shows that BERT was significantly undertrained and propose an improved training recipe called RoBERTa. The key ideas are: (i) training longer with bigger batches over more data, (ii) removing NSP, (iii) training over long sequences, and (iv) dynamically changing the masking pattern. The proposed RoBERTa achieves/matches state-of-the-art performance on many standard NLU downstream tasks. 

The in-depth experimental analysis of the BERT pretraining process in this paper answers many open questions (e.g., the usefulness of NSP objective) and also provide some guidance in how to effectively tweak the performance of pretrained model (e.g., large batch size). It also further demonstrates that the BERT model, once fully tuned, could achieve SOTA/competitive performance compared to the recent new models (e.g., XLNet). The main weakness of the paper is that it is mainly based on further tuning the existing BERT model and lacks novel contribution in model architecture. However, the BERT analysis results provided in this paper should also be valuable to the community. 

Questions & Comments:
•	It is stated that the performance is sensitive to epsilon in AdamW. This reminds us of the sensitivity of BERT pretraining to the optimizers. Since one of the main contributions of this paper is the analysis of the BERT pretraining process, more experimental analysis on the optimizer should also be included.
•	It is stated that (page 7) the submission to GLUE leaderboard uses only single-task finetuning. Is there any special reason for restraining it to single-task finetuning if earlier results demonstrates multi-task finetuning is better? Of course, it is valuable to see the great performance achieved by single-task finetuning for RoBERTa. But there should be no reason that it is restricted to be so. An additional experimental results with multi-task finetuning should also be added.

review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This paper presents a detailed replication study of the BERT pre-training model considering alternative design choices such as dynamic masking, removal of next sentence prediction loss, longer training time, larger batch sizes, additional training data, training on longer single document or cross-document sequences etc. to demonstrate their efficacy on several benchmark datasets and tasks by achieving the new state-of-the-art results. Overall, the paper is very well-written and the experimental setups are reasonable and thoroughly presented that would benefit the community for future research and exploration. However, I am not sure if the paper presents a case of adequate novelty in terms of ideas as many of them are rather obvious and the current state-of-the-art models could also improve considerably using similar experimental setups, which authors also acknowledged in footnote 2.

Other comments:

- Section 3.1: please clarify how exactly setting Beta2=0.98 would improve stability when training with large batch sizes.

- It's not clear what exactly was the motivation for proposing full-sentences and doc-sentences input formats. Please explain.

- Although the presented metrics show that the removal of NSP loss helps, however, no explanation was provided based on qualitative evaluation as to why this is the case. Some task-specific examples would have been nice to discuss the effects of NSP loss.

- Section 5: Please provide details on why you think the longest-trained model does not appear to overfit. 
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: This paper is a replication study of BERT for training large language models. Its main modification is simple: training longer with more data. Significantly improvements have been reported, and the work achieves on-par or higher accuracy over a large set of downstream tasks compared to XLNet, which is a state-of-the-art autoregressive language model. 

Pros:
+ The paper incorporates robust optimization into BERT training with more data, and shows that together it significantly improves BERT's performance on downstream tasks.
+ The experimental results show that RoBERTa can significantly advance the baseline BERT model and achieve on-par or new state-of-the-art accuracy on a large range of downstream tasks.

Cons:
- While the replication study is well appreciated, the novelty contribution of the paper is marginally incremental as the model structure is largely unchanged from BERT. The other techniques applied also are somewhat trivial.
- Very little can be deduced from the experiments, as performance is often improved by training over more data. 

Overall, I believe this paper comes at the right time and is addressing an interesting problem. The paper is well-organized and well-written. The contribution of the paper comes mostly from carefully taking into account several additional design choices and show that they could help train BERT with more data and can achieve SoTA performance on downstream tasks. Those modifications can be summarized as: (1) large-batch training with batch size 8k; (2) no mixed sequence length training only used 512 for the entire run; (3) no next sentence prediction; (4) dynamic masking instead of static; (5) larger byte-level BPE vocab (which increases BERT-large size by 20M parameters). Although they are interesting, a major concern is that it is difficult to find one thing that would have catapulted it over others to ensure publication.

Question:
Do you plan to release the datasets used for training in this work?
review_title: Official Blind Review #3

Meta-review: This paper conducts an extensive study of training BERT and shows that its performance can be improved significantly by choosing a better training setup (e.g., hyperparameters, objective functions). I think this paper clearly offers a better understanding of the importance of tuning a language model to get the best performance on downstream tasks. However, most of the findings are obvious (careful tuning helps, more data helps). I think the novelty and technical contributions are rather limited for a conference such as ICLR. These concerns are also shared by all the reviewers. The review scores are borderline, so I recommend to reject the paper.
Decision: Reject