Title: ROBERTA: A ROBUSTLY OPTIMIZED BERT PRE-TRAINING APPROACH

Abstract: Language model pretraining has led to significant performance gains but careful comparison between different approaches is challenging. Training is computationally expensive, often done on private datasets of different sizes, and, as we show, hyperparameter choices have significant impact on the final results. We present a replication study of BERT pretraining (Devlin et al., 2019) that carefully measures the impact of many key hyperparameters and training data size. We find that BERT was significantly undertrained, and can match or exceed the performance of every model published after it. Our best model achieves state-of-the-art results on GLUE, RACE, SQuAD, SuperGLUE and XNLI. These results highlight the importance of previously overlooked design choices, and raise questions about the source of recently reported improvements. We release our models and code. 1

1. INTRODUCTION:

Self-training methods such as ELMo (Peters et al., 2018) , GPT (Radford et al., 2018) , BERT (Devlin et al., 2019) , XLM (Lample & Conneau, 2019) , and XLNet (Yang et al., 2019) have brought significant performance gains, but it can be challenging to determine which aspects of the methods contribute the most. Training is computationally expensive, limiting the amount of tuning that can be done, and modeling advances are often conflated with changes in data size or composition.

We present a replication study of BERT pretraining (Devlin et al., 2019) , which includes a careful evaluation of the effects of hyperparameter tuning and training set size. We find that BERT was significantly undertrained and propose an improved training recipe, which we call RoBERTa, that can match or exceed the performance of all of the post-BERT methods. Our modifications are simple, they include: (1) training the model longer, with bigger batches, over more data; (2) removing the next sentence prediction objective; (3) training on longer sequences; and (4) dynamically changing the masking pattern applied to the training data. We also collect a large new dataset (CC-NEWS) of comparable size to other privately used datasets, to better control for training set size effects.

When controlling for training data, our improved training procedure improves upon the published BERT results on the GLUE (Wang et al., 2019b) and SQuAD (Rajpurkar et al., 2016) benchmarks. When trained for longer over additional data, our model achieves a score of 88.5 on the public GLUE leaderboard, matching the 88.4 reported by Yang et al. (2019) . Our model establishes a new stateof-the-art on 4/9 of the GLUE tasks, as well as RACE (Lai et al., 2017) , SuperGLUE (Wang et al., 2019a) , and XNLI (Conneau et al., 2018) , and matches the state-of-the-art on SQuAD. Overall, we re-establish that BERT's masked language model training objective is competitive with recently proposed alternatives such as perturbed autoregressive language modeling (Yang et al., 2019). 2 In summary, the contributions of this paper are: (1) We present a set of important BERT design choices and training strategies and introduce alternatives that lead to better downstream task performance; (2) We use a novel dataset, CC-NEWS, and confirm that using more data for pretraining further improves performance on downstream tasks; (3) Our training improvements show that masked language model pretraining, under the right design choices, is competitive with all other recently published methods. We release our model, pretraining and fine-tuning code.

2. BACKGROUND:

Setup: BERT (Devlin et al., 2019) takes as input a concatenation of two segments (sequences of tokens), x 1 , . . . , x N and y 1 , . . . , y M . Segments usually consist of more than one natural sentence. The two segments are presented as a single input sequence to BERT with special tokens delimiting them: [CLS ], x 1 , . . . , x N , [SEP ], y 1 , . . . , y M , [EOS ]. M and N are constrained such that M +N < T , where T is a parameter that controls the maximum sequence length during training.

Architecture: BERT uses the now ubiquitous transformer architecture (Vaswani et al., 2017) , which we will not review in detail. We use a transformer architecture with L layers. Each block has A self-attention heads and hidden dimension H.

Training Objectives: BERT uses two pretraining objectives: masked language modeling and next sentence prediction. For the Masked Language Model (MLM) objective, BERT is trained via a crossentropy loss to predict 15% of the input tokens, selected at random. To prevent the model from cheating, 80% of these selected tokens are replaced by a special [MASK ] symbol in the input, 10% are replaced by a random token from the vocabulary, and 10% are left unchanged. Optimization: BERT is optimized with AdamW (Kingma & Ba, 2015) using the following parameters: β 1 = 0.9, β 2 = 0.999, = 1e-6 and decoupled weight decay of 0.01 (Loshchilov & Hutter, 2019) . The learning rate is warmed up over the first 10,000 steps to a peak value of 1e-4, and then linearly decayed. BERT trains with a dropout of 0.1 on all layers and attention weights, and a GELU activation function (Hendrycks & Gimpel, 2016) . Models are pretrained for S = 1,000,000 updates, with mini-batches containing B = 256 sequences of maximum length T = 512 tokens.

Next Sentence:

Data: BERT is trained on a combination of BOOKCORPUS (Zhu et al., 2015) plus English WIKIPEDIA, which totals 16GB of uncompressed text.3 

3. EXPERIMENTAL SETUP:



3.1. IMPLEMENTATION:

We reimplement BERT in FAIRSEQ (Ott et al., 2019) . We primarily follow the original BERT optimization hyperparameters, given in Section 2, except for the peak learning rate and number of warmup steps, which are tuned separately for each setting. We found training to be very sensitive to the Adam epsilon term, and in some cases we obtained better performance or improved stability after tuning it. We also set β 2 = 0.98 to improve stability when training with large batch sizes.

We pretrain with sequences of at most T = 512 tokens. Unlike Devlin et al. (2019) , we do not randomly inject short sequences, and we do not train with a reduced sequence length for the first 90% of updates. We train only with full-length sequences.

We train with mixed precision floating point arithmetic on DGX-1 machines, each with 8 × 32GB Nvidia V100 GPUs interconnected by Infiniband (Micikevicius et al., 2018) .

3.2. DATA:

BERT-style pretraining crucially relies on large quantities of text. Baevski et al. (2019) demonstrate that increasing data size can result in improved end-task performance. Several efforts have trained on datasets larger and more diverse than the original BERT (Radford et al., 2019; Yang et al., 2019; Zellers et al., 2019) . Unfortunately, not all of the additional datasets can be publicly released. For our study, we focus on gathering as much data as possible for experimentation, allowing us to match the overall quality and quantity of data as appropriate for each comparison.

We consider five English-language corpora of varying sizes and domains, totaling over 160GB of uncompressed text: (1&2) BOOKCORPUS (Zhu et al., 2015) plus English WIKIPEDIA, which is the original data used to train BERT (16GB); (3) CC-NEWS, which we collect from the English portion of the CommonCrawl News dataset (Nagel, 2016) , containing 63 million English news articles crawled between September 2016 and February 2019 (76GB after filtering);4 (4) OPEN-WEBTEXT (Gokaslan & Cohen, 2019) , an open-source recreation of the WebText corpus described in Radford et al. (2019) , containing web content extracted from URLs shared on Reddit with at least three upvotes (38GB);5 (5) STORIES, a dataset introduced in Trinh & Le (2018) containing a subset of CommonCrawl data filtered to match the story-like style of Winograd schemas (31GB).

3.3. EVALUATION:

Following previous work, we evaluate our pretrained models by finetuning on downstream tasks:

• GLUE: The General Language Understanding Evaluation (GLUE) benchmark (Wang et al., 2019b) is a collection of 9 datasets for evaluating natural language understanding systems. Tasks are framed as either single-sentence classification or sentence-pair classification tasks. The GLUE organizers provide training and development data splits as well as a submission server and leaderboard that allows participants to evaluate and compare their systems on private held-out test data. • SQuAD: The Stanford Question Answering Dataset (SQuAD) provides a paragraph of context and a question. The task is to answer the question with a span extracted from the context. We evaluate on SQuAD V1.1 and V2.0 (Rajpurkar et al., 2016; 2018) . In V1.1 the context always contains an answer, while in V2.0 some questions are not answered in the provided context. • RACE: ReAding Comprehension from Examinations (RACE) (Lai et al., 2017 ) is a large-scale reading comprehension dataset collected from English examinations in China. The task is to choose among four possible answers to a given question, using a given passage of text as context. • Additional Benchmarks: In the Appendix we present additional results for SuperGLUE (Wang et al., 2019a) and XNLI (Conneau et al., 2018) .

4. TRAINING PROCEDURE ANALYSIS:

This section explores and quantifies which choices are important for successfully pretraining BERT models. We keep the model architecture fixed.6 Specifically, we begin by training BERT models with the same configuration as BERT BASE (L = 12, H = 768, A = 12, 110M params).

4.1. STATIC VS. DYNAMIC MASKING:

As discussed in Section 2, BERT relies on predicting randomly masked tokens. The original BERT implementation performed masking once during data preprocessing, resulting in a single static mask.

To avoid repeating the same masks at every epoch, training data was duplicated 10 times prior to preprocessing, so that each training sequence was seen with the same mask only four times over the course of 40 training epochs. We instead train with dynamic masking, where we generate the masking pattern on-the-fly each time we input a sequence to the model. This becomes crucial when pretraining for more steps or with larger datasets, and additionally performs marginally better than static masking on some downstream tasks (see Appendix A).

4.2. MODEL INPUT FORMAT AND NEXT SENTENCE PREDICTION:

In the original BERT pretraining procedure, the model observes two concatenated document segments and is trained via an auxiliary Next Sentence Prediction (NSP) loss to predict whether these segments were sampled contiguously from the same document or from distinct documents. The NSP objective was designed to improve performance on downstream tasks, such as Natural Language Inference (Bowman et al., 2015) , which require predicting relationships between pairs of sentences. Devlin et al. (2019) observe that removing NSP hurts performance, with significant performance degradation on QNLI, MNLI, and SQuAD 1.1. However, recent work has questioned the necessity of the NSP loss (Lample & Conneau, 2019; Yang et al., 2019; Joshi et al., 2019) .

To better understand this discrepancy, we compare several alternative training formats:

• SEGMENT-PAIR+NSP: This follows the original input format used in BERT (Devlin et al., 2019) , with the NSP loss. Each input has a pair of segments, which can each contain multiple natural sentences, but the total combined length must be less than 512 tokens. • SENTENCE-PAIR+NSP: Each input contains a pair of natural sentences, either sampled from a contiguous portion of one document or from separate documents. Since these inputs are significantly shorter than 512 tokens, we increase the batch size so that the total number of tokens remains similar to SEGMENT-PAIR+NSP. We retain the NSP loss. • FULL-SENTENCES: Each input is packed with full sentences sampled contiguously from one or more documents, such that the total length is at most 512 tokens. Inputs may cross document boundaries. When we reach the end of one document, we begin sampling sentences from the next document and add an extra separator token between documents. We remove the NSP loss. • DOC-SENTENCES: Inputs are constructed similarly to FULL-SENTENCES, except that they may not cross document boundaries. Inputs sampled near the end of a document may be shorter than 512 tokens, so we dynamically increase the batch size in these cases to achieve a similar number of total tokens as FULL-SENTENCES. We remove the NSP loss.

Results Table 1 shows results for the four different settings. We first compare the original SEGMENT-PAIR input format from Devlin et al. (2019) to the SENTENCE-PAIR format; both formats retain the NSP loss, but the latter uses single sentences. We find that using individual sentences hurts performance on downstream tasks, which we hypothesize is because the model is not able to learn long-range dependencies.

We next compare training without the NSP loss and training with blocks of text from a single document (DOC-SENTENCES). We find that this setting outperforms the originally published BERT BASE results and that removing the NSP loss matches or slightly improves downstream task performance, in contrast to Devlin et al. (2019) . It is possible that the original BERT implementation may only have removed the loss term while still retaining the SEGMENT-PAIR input format.

Finally we find that restricting sequences to come from a single document (DOC-SENTENCES) performs slightly better than packing sequences from multiple documents (FULL-SENTENCES 

4.3. TRAINING WITH LARGE BATCHES:

Past work in neural machine translation has shown that training with large mini-batches can improve optimization speed and end-task performance when the learning rate is tuned appropriately (Ott et al., 2018) . Large batches are also easily parallelized via data parallel training.7 

Table 2 shows the masked LM perplexity and end-task accuracy for BERT BASE as we increase the batch size, while tuning the learning rate. Devlin et al. (2019) originally trained BERT BASE for 1M steps with a batch size of 256 sequences; however a batch size of 2K sequences performs better, even controlling for the number of epochs, suggesting that the original BERT batch size was too small. We also observe that training with extremely large batches (8K) becomes more efficient as we train for more epochs. 8 In the remainder of our experiments we train with batches of 8K sequences.

4.4. TEXT ENCODING:

Byte-Pair Encoding (BPE) (Sennrich et al., 2016 ) is a hybrid between character-and word-level modeling based on subwords units. BPE vocabulary sizes typically range from 10K-100K subword units; however, unicode characters can account for a sizeable portion of this vocabulary when modeling large and diverse corpora, such as the ones considered in this work.

The original BERT implementation (Devlin et al., 2019) Early experiments revealed only minor differences between these encodings, with the byte-level BPE achieving slightly worse end-task performance on some tasks. Nevertheless, we believe the advantages of a universal encoding scheme outweighs the minor degredation in performance and use this encoding in the remainder of our experiments.

5. ROBERTA:

In the previous section we propose modifications to the BERT pretraining procedure that improve end-task performance. We now aggregate these improvements and evaluate their combined impact. We call this configuration RoBERTa for Robustly optimized BERT approach. Specifically, RoBERTa is trained with dynamic masking (Section 4.1), FULL-SENTENCES without NSP loss (Section 4.2), large mini-batches (Section 4.3) and a larger byte-level BPE (Section 4.4). For example, XLNet (Yang et al., 2019) was pretrained using 10 times more data than BERT, with a batch size eight times larger for half as many optimization steps, thus seeing four times as many sequences in pretraining compared to Devlin et al. (2019) .

To help disentangle the importance of these factors from other modeling choices (e.g., the pretraining objective), we begin by training RoBERTa following the BERT LARGE architecture (L = 24, H = 1024, A = 16, 355M parameters). We pretrain for 100K steps over a comparable BOOKCORPUS plus WIKIPEDIA dataset as was used in Devlin et al. (2019) . We pretrain our model using 1024 V100 GPUs, which takes approximately one day per 100K steps.

Results:

We present our results in Table 3 . When controlling for training data, we observe that RoBERTa provides a large improvement over the originally reported BERT LARGE results, reaffirming the importance of the design choices we explored in Section 4.

Next, we combine this data with the three additional datasets described in Section 3.2. We train RoBERTa over the combined data with the same number of training steps as before (100K). In total, we pretrain over 160GB of text. We observe further improvements in performance across all downstream tasks, validating the importance of data size and diversity in pretraining.9 

Finally, we pretrain RoBERTa for significantly longer, increasing the number of pretraining steps from 100K to 300K, and then further to 500K. We again observe significant gains in downstream task performance, and the 300K and 500K step models outperform XLNet LARGE across most tasks. We note that even our longest-trained model does not appear to overfit our data and would likely benefit from additional training.

5.1. GLUE RESULTS:

For GLUE, we consider two finetuning settings. In the first setting (single-task, dev), we finetune RoBERTa separately for each of the GLUE tasks, using only the training data for the corresponding task. We consider a limited hyperparameter sweep with batch sizes ∈ {16, 32} and learning rates ∈ {1e-5, 2e-5, 3e-5}, with a linear warmup for the first 6% of steps followed by a linear decay to 0. We finetune for 10 epochs with early stopping based on each task's dev set. The rest of the hyperparameters remain the same as during pretraining. In this setting, we report the median development set results for each task over five random initializations, without model ensembling.

In the second setting (ensembles, test), we compare RoBERTa to other approaches on the test set via the GLUE leaderboard. While many submissions to the GLUE leaderboard depend on multi-task 2019) and Yang et al. (2019) , respectively. finetuning, our submission depends only on single-task finetuning. For RTE, STS and MRPC we finetune starting from the MNLI single-task model, following Phang et al. (2018) . We explore a slightly wider hyperparameter space, described in Appendix C, and ensemble between 5 and 7 models per task. Two of the GLUE tasks require task-specific finetuning approaches to achieve competitive leaderboard results; these approaches are described in Appendix B.

We present our results in Table 4 . In the first setting (single-task, dev), RoBERTa achieves state-of-the-art results on all 9 of the GLUE task development sets. Crucially, RoBERTa uses the same masked language modeling pretraining objective and architecture as BERT LARGE , yet consistently outperforms both BERT LARGE and XLNet LARGE . This raises questions about the relative importance of model architecture and pretraining objective, compared to more mundane details like dataset size and training time that we explore in this work. A more comprehensive comparison of the BERT and XLNet pretraining objectives is needed, but is left to future work.

In the second setting (ensembles, test), we submit RoBERTa to the GLUE leaderboard and achieve state-of-the-art results on 4 out of 9 tasks and the highest average score to date. Notably, RoBERTa does not depend on multi-task finetuning, and we expect future work may further improve these results by incorporating more sophisticated multi-task finetuning procedures.

5.2. SQUAD RESULTS:

We adopt a much simpler approach for SQuAD compared to past work. While BERT (Devlin et al., 2019) and XLNet (Yang et al., 2019) augment their training data with additional QA datasets, we only finetune RoBERTa using the provided SQuAD training data. We also use a single learning rate for all layers, in contrast to the custom layer-wise learning rate scheduled used by Yang et al. (2019) .

For SQuAD v1.1 we follow the same finetuning procedure as Devlin et al. (2019) . For SQuAD v2.0, we additionally classify whether a given question is answerable; we train this classifier jointly with the span predictor by summing the classification and span loss terms.

Model:

Accuracy 

We present our results in Table 5 . On the SQuAD v1.1 development set, RoBERTa matches the state-of-the-art set by XLNet. On the SQuAD v2.0 development set, RoBERTa sets a new state-of-the-art, improving over XLNet by 0.4 points (EM) and 0.6 points (F1).

We also submit RoBERTa to the public SQuAD 2.0 leaderboard. Most of the top systems build upon either BERT (Devlin et al., 2019) or XLNet (Yang et al., 2019) and therefore rely on additional external training data. Our single RoBERTa model outperforms all but one of the single model submissions, and is the top scoring system among those that do not rely on additional external data.

5.3. RACE RESULTS:

In RACE, systems are provided with a passage of text, an associated question, and must classify which of four candidate answers is correct. We modify RoBERTa for this task by concatenating each candidate answer with the corresponding question and passage. We encode each of these four sequences and pass the resulting [CLS] representations through a fully-connected layer, which is used to predict the correct answer. We truncate question-answer pairs that are longer than 128 tokens and, if needed, the passage so that the total length is at most 512 tokens.

Results are presented in Table 6 . RoBERTa achieves state-of-the-art accuracy across all settings.

A STATIC VS. DYNAMIC MASKING Yang et al. (2019) . We find that our reimplementation with static masking performs similar to the original BERT model, and dynamic masking is comparable or slightly better than static masking.

B TASK-SPECIFIC MODIFICATIONS FOR GLUE:

Two of the GLUE tasks require task-specific finetuning approaches to achieve competitive leaderboard results:

QNLI Recent submissions on the GLUE leaderboard adopt a pairwise ranking formulation for the QNLI task, in which candidate answers are mined from the training set and compared to one another, and a single (question, candidate) pair is classified as positive (Liu et al., 2019b; a; Yang et al., 2019) . This formulation significantly simplifies the task, but is not directly comparable to BERT (Devlin et al., 2019) . Following recent work, we adopt the ranking approach for our test submission, but for direct comparison with BERT all reported development set results are based on a pure classification approach.

WNLI We found the provided NLI-format data to be challenging to work with. Instead we use the reformatted WNLI data from SuperGLUE (Wang et al., 2019a) , which indicates the span of the query pronoun and referent. We then finetune RoBERTa using a variation of the approach from Kocijan et al. (2019) . In particular, for a given input sentence, we first use spaCy (Honnibal & Montani, 2017) to extract additional candidate noun phrases from the sentence, and then finetune our model so that it assigns higher scores to positive referent phrases than for any of the generated negative candidate phrases.

In contrast to Kocijan et al. (2019) , who finetune BERT using a margin ranking loss between (query, candidate) pairs, we instead use a single cross entropy loss term over the log-probabilities for the query and all mined candidates. This reduces the number of hyperparameters that need to be tuned and in practice produces more stable results on the development set. Our best model achieved 92.3% development set accuracy, compared to 90.2% accuracy for the margin loss approach.

One unfortunate consequence of our overall approach is that we can only make use of the (Dagan et al., 2006; Bar Haim et al., 2006; Giampiccolo et al., 2007; Bentivogli et al., 2009) and Winograd NLI (WNLI) (Levesque et al., 2011) . We also evaluate RoBERTa on the SuperGLUE benchmark (Wang et al., 2019a) , which consists of 8 natural language understanding tasks. 12 We largely follow the same setup for SuperGLUE as we did for GLUE, with several task-specific modifications:

• BoolQ and MultiRC: we follow the same input format as the Wang et al. (2019a) baseline.

• CB: we finetune starting from the MNLI model, following Phang et al. (2018) .

• COPA: we concatenate the premise and each alternative with because and so markers for cause and effect questions, respectively. This input format more closely matches the pretraining data format and provides better results in practice. • ReCoRD: during training we adopt a pairwise ranking formulation with one negative and positive entity for each (passage, query) pair. At evaluation time, we pick the entity with the highest score for each question. • WiC: we input the pair of sentences as normal. We then feed the concatenation of the representations of the two marked words and the [CLS] token to the classification layer. • RTE and WSC: we reused our submission to the GLUE leaderboard.

In Table 11 Table 12 : Results on XNLI (Conneau et al., 2018) for RoBERTa LARGE in the TRANSLATE-TEST setting. We report macro-averaged accuracy (∆) using the provided English translations of the XNLI test sets. RoBERTa achieves state of the art results on all 15 languages.

:

Our models and code are available at: anonymous URL.

These other methods could possibly improve with more tuning as well; we leave this to future work.

Yang et al. (2019) use the same dataset but report having only 13GB of text after data cleaning. This is most likely due to subtle differences in the underlying data collection or preprocessing.

We use news-please(Hamborg et al., 2017) to collect and extract CC-NEWS. CC-NEWS is similar to the REALNEWS dataset described inZellers et al. (2019).

The authors and their affiliated institutions are not affiliated with the creation of the OpenWebText dataset.

Studying architectural changes, including larger architectures, is an important area for future work.

Even without large scale parallel hardware, large batch training can improve training efficiency through gradient accumulation -i.e., accumulating gradients from multiple mini-batches before each optimization step.

You et al. (2019) train BERT with even larger batch sizes, up to 32K sequences. We leave further exploration of the limits of large batch training to future work.

A more careful analysis disentangling data size and diversity is needed, but is left to future work.

RELATED WORKPretraining methods have been designed with different training objectives, including language modeling(Dai & Le, 2015;Peters et al., 2018;Howard & Ruder, 2018), machine translation(McCann et al., 2017), and masked language modeling(Devlin et al., 2019;Lample & Conneau, 2019). Many recent papers have used a basic recipe of finetuning models for each end task(Howard & Ruder, 2018;Radford et al., 2018), and pretraining with some variant of a masked language model objective. However, newer methods have improved performance by multi-task fine tuning(Dong et al., 2019), incorporating entity embeddings(Sun et al., 2019), span prediction(Joshi et al., 2019), and multiple variants of autoregressive pretraining(Song et al., 2019;Chan et al., 2019;Yang et al., 2019). Performance is also typically improved by training bigger models on more data(Devlin et al., 2019;Baevski et al., 2019;Yang et al., 2019;Radford et al., 2019). Our goal was to replicate, simplify, and better tune the training of BERT, as a reference point for better understanding the relative performance of all of these methods.

While we only use the provided WNLI training data, our results could potentially be improved by augmenting this with additional pronoun disambiguation datasets.

11 The GLUE datasets are: CoLA(Warstadt et al., 2018), Stanford Sentiment Treebank (SST)(Socher et al., 2013), Microsoft Research Paragraph Corpus (MRPC)(Dolan & Brockett, 2005), Semantic Textual Similarity Benchmark (STS)(Agirre et al., 2007), Quora Question Pairs (QQP)(Iyer et al., 2016), Multi-Genre NLI (MNLI)(Williams et al., 2018), Question NLI (QNLI)(Rajpurkar et al., 2016), Recognizing Textual Entail-

The SuperGLUE datasets are: BoolQ(Clark et al., 2019), CommitmentBank (CB)(De Marneffe et al., 2019), Choice of Plausible Alternatives (COPA)(Roemmele et al., 2011), Multi-Sentence Reading Comprehension (MultiRC)(Khashabi et al., 2018), Reading Comprehension with Commonsense Reasoning (ReCoRD)(Zhang et al., 2018), Recognizing Textual Entailment (RTE)(Dagan et al., 2006;Bar Haim et al., 2006;Giampiccolo et al., 2007;Bentivogli et al., 2009), Words in Context (WiC)(Pilehvar & Camacho- Collados, 2019), the Winograd Schema Challenge (WSC)(Levesque et al., 2011), and Winogender Schema Diagnostics(Poliak et al., 2018;Rudinger et al., 2018).

Figures and Tables:

Prediction (NSP) is a binary classification loss for predicting whether two segments follow each other in the original text. Positive examples are created by taking consecutive sentences from the text corpus. Negative examples are created by pairing segments from different documents. Positive and negative examples are sampled with equal probability.

Development set results for base models pretrained over BOOKCORPUS and WIKIPEDIA. All models are trained for 1M steps with a batch size of 256 sequences. We report F1 for SQuAD and accuracy for MNLI-m, SST-2 and RACE. Reported results are medians over five random initializations (seeds). Results for BERT BASE and XLNet BASE are fromYang et al. (2019).

Table content: <table><tr><td>Model</td><td colspan="4">SQuAD 1.1/2.0 MNLI-m SST-2 RACE</td></tr><tr><td colspan="2">Our reimplementation (with NSP loss):</td><td/><td/><td/></tr><tr><td>SEGMENT-PAIR</td><td>90.4/78.7</td><td>84.0</td><td>92.9</td><td>64.2</td></tr><tr><td>SENTENCE-PAIR</td><td>88.7/76.2</td><td>82.9</td><td>92.1</td><td>63.0</td></tr><tr><td colspan="2">Our reimplementation (without NSP loss):</td><td/><td/><td/></tr><tr><td>FULL-SENTENCES</td><td>90.4/79.1</td><td>84.7</td><td>92.5</td><td>64.8</td></tr><tr><td>DOC-SENTENCES</td><td>90.6/79.7</td><td>84.7</td><td>92.7</td><td>65.6</td></tr><tr><td>BERT BASE</td><td>88.5/76.3</td><td>84.3</td><td>92.8</td><td>64.3</td></tr><tr><td>XLNet BASE (K = 7)</td><td>-/81.3</td><td>85.8</td><td>92.7</td><td>66.1</td></tr><tr><td>XLNet BASE (K = 6)</td><td>-/81.0</td><td>85.6</td><td>93.4</td><td>66.7</td></tr></table>

Perplexity on held-out validation data and dev set accuracy on MNLI-m and SST-2 for various batch sizes (# sequences) as we vary the number of passes (epochs) through the BOOKS + WIKI data. Reported results are medians over five random initializations (seeds). The learning rate is tuned for each batch size. All results are for BERT BASE with FULL-SENTENCE inputs.

Table content: <table><tr><td colspan="7">batch size learning rate epochs steps perplexity MNLI-m SST-2</td></tr><tr><td>256</td><td>1e-4</td><td>32</td><td>1M</td><td>3.99</td><td>84.7</td><td>92.5</td></tr><tr><td/><td/><td>32</td><td>125K</td><td>3.68</td><td>85.2</td><td>93.1</td></tr><tr><td>2K</td><td>7e-4</td><td>64</td><td>250K</td><td>3.59</td><td>85.3</td><td>94.1</td></tr><tr><td/><td/><td>128</td><td>500K</td><td>3.51</td><td>85.4</td><td>93.5</td></tr><tr><td/><td/><td>32</td><td>31K</td><td>3.77</td><td>84.4</td><td>93.2</td></tr><tr><td>8K</td><td>1e-3</td><td>64</td><td>63K</td><td>3.60</td><td>85.3</td><td>93.5</td></tr><tr><td/><td/><td>128</td><td>125K</td><td>3.50</td><td>85.8</td><td>94.1</td></tr></table>

used a character-level BPE vocabulary of size 30K. We instead adopt the larger byte-level BPE vocabulary of size 50K introduced in Radford et al. (2019), which uses bytes rather than unicode characters as the base subword units and can therefore encode any input text without introducing "unknown" tokens. This adds approximately 15M and 20M extra parameters for BERT BASE and BERT LARGE , respectively.

Table content: <table/>

Development set results for RoBERTa as we pretrain over more data (16GB → 160GB of text) and pretrain for longer (100K → 300K → 500K steps). Each row accumulates improvements from the rows above. RoBERTa matches the architecture and training objective of BERT LARGE . Results for BERT LARGE and XLNet LARGE are from Devlin et al. (2019) and Yang et al. (2019), respectively. Complete results on all GLUE tasks can be found in Appendix C.

Table content: <table><tr><td>Model</td><td>data</td><td colspan="5">batch size steps SQuAD (v1.1/2.0) MNLI-m SST-2</td></tr><tr><td>RoBERTa</td><td/><td/><td/><td/><td/><td/></tr><tr><td>with BOOKS + WIKI</td><td>16GB</td><td>8K</td><td>100K</td><td>93.6/87.3</td><td>89.0</td><td>95.3</td></tr><tr><td colspan="2">+ additional data ( §3.2) 160GB</td><td>8K</td><td>100K</td><td>94.0/87.7</td><td>89.3</td><td>95.6</td></tr><tr><td>+ pretrain longer</td><td>160GB</td><td>8K</td><td>300K</td><td>94.4/88.7</td><td>90.0</td><td>96.1</td></tr><tr><td>+ pretrain even longer</td><td>160GB</td><td>8K</td><td>500K</td><td>94.6/89.4</td><td>90.2</td><td>96.4</td></tr><tr><td>BERTLARGE</td><td/><td/><td/><td/><td/><td/></tr><tr><td>with BOOKS + WIKI</td><td>13GB</td><td>256</td><td>1M</td><td>90.9/81.8</td><td>86.6</td><td>93.7</td></tr><tr><td>XLNetLARGE</td><td/><td/><td/><td/><td/><td/></tr><tr><td>with BOOKS + WIKI</td><td>13GB</td><td>256</td><td>1M</td><td>94.0/87.8</td><td>88.4</td><td>94.4</td></tr><tr><td>+ additional data</td><td>126GB</td><td>2K</td><td>500K</td><td>94.5/88.8</td><td>89.8</td><td>95.6</td></tr></table>

Results on GLUE. All results are based on a 24-layer architecture. BERT LARGE and XLNet LARGE results are from Devlin et al. (2019) and Yang et al. (2019), respectively. RoBERTa results on the dev set are a median over five runs. RoBERTa results on the test set are ensembles of single-task models. For RTE, STS and MRPC we finetune starting from the MNLI model.

Table content: <table><tr><td/><td>MNLI</td><td colspan="9">QNLI QQP RTE SST MRPC CoLA STS WNLI Avg</td></tr><tr><td colspan="3">Single-task single models on dev</td><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>BERTLARGE</td><td>86.6/-</td><td>92.3</td><td>91.3</td><td colspan="2">70.4 93.2</td><td>88.0</td><td>60.6</td><td>90.0</td><td>-</td><td>-</td></tr><tr><td>XLNetLARGE</td><td>89.8/-</td><td>93.9</td><td>91.8</td><td colspan="2">83.8 95.6</td><td>89.2</td><td>63.6</td><td>91.8</td><td>-</td><td>-</td></tr><tr><td>RoBERTa</td><td>90.2/90.2</td><td>94.7</td><td>92.2</td><td colspan="2">86.6 96.4</td><td>90.9</td><td>68.0</td><td>92.4</td><td>91.3</td><td>-</td></tr><tr><td colspan="5">Ensembles on test (from leaderboard as of July 25, 2019)</td><td/><td/><td/><td/><td/></tr><tr><td>ALICE</td><td>88.2/87.9</td><td>95.7</td><td>90.7</td><td colspan="2">83.5 95.2</td><td>92.6</td><td>68.6</td><td>91.1</td><td>80.8</td><td>86.3</td></tr><tr><td>MT-DNN</td><td>87.9/87.4</td><td>96.0</td><td>89.9</td><td colspan="2">86.3 96.5</td><td>92.7</td><td>68.4</td><td>91.1</td><td>89.0</td><td>87.6</td></tr><tr><td>XLNet</td><td>90.2/89.8</td><td>98.6</td><td>90.3</td><td colspan="2">86.3 96.8</td><td>93.0</td><td>67.8</td><td>91.6</td><td>90.4</td><td>88.4</td></tr><tr><td>RoBERTa</td><td>90.8/90.2</td><td>98.9</td><td>90.2</td><td colspan="2">88.2 96.7</td><td>92.3</td><td>67.8</td><td>92.2</td><td>89.0</td><td>88.5</td></tr><tr><td>Model</td><td colspan="2">SQuAD 1.1 EM F1</td><td colspan="2">SQuAD 2.0 EM F1</td><td colspan="2">Model</td><td/><td/><td colspan="2">SQuAD 2.0 EM F1</td></tr><tr><td colspan="5">Single models on dev, w/o data augmentation</td><td colspan="6">Single models on test (as of July 25, 2019)</td></tr><tr><td>BERT LARGE</td><td colspan="3">84.1 90.9 79.0</td><td>81.8</td><td colspan="2">XLNet LARGE</td><td/><td/><td colspan="2">86.3  † 89.1  †</td></tr><tr><td colspan="4">XLNet LARGE 89.0 94.5 86.1</td><td>88.8</td><td colspan="2">RoBERTa</td><td/><td/><td>86.8</td><td>89.8</td></tr><tr><td>RoBERTa</td><td colspan="3">88.9 94.6 86.5</td><td>89.4</td><td colspan="6">XLNet + SG-Net Verifier 87.0  † 89.9  †</td></tr></table>

Results on SQuAD. † indicates results that depend on additional external training data. RoBERTa uses only the provided SQuAD data in both dev and test settings. BERT LARGE and XLNet LARGE results are from Devlin et al. (

Table content: <table/>

Middle High Results on the RACE test set. BERT LARGE and XLNet LARGE results from Yang et al. (2019).

Table content: <table><tr><td colspan="4">Single models on test (as of July 25, 2019)</td></tr><tr><td>BERT LARGE</td><td>72.0</td><td>76.6</td><td>70.1</td></tr><tr><td>XLNet LARGE</td><td>81.7</td><td>85.4</td><td>80.2</td></tr><tr><td>RoBERTa</td><td>83.2</td><td>86.5</td><td>81.3</td></tr></table>

Comparison between the published BERT BASE results fromDevlin et al. (2019) to our reimplementation with either static or dynamic masking. We report F1 for SQuAD and accuracy for MNLI-m and SST-2. Reported results are medians over 5 random initializations (seeds). Reference results are from

Table content: <table><tr><td colspan="4">Masking SQuAD 2.0 MNLI-m SST-2</td></tr><tr><td>reference</td><td>76.3</td><td>84.3</td><td>92.8</td></tr><tr><td colspan="2">Our reimplementation:</td><td/><td/></tr><tr><td>static</td><td>78.3</td><td>84.3</td><td>92.5</td></tr><tr><td>dynamic</td><td>78.7</td><td>84.0</td><td>92.9</td></tr></table>

positive training examples, which excludes over half of the provided training data. 10C FULL RESULTS ON GLUEIn Table8we present the full set of development set results for RoBERTa on all 9 GLUE datasets.11  We present results for a LARGE configuration with 355M parameters that follows BERT LARGE , as well as a BASE configuration with 125M parameters that follows BERT BASE . Development set results on GLUE tasks for various configurations of RoBERTa. All results are a median over five runs.

Table content: <table><tr><td colspan="8">MNLI QNLI QQP RTE SST MRPC CoLA STS</td></tr><tr><td>RoBERTa BASE</td><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>+ all data + 500k steps</td><td>87.6</td><td>92.8</td><td>91.9</td><td>78.7 94.8</td><td>90.2</td><td>63.6</td><td>91.2</td></tr><tr><td>RoBERTa LARGE</td><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>with BOOKS + WIKI</td><td>89.0</td><td>93.9</td><td>91.9</td><td>84.5 95.3</td><td>90.2</td><td>66.3</td><td>91.6</td></tr><tr><td>+ additional data ( §3.2)</td><td>89.3</td><td>94.0</td><td>92.0</td><td>82.7 95.6</td><td>91.4</td><td>66.1</td><td>92.2</td></tr><tr><td>+ pretrain longer 300k</td><td>90.0</td><td>94.5</td><td>92.2</td><td>83.3 96.1</td><td>91.1</td><td>67.4</td><td>92.3</td></tr><tr><td>+ pretrain longer 500k</td><td>90.2</td><td>94.7</td><td>92.2</td><td>86.6 96.4</td><td>90.9</td><td>68.0</td><td>92.4</td></tr><tr><td colspan="4">D PRETRAINING HYPERPARAMETERS</td><td/><td/><td/><td/></tr><tr><td colspan="2">Hyperparam</td><td colspan="4">RoBERTa LARGE RoBERTa BASE</td><td/><td/></tr><tr><td colspan="2">Number of Layers</td><td/><td>24</td><td>12</td><td/><td/><td/></tr><tr><td colspan="2">Hidden size</td><td/><td>1024</td><td>768</td><td/><td/><td/></tr><tr><td colspan="3">FFN inner hidden size</td><td>4096</td><td>3072</td><td/><td/><td/></tr><tr><td colspan="2">Attention heads</td><td/><td>16</td><td>12</td><td/><td/><td/></tr><tr><td colspan="2">Attention head size</td><td/><td>64</td><td>64</td><td/><td/><td/></tr><tr><td>Dropout</td><td/><td/><td>0.1</td><td>0.1</td><td/><td/><td/></tr><tr><td colspan="2">Attention Dropout</td><td/><td>0.1</td><td>0.1</td><td/><td/><td/></tr><tr><td colspan="2">Warmup Steps</td><td/><td>24k</td><td>24k</td><td/><td/><td/></tr><tr><td colspan="2">Peak Learning Rate</td><td/><td>4e-4</td><td>6e-4</td><td/><td/><td/></tr><tr><td>Batch Size</td><td/><td/><td>8k</td><td>8k</td><td/><td/><td/></tr><tr><td colspan="2">Weight Decay</td><td/><td>0.01</td><td>0.01</td><td/><td/><td/></tr><tr><td>Max Steps</td><td/><td/><td>500k</td><td>500k</td><td/><td/><td/></tr><tr><td colspan="3">Learning Rate Decay</td><td>Linear</td><td colspan="2">Linear</td><td/><td/></tr><tr><td>Adam</td><td/><td/><td>1e-6</td><td>1e-6</td><td/><td/><td/></tr><tr><td>Adam β 1</td><td/><td/><td>0.9</td><td>0.9</td><td/><td/><td/></tr><tr><td>Adam β 2</td><td/><td/><td>0.98</td><td>0.98</td><td/><td/><td/></tr><tr><td colspan="2">Gradient Clipping</td><td/><td>0.0</td><td>0.0</td><td/><td/><td/></tr></table>

Hyperparameters for pretraining RoBERTa LARGE and RoBERTa BASE .

Table content: <table><tr><td colspan="3">E FINETUNING HYPERPARAMETERS</td><td/><td/></tr><tr><td>Hyperparam</td><td colspan="2">RACE SQuAD</td><td>GLUE</td><td>SuperGLUE</td></tr><tr><td>Learning Rate</td><td>1e-5</td><td>1.5e-5</td><td colspan="2">{1e-5, 2e-5, 3e-5} {1e-5, 2e-5, 3e-5}</td></tr><tr><td>Batch Size</td><td>16</td><td>48</td><td>{16, 32}</td><td>32</td></tr><tr><td>Weight Decay</td><td>0.1</td><td>0.01</td><td>0.1</td><td>0.1</td></tr><tr><td>Max Epochs</td><td>4</td><td>2</td><td>10</td><td>{10, 50}</td></tr><tr><td colspan="2">Learning Rate Decay Linear</td><td>Linear</td><td>Linear</td><td>Linear</td></tr><tr><td>Warmup ratio</td><td>0.06</td><td>0.06</td><td>0.06</td><td>0.10</td></tr></table>

Hyperparameters for finetuning RoBERTa LARGE on RACE, SQuAD and GLUE. We select the best hyperparameter values based on the median of 5 random seeds for each task.

Table content: <table/>

Results on SuperGLUE. All results are based on a 24-layer architecture. RoBERTa results on the development set are a median over five runs. RoBERTa results on the test set are ensembles of single-task models. Averages are obtained from the SuperGLUE leaderboard.

Table content: <table><tr><td colspan="4">F RESULTS ON SUPERGLUE</td><td/><td/><td/><td/><td/></tr><tr><td/><td>BoolQ</td><td>CB</td><td colspan="6">COPA MultiRC ReCoRD RTE WiC WSC Avg</td></tr><tr><td colspan="3">Single-task single models on dev</td><td/><td/><td/><td/><td/><td/></tr><tr><td>BERT++</td><td>80.1</td><td>96.4/95.0</td><td>78.0</td><td colspan="2">70.7/24.7 70.6/69.8 82.3</td><td>74.9</td><td>68.3</td><td>74.6</td></tr><tr><td>RoBERTa</td><td>86.9</td><td>98.2/-</td><td>94.0</td><td>85.7/-</td><td>89.5/89.0 86.6</td><td>75.6</td><td>-</td><td>-</td></tr><tr><td colspan="5">Ensembles on test (from leaderboard as of August 12, 2019)</td><td/><td/><td/><td/></tr><tr><td>BERT</td><td>77.4</td><td>75.7/83.6</td><td>70.6</td><td colspan="2">70.0/24.1 72.0/71.3 71.7</td><td>69.6</td><td>64.4</td><td>69.0</td></tr><tr><td>BERT++</td><td>79.0</td><td>84.8/90.4</td><td>73.8</td><td colspan="2">70.0/24.1 72.0/71.3 79.0</td><td>69.6</td><td>64.4</td><td>71.5</td></tr><tr><td>Outside Best</td><td>80.4</td><td>-</td><td>84.4</td><td colspan="2">70.4/24.5 74.8/73.0 82.7</td><td>-</td><td>-</td><td>-</td></tr><tr><td>RoBERTa</td><td>87.1</td><td>90.5/95.2</td><td>90.6</td><td colspan="2">84.4/52.5 90.6/90.0 88.2</td><td>69.9</td><td>89.0</td><td>84.6</td></tr><tr><td>Human (est.)</td><td>89.0</td><td>95.8/98.9</td><td>100.0</td><td colspan="2">81.8/51.9 91.7/91.3 93.6</td><td colspan="3">80.0 100.0 89.8</td></tr></table>

we present RoBERTa results on the 8 SuperGLUE datasets. RoBERTa achieves stateof-the-art results on the development and test sets for BoolQ, CB, COPA, MultiRC and ReCoRD and the highest average score to date on the SuperGLUE leaderboard. Machine translation baselines (TRANSLATE-TEST) XLM (MLM+TLM) 85.0 79.0 79.5 78.1 77.8 77.6 75.5 73.7 73.7 70.8 70.4 73.6 69.0 64.7 65.1 74.2 XLM-en 88.8 81.4 82.3 80.1 80.3 80.9 76.2 76.0 75.4 72.0 71.9 75.6 70.0 65.8 65.8 76.2 RoBERTa 91.3 82.9 84.3 81.2 81.7 83.1 78.3 76.8 76.6 74.2 74.0 77.5 70.9 66.6 66.8 77.8

Table content: <table><tr><td colspan="3">G RESULTS ON XNLI</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>en</td><td>fr</td><td>es</td><td>de</td><td>el</td><td>bg</td><td>ru</td><td>tr</td><td>ar</td><td>vi</td><td>th</td><td>zh</td><td>hi</td><td>sw</td><td>ur</td><td>∆</td></tr></table>

Peer review (iclr2020):
This paper presents a detailed replication study of the BERT pre-training model considering alternative design choices such as dynamic masking, removal of next sentence prediction loss, longer training time, larger batch sizes, additional training data, training on longer single document or cross-document sequences etc. to demonstrate their efficacy on several benchmark datasets and tasks by achieving the new state-of-the-art results. Overall, the paper is very well-written and the experimental setups are reasonable and thoroughly presented that would benefit the community for future research and exploration. However, I am not sure if the paper presents a case of adequate novelty in terms of ideas as many of them are rather obvious and the current state-of-the-art models could also improve considerably using similar experimental setups, which authors also acknowledged in footnote 2. Other comments: - Section 3.1: please clarify how exactly setting Beta2=0.98 would improve stability when training with large batch sizes. - It*s not clear what exactly was the motivation for proposing full-sentences and doc-sentences input formats. Please explain. - Although the presented metrics show that the removal of NSP loss helps, however, no explanation was provided based on qualitative evaluation as to why this is the case. Some task-specific examples would have been nice to discuss the effects of NSP loss. - Section 5: Please provide details on why you think the longest-trained model does not appear to overfit.