RoBERTa: A Robustly Optimized BERT Pretraining Approach

1 INTRODUCTION . Self-training methods such as ELMo ( Peters et al. , 2018 ) , GPT ( Radford et al. , 2018 ) , BERT ( Devlin et al. , 2019 ) , XLM ( Lample & Conneau , 2019 ) , and XLNet ( Yang et al. , 2019 ) have brought significant performance gains , but it can be challenging to determine which aspects of the methods contribute the most . Training is computationally expensive , limiting the amount of tuning that can be done , and modeling advances are often conflated with changes in data size or composition . We present a replication study of BERT pretraining ( Devlin et al. , 2019 ) , which includes a careful evaluation of the effects of hyperparameter tuning and training set size . We find that BERT was significantly undertrained and propose an improved training recipe , which we call RoBERTa , that can match or exceed the performance of all of the post-BERT methods . Our modifications are simple , they include : ( 1 ) training the model longer , with bigger batches , over more data ; ( 2 ) removing the next sentence prediction objective ; ( 3 ) training on longer sequences ; and ( 4 ) dynamically changing the masking pattern applied to the training data . We also collect a large new dataset ( CC-NEWS ) of comparable size to other privately used datasets , to better control for training set size effects . When controlling for training data , our improved training procedure improves upon the published BERT results on the GLUE ( Wang et al. , 2019b ) and SQuAD ( Rajpurkar et al. , 2016 ) benchmarks . When trained for longer over additional data , our model achieves a score of 88.5 on the public GLUE leaderboard , matching the 88.4 reported by Yang et al . ( 2019 ) . Our model establishes a new stateof-the-art on 4/9 of the GLUE tasks , as well as RACE ( Lai et al. , 2017 ) , SuperGLUE ( Wang et al. , 2019a ) , and XNLI ( Conneau et al. , 2018 ) , and matches the state-of-the-art on SQuAD . Overall , we re-establish that BERT ’ s masked language model training objective is competitive with recently proposed alternatives such as perturbed autoregressive language modeling ( Yang et al. , 2019 ) .2 In summary , the contributions of this paper are : ( 1 ) We present a set of important BERT design choices and training strategies and introduce alternatives that lead to better downstream task performance ; ( 2 ) We use a novel dataset , CC-NEWS , and confirm that using more data for pretraining further improves performance on downstream tasks ; ( 3 ) Our training improvements show that masked language model pretraining , under the right design choices , is competitive with all other recently published methods . We release our model , pretraining and fine-tuning code . 1Our models and code are available at : anonymous URL . 2These other methods could possibly improve with more tuning as well ; we leave this to future work . 2 BACKGROUND . Setup : BERT ( Devlin et al. , 2019 ) takes as input a concatenation of two segments ( sequences of tokens ) , x1 , . . . , xN and y1 , . . . , yM . Segments usually consist of more than one natural sentence . The two segments are presented as a single input sequence to BERT with special tokens delimiting them : [ CLS ] , x1 , . . . , xN , [ SEP ] , y1 , . . . , yM , [ EOS ] . M andN are constrained such thatM+N < T , where T is a parameter that controls the maximum sequence length during training . Architecture : BERT uses the now ubiquitous transformer architecture ( Vaswani et al. , 2017 ) , which we will not review in detail . We use a transformer architecture with L layers . Each block has A self-attention heads and hidden dimension H . Training Objectives : BERT uses two pretraining objectives : masked language modeling and next sentence prediction . For the Masked Language Model ( MLM ) objective , BERT is trained via a crossentropy loss to predict 15 % of the input tokens , selected at random . To prevent the model from cheating , 80 % of these selected tokens are replaced by a special [ MASK ] symbol in the input , 10 % are replaced by a random token from the vocabulary , and 10 % are left unchanged . Next Sentence Prediction ( NSP ) is a binary classification loss for predicting whether two segments follow each other in the original text . Positive examples are created by taking consecutive sentences from the text corpus . Negative examples are created by pairing segments from different documents . Positive and negative examples are sampled with equal probability . Optimization : BERT is optimized with AdamW ( Kingma & Ba , 2015 ) using the following parameters : β1 = 0.9 , β2 = 0.999 , = 1e-6 and decoupled weight decay of 0.01 ( Loshchilov & Hutter , 2019 ) . The learning rate is warmed up over the first 10,000 steps to a peak value of 1e-4 , and then linearly decayed . BERT trains with a dropout of 0.1 on all layers and attention weights , and a GELU activation function ( Hendrycks & Gimpel , 2016 ) . Models are pretrained for S = 1,000,000 updates , with mini-batches containing B = 256 sequences of maximum length T = 512 tokens . Data : BERT is trained on a combination of BOOKCORPUS ( Zhu et al. , 2015 ) plus English WIKIPEDIA , which totals 16GB of uncompressed text.3 3 EXPERIMENTAL SETUP . 3.1 IMPLEMENTATION . We reimplement BERT in FAIRSEQ ( Ott et al. , 2019 ) . We primarily follow the original BERT optimization hyperparameters , given in Section 2 , except for the peak learning rate and number of warmup steps , which are tuned separately for each setting . We found training to be very sensitive to the Adam epsilon term , and in some cases we obtained better performance or improved stability after tuning it . We also set β2 = 0.98 to improve stability when training with large batch sizes . We pretrain with sequences of at most T = 512 tokens . Unlike Devlin et al . ( 2019 ) , we do not randomly inject short sequences , and we do not train with a reduced sequence length for the first 90 % of updates . We train only with full-length sequences . We train with mixed precision floating point arithmetic on DGX-1 machines , each with 8 × 32GB Nvidia V100 GPUs interconnected by Infiniband ( Micikevicius et al. , 2018 ) . 3.2 DATA . BERT-style pretraining crucially relies on large quantities of text . Baevski et al . ( 2019 ) demonstrate that increasing data size can result in improved end-task performance . Several efforts have trained on datasets larger and more diverse than the original BERT ( Radford et al. , 2019 ; Yang et al. , 2019 ; Zellers et al. , 2019 ) . Unfortunately , not all of the additional datasets can be publicly released . For 3Yang et al . ( 2019 ) use the same dataset but report having only 13GB of text after data cleaning . This is most likely due to subtle differences in the underlying data collection or preprocessing . our study , we focus on gathering as much data as possible for experimentation , allowing us to match the overall quality and quantity of data as appropriate for each comparison . We consider five English-language corpora of varying sizes and domains , totaling over 160GB of uncompressed text : ( 1 & 2 ) BOOKCORPUS ( Zhu et al. , 2015 ) plus English WIKIPEDIA , which is the original data used to train BERT ( 16GB ) ; ( 3 ) CC-NEWS , which we collect from the English portion of the CommonCrawl News dataset ( Nagel , 2016 ) , containing 63 million English news articles crawled between September 2016 and February 2019 ( 76GB after filtering ) ; 4 ( 4 ) OPENWEBTEXT ( Gokaslan & Cohen , 2019 ) , an open-source recreation of the WebText corpus described in Radford et al . ( 2019 ) , containing web content extracted from URLs shared on Reddit with at least three upvotes ( 38GB ) ; 5 ( 5 ) STORIES , a dataset introduced in Trinh & Le ( 2018 ) containing a subset of CommonCrawl data filtered to match the story-like style of Winograd schemas ( 31GB ) . 3.3 EVALUATION . Following previous work , we evaluate our pretrained models by finetuning on downstream tasks : • GLUE : The General Language Understanding Evaluation ( GLUE ) benchmark ( Wang et al. , 2019b ) is a collection of 9 datasets for evaluating natural language understanding systems . Tasks are framed as either single-sentence classification or sentence-pair classification tasks . The GLUE organizers provide training and development data splits as well as a submission server and leaderboard that allows participants to evaluate and compare their systems on private held-out test data . • SQuAD : The Stanford Question Answering Dataset ( SQuAD ) provides a paragraph of context and a question . The task is to answer the question with a span extracted from the context . We evaluate on SQuAD V1.1 and V2.0 ( Rajpurkar et al. , 2016 ; 2018 ) . In V1.1 the context always contains an answer , while in V2.0 some questions are not answered in the provided context . • RACE : ReAding Comprehension from Examinations ( RACE ) ( Lai et al. , 2017 ) is a large-scale reading comprehension dataset collected from English examinations in China . The task is to choose among four possible answers to a given question , using a given passage of text as context . • Additional Benchmarks : In the Appendix we present additional results for SuperGLUE ( Wang et al. , 2019a ) and XNLI ( Conneau et al. , 2018 ) . 4 TRAINING PROCEDURE ANALYSIS . This section explores and quantifies which choices are important for successfully pretraining BERT models . We keep the model architecture fixed.6 Specifically , we begin by training BERT models with the same configuration as BERTBASE ( L = 12 , H = 768 , A = 12 , 110M params ) . 4.1 STATIC VS . DYNAMIC MASKING . As discussed in Section 2 , BERT relies on predicting randomly masked tokens . The original BERT implementation performed masking once during data preprocessing , resulting in a single static mask . To avoid repeating the same masks at every epoch , training data was duplicated 10 times prior to preprocessing , so that each training sequence was seen with the same mask only four times over the course of 40 training epochs . We instead train with dynamic masking , where we generate the masking pattern on-the-fly each time we input a sequence to the model . This becomes crucial when pretraining for more steps or with larger datasets , and additionally performs marginally better than static masking on some downstream tasks ( see Appendix A ) . 4.2 MODEL INPUT FORMAT AND NEXT SENTENCE PREDICTION . In the original BERT pretraining procedure , the model observes two concatenated document segments and is trained via an auxiliary Next Sentence Prediction ( NSP ) loss to predict whether these segments were sampled contiguously from the same document or from distinct documents . 4We use news-please ( Hamborg et al. , 2017 ) to collect and extract CC-NEWS . CC-NEWS is similar to the REALNEWS dataset described in Zellers et al . ( 2019 ) . 5The authors and their affiliated institutions are not affiliated with the creation of the OpenWebText dataset . 6Studying architectural changes , including larger architectures , is an important area for future work . The NSP objective was designed to improve performance on downstream tasks , such as Natural Language Inference ( Bowman et al. , 2015 ) , which require predicting relationships between pairs of sentences . Devlin et al . ( 2019 ) observe that removing NSP hurts performance , with significant performance degradation on QNLI , MNLI , and SQuAD 1.1 . However , recent work has questioned the necessity of the NSP loss ( Lample & Conneau , 2019 ; Yang et al. , 2019 ; Joshi et al. , 2019 ) . To better understand this discrepancy , we compare several alternative training formats : • SEGMENT-PAIR+NSP : This follows the original input format used in BERT ( Devlin et al. , 2019 ) , with the NSP loss . Each input has a pair of segments , which can each contain multiple natural sentences , but the total combined length must be less than 512 tokens . • SENTENCE-PAIR+NSP : Each input contains a pair of natural sentences , either sampled from a contiguous portion of one document or from separate documents . Since these inputs are significantly shorter than 512 tokens , we increase the batch size so that the total number of tokens remains similar to SEGMENT-PAIR+NSP . We retain the NSP loss . • FULL-SENTENCES : Each input is packed with full sentences sampled contiguously from one or more documents , such that the total length is at most 512 tokens . Inputs may cross document boundaries . When we reach the end of one document , we begin sampling sentences from the next document and add an extra separator token between documents . We remove the NSP loss . • DOC-SENTENCES : Inputs are constructed similarly to FULL-SENTENCES , except that they may not cross document boundaries . Inputs sampled near the end of a document may be shorter than 512 tokens , so we dynamically increase the batch size in these cases to achieve a similar number of total tokens as FULL-SENTENCES . We remove the NSP loss . Results Table 1 shows results for the four different settings . We first compare the original SEGMENT-PAIR input format from Devlin et al . ( 2019 ) to the SENTENCE-PAIR format ; both formats retain the NSP loss , but the latter uses single sentences . We find that using individual sentences hurts performance on downstream tasks , which we hypothesize is because the model is not able to learn long-range dependencies . We next compare training without the NSP loss and training with blocks of text from a single document ( DOC-SENTENCES ) . We find that this setting outperforms the originally published BERTBASE results and that removing the NSP loss matches or slightly improves downstream task performance , in contrast to Devlin et al . ( 2019 ) . It is possible that the original BERT implementation may only have removed the loss term while still retaining the SEGMENT-PAIR input format . Finally we find that restricting sequences to come from a single document ( DOC-SENTENCES ) performs slightly better than packing sequences from multiple documents ( FULL-SENTENCES ) . However , because the DOC-SENTENCES format results in variable batch sizes , we use FULL-SENTENCES in the remainder of our experiments for easier comparison with related work .

Summary: This paper presents a replication study of BERT pretraining and carefully measures the impact of many key hyperparameters and training data size. It shows that BERT was significantly undertrained and propose an improved training recipe called RoBERTa. The key ideas are: (i) training longer with bigger batches over more data, (ii) removing NSP, (iii) training over long sequences, and (iv) dynamically changing the masking pattern. The proposed RoBERTa achieves/matches state-of-the-art performance on many standard NLU downstream tasks.