RoBERTa: A Robustly Optimized BERT Pretraining Approach
Authors: Yinhan Liu
Myle Ott
Naman Goyal
Jingfei Du
Mandar Joshi
Danqi Chen
Omer Levy
Mike Lewis
Luke Zettlemoyer
Veselin Stoyanov
Venue:  (2020)
Keywords: Deep learning
language representation learning
natural language understanding

Abstract: Language model pretraining has led to significant performance gains but careful comparison between different approaches is challenging. Training is computationally expensive, often done on private datasets of different sizes, and, as we will show, hyperparameter choices have significant impact on the final results. We present a replication study of BERT pretraining (Devlin et al., 2019) that carefully measures the impact of many key hyperparameters and training data size. We find that BERT was significantly undertrained, and can match or exceed the performance of every model published after it. Our best model achieves state-of-the-art results on GLUE, RACE, SQuAD, SuperGLUE and XNLI. These results highlight the importance of previously overlooked design choices, and raise questions about the source of recently reported improvements. We release our models and code.

Under review as a conference paper at ICLR 2020
ROBERTA: A ROBUSTLY OPTIMIZED BERT PRE- TRAINING APPROACH
Anonymous authors Paper under double-blind review
ABSTRACT
Language model pretraining has led to significant performance gains but careful comparison between different approaches is challenging. Training is computation- ally expensive, often done on private datasets of different sizes, and, as we show, hyperparameter choices have significant impact the final results. We present a replication study of BERT pretraining (Devlin et al. (219)) that carefully mea- sures the impact of many key hyperparameters and training data size. We find that BERT was significantly undertrained, and can match or exceed the performance of every model published after Our best model achieves state-of-the-ar results on GLUE, RACE, SQuAD, SuperGLUE and XNLI. These results highlight the importance of previously overlooked design choices, and raise questions about the source of recently reported improvements. We release our models and code
INTRODUCTION
Self-training methods such as ELMo (Peters al. 2018). GPT (Radford al. 20188. BERT (De- vlin et al. 2019). XLM (Lample & Conncau 2019). and XLNet (Yang et al. 2019) have brought significant performance gains, but it can be challenging determine which aspects of the methods contribute the most. Training is computationally expensive, limiting the amount of tuning that can be done, and modeling advances are often conflated with changes in data size or composition.
We present replication study of BERT pretraining (Devlin et al.) 2019), which includes a careful evaluation of the effects of hyperparameter tuning and training set size. We find that BERT was sig- nificantly undertrained and propose an improved training recipe, which we call RoBERTa, that can match or exceed the performance of all of the -Ost-BERT methods. Our modifications are simple, they include: (1) training the model longer, with bigger batches, over more data; (2) removing the next sentence prediction objective; (3) training on longer sequences; and (4) dynamically changing the masking pattern applied to the training data. We also collect a large dataset (CC-NEwS) of comparable size to other privately used datasets, to better control for training set size effects.
When controlling for training data, improved training procedure improves upon the published BERT results on the GLUE (Wang et al., 2019b) and SQuAD (Rajpurkar et al. (016) benchmarks. When trained for longer over additional data, model achieves score of 88.5 the public GLUE leaderboard, matching the 88.4 reported by Yang et al. (2019). Our model establishes new state- of-the-art on 4/9 of the GLUE tasks, as well as RACE (Lai et 2017), SuperGLUE (Wang al. 2019a), and XNLI (Conneau et al.] 2018), and matches the state-of the-re on SQuAD. Overall, we re-establish that BERT's masked language model training objective competitive with recently proposed alternatives such perturbed autoregressive language modeling (Yang et al. 2
In summary, the contributions of this paper are: (1) We present a set of important BERT design choices and training strategies and introduce alternatives that lead to better downstream task perfor- mance: (2) We use a novel dataset, CC-NEWS, and confirm that using more data for pretraining fur- ther improves performance downstream tasks; (3) Our training improvements show that masked language model pretraining, under the right design choices, is competitive with all other recently published methods. We release model, pretraining and fine-tuning code.
'Ou models and code are available at: anonymous URI.
"hheee other methods could possibly improve with more tuning as well; leave this future work.

---

Under review as a conference paper at ICLR 2020
2 BACKGROUND
Setup: BERT (Devlin et al. 2019) takes as imput a concatenation of two segments (sequences of tokens), 21,..... NN and V1, ...... VM. Segments usually consist of more than one natural sentence. The two segments are presented as a single input sequence to BERT with special tokens delimiting them:
2N [SEP], y1... ... yMr [EOS]. M and N are constrained such that M M+N < T, where T' is parameter that controls the maximum sequence length during training.
Architecture: BERT uses the now ubiquitous transformer architecture (Vaswani et al. 2017) which we will review in detail. We use transformer architecture with L layers. Each block has A self-attention heads and hidden dimension H.
Training Objectives: BERT uses two pretraining objectives: masked language modeling and next sentence prediction. For the Masked Language Model (MLM) objective, BERT is trained via cross- entropy loss to predict 15% of the input tokens, selected at random. To prevent the model from cheating, 80% of these selected tokens are replaced by special [MASK] symbol in the input, 10% are replaced by random token from the vocabulary, and 10% are left unchanged.
Next Sentence Prediction (NSP) is binary classification loss for predicting whether two segments follow each other in the original text. Positive examples are created by taking consecutive sentences from the text corpus. Negative examples are created by pairing segments from different documents. Positive and negative examples are sampled with equal probability.
Optimization: BERT is optimized with AdamW (Kingma Ba 2015) using the following pa- rameters: 3 0.9, 32 0.999, lc-6 and decoupled weight decay of 0.01 (Loshchilov & Hutter 2019). The learning rate is warmed up over the first 10,000 steps to a peak value le-4, and then linearly decayed. BERT trains with dropout of 0.1 on all layers and attention weights, and GELU activation function (Hendrycks & Gimpel 2016). Models are pretrained for S = 1,000,000 updates, with mini-batches containing B 256 sequences of maximum length T 512 tokens.
Data: BERT is trained combination of BOOK ZZhu al. 2015) plus English WIKIPEDIA, which totals 16GB of uncompressed text
3
EXPERIMENTAL SETUP
3.1 IMPLEMENTATION
We reimplement BERT in FAIRSEQ OOt et al. 2019). We primarily follow the original BERT optimization hyperparameters, given in Section except for the peak learning rate and number of warmup steps, which are tuned separately for each setting. We found training to be very sensitive to the Adam epsilon term, and in after tuning it. We also set 32 = 0.98 to improve stability when training with large batch sizes.
cases obtained better performance improved stability
We pretrain with sequences of at most T 512 tokens. Unlike Devlin et al. (2019), we do not randomly inject short sequences, and we do not train with a reduced sequence length for the first 90% of updates. We train only with full-length sequences.
We train with mixed precision floating point arithmetic on DGX-1 machines, each with x 32GB Nvidia V100 GPUs interconnected by Infiniband (Micikevicius et al., 2018).
3.2 DATA
BERT-style pretraining crucially relies on large quantities of text. Baevski et al. (2019) demonstrate that increasing data size can result in improved end-task performance. Several efforts have trained on datasets larger and more diverse than the original BERT (Radford et al.) 2019 Yang al. 2019;
Zellers et al. 2019). Unfortunately, not all of the additional datasets can be publicly released. For
Yang al. 2019 use the same dataset but report having only 13GB of text after data cleaning. This is most likely due subtic differences in the underlying data collection preprocessing.
2

---

Under review as a conference paper at ICLR 2020
study, focus gathering much data possible for experimentation, allowing match the overall quality and quantity of data as appropriate for each comparison.
We consider five English-language corpora of varying sizes and domains, totaling 160GB of uncompressed text: (1&2) BOOKCORPUS (Zhu et al. 2013) plus English WIKIPEDIA, which is the original data used to train BERT (16GB); (3) NEWS, which collect from the English portion of the CommonCrawi News dataset (Nagel 2016), containing 63 million English news articles crawled between September 2016 and February 2019 (76GB after filtering) | (4) OPEN- WEBTEXT (Gokaslan & Cohen 2019), an open-source recreation of the WebText corpus described in Radford et al. (2019). containing web content extracted from URLs shared on Reddit with least three upvotes (38GB) (5) STORIES, a dataset introduced in Trinh & Le (2018) containing a subset of CommonCrawl data filtered to match the story-like style of Winograd schemas (31GB).
3.3 EVALUATION
Following previous work, we evaluate our pretrained models by finetuning on downstream tasks:
The General Language Understanding Evaluation (GLUE) benchmark (Wang et al. 2019b) is collection of 9 datasets for evaluating natural language understanding systems. Tasks are framed as either angle-sentence classification or sentence-pan -aann classification tasks. The GLUE organizers provide training and development data splits well as a submission server and leader- board that allows participants to evaluate and compare their systems on private held-out test data. The Stanford Question Answering Dataset (SQuAD) provides paragraph of context and a question. The task is to answer the question with span extracted from the context. We evaluate on SQuAD V1.1 and V2.0 (Rajpurkar et al. 2016 2018). In V1.1 the context always contains answer, while in V2.0 some questions
GLUE:
SQuAD:
answered in the provided context.
RACE: ReAding Comprehension from Examinations (RACE) (Lai al.. 2017) is large-scale reading comprehension dataset collected from English examinations in China. The task is to choose among four possible answers given question, using given passage of text as context. Additional Benchmarks: In the Appendix we present additional results for SuperGLUE (Wang
et al. 2019a) and XNI.I (Conneau et al. 2018).
4 TRAINING PROCEDURE ANALYSIS
This section explores and quantifies which choices are important for successfully pretraining BERT models. We keep the model architecture fixed Specifically, we begin by training BERT models with the same configuration as BERT BASD (I. T2, 768, A 12, 1 110M params).
4.1 STATIC VS. DYNAMIC MASKING
BERT relies predicting randomly masked tokens. The original BERT implementation performed masking once during data preprocessing, resulting in single static mask. To avoid repeating the same masks at every epoch, training data duplicated 10 times prior to preprocessing, so that each training sequence was seen with the same mask only four times over the course of 40 training epochs. We instead train with dynamic masking, where generate the masking pattern on-the-fly each time we input a sequence to the model. This becomes crucial when pretraining for more steps or with larger datasets, and additionally performs marginally better than static masking on some downstream tasks (see Appendix
As discussed in Section 2
4.2 MODEL INPUT FORMAT AND NEXT SENTENCE PREDICTION
In the original BERT pretraining procedure, the model observes two concatenated document seg- ments and is trained via an auxiliary Next Sentence Prediction (NSP) loss to predict whether these segments were sampled contiguously from the same document or from distinct documents.
"We use news-pl peess-plessee (Hamborg et al. 2017) to collect and extract CC-NEWS CC-NEWS is similar to the REALNEWS dataset described in Zellers al. (2019.
'The authors and their affiliated institutions are not affi liated with the creation of the OpenWebTere dataset. "Studying architectural changes, including larger architectures, is important for future work.
3

---

Under review as a conference paper at ICLR 2020
Model
SQuAD 1.1/2.0
MNLI-m
SST-2
RACE
Our reimplementation (with NSP loss):
SEGMENT-PAIR
90.4/78.7
84.0
92.9
64.2
SENTENCE-PAIR 88.7/76.2
82.9
92.1
63.0
Our reimplementation (without NSP loss):
FULL-SENTENCES
90.4/79.1
84.7
92.5
64.8
DOC-SENTENCES
90.6/79.7
84.7
92.7
65.6
BERT BASE
88.5/76.3
84.3
92.8
64.3
XLNetRASE (K=7)
-/81.3
85,8
92.7
66.1
XINNetRAE (K
-/81.0
85.6
93.4
66.7
Table Development set results for base models pretrained over BOOKCoRPUS and WIKIPEDIA. All models are trained for 1M steps with batch size of 256 sequences. We report FI for SQuAD and accuracy for MNLI-m, SST-2 and RACE, Reported results are medians over five random initial- izations (seeds). Results for BERT BASE and XLNelgAsE are from Yang et al. (2019)
The NSP objective was designed improve performance downstream tasks, such Natural Language Inference (Bowman et al. (2015), which require predicting relationships between pairs of sentences. Devlin al. (2019) observe that removing NSP hurts performance, with significant performance degradation on QNLI, MNLI, and SQuAD 1.1. However, recent work has questioned the necessity of the NSP loss (Lample Conneau 2019 Yang al.) 2019) Joshi et al.] 2019).
To better understand this discrepancy, we compare several alternative training formats:
This follows the original input format used in BERT (Devlin et al. 2019), with the NSP loss. Each input has a pair of segments, which can each contain multiple natural sentences, but the total combined length must be less than 512 tokens.
SEGMENT-PAIR+NSE
SENTENCE- PARRIE--P Each input contains a pair natural sentences, either sampled from a contiguous portion of one document from separate documents. Since these inputs are signif- icantly shorter than 512 tokens, we increase the batch size so that the total number of tokens remains similar SEGMENT-PAIR+NE
We retain the NSP loss.
FULL-SENTENCES: Each input is packed with full sentences sampled contiguously from one or more documents, such that the total length at most 512 tokens. Inputs may cross document boundaries. When we reach the end of one document, we begin sampling sentences from the next document and add an extra separator token between documents. We remove the NSP loss.
DOC-SENTENCES:
Inputs constructed similarly to FULL-SENTENCES, except that they may not cross document boundaries. Inputs sampled near the end document may be shorter than 512 tokens, we dynamically increase the batch size in these cases to achieve similar number of total tokens as FULL--EENTENCES We remove the NSP loss.
Results Table I shows results for the four different settings. We first compare the original SEGMENT-PAIR input format from Devlin et al. (2019) to the SENTENCE-PAIR format; both formats retain the NSP loss, but the latter uses single sentences. We find that using individual sentences hurts performance on downstream tasks, which we hypothesize is because the model is not able to learn long-range dependencies.
We next compare training without the NSP loss and training with blocks of text from a single docu- ment DOC-SSNNEECCES), We find that this setting outperforms the originally published BERT, baSE results and that removing the NSP loss matches mance, in contrast to Devlin et al. (2019). It is possible that the original BERT implementation may only have removed the loss term while still retaining the SEGMENT- PAIR input format.
slightly improves downstream task perfor-
Finally we find that restricting sequences to come from single document (DOC- (OCC-SENTNNCES per- forms slightly better than packing sequences from multiple documents (FULL- FFULL-SENTENCER How- ever, because the DOC-SENTENCER format results in variable batch sizes, we use FULL-SENTENCES in the remainder of our experiments for easier comparison with related work.

---

Under review as a conference paper at ICLR 2020
batch size
learning rate
epochs
steps
perplexity
MNLI-m
SST-2
256
le-4
32
1M
3.99
84.7
92.5
32
125K
3.68
85.2
93.1
2K
7c-4
64
250K
3.59
85.3
94.1
128
500K
3.51
85.4
93.5
32
31K
3.77
84.4
93,2
8K
le-3
64
63K
3.60
85.3
93.5
128
125K
3.50
85.8
94.1
Perplexity on held-out validation data and dev set accuracy on MNLI-m and SST-2 for various batch sizes (# sequences) as we vary the number of passes (epochs) through the BookS WIKI data. Reported results are medians over five random initializations (seeds). The learning rate tuned for each batch size. All results are for BERT with FULL-SENTENCE inputs.
Table 2:
4.3 TRAINING WITH LARGE BATCHES
Past work in neural machine translation has shown that training with large mini-batches improve optimization speed and end-task performance when the learning rate is tuned appropriately (Ot)
al. 2018 Large batches also easily parallelized via data parallel training
Table shows the masked LM perplexity and end-task accuracy for BERT as we increase the batch size, while tuning the learning rate. [Devlin al. (2019) originally trained BERT BASE for 1M steps with a batch size of 256 sequences; however a batch size of 2K sequences performs better, even controlling for the number of epochs, suggesting that the original BERT batch size was too small. We also observe that training with extremely large batches (8K) becomes more efficient as we train for more epochs In the remainder of our experiments we train with batches of 8K sequences.
4.4 TEXT ENCODING
Byte-Pair Encoding (BPE) (Sennrich al., 2016) is hybrid between character- and word-level modeling based on subwords units. BPE vocabulary sizes typically range from 10K-100K sub- word units; however, unicode characters can account for sizeable portion of this vocabulary when modeling large and diverse corpora, such as the ones considered in this work.
The original BERT implementation (Devlin al. 2019) used character-level BPE vocabulary of size 30K. We instead adopt the larger byte- BPE vocabulary of size 50K introduced in Radford al.) (2019), which uses bytes rather than unicode characters as the base subword units and can therefore encode any input text without introducing "unknown" tokens. This adds approximately 15M and 20M extra parameters for BERT rASE and BERTIARGES respectively.
Early experiments revealed only minor differences between these encodings, with the byte-level BPE achieving slightly worse end-task performance on some tasks. Nevertheless, we believe the advantages of universal encoding scheme outweighs the minor degredation in performance and use this encoding in the remainder of our experiments.
5 ROBERTA
In the previous section we propose modifications to the BERT pretraining procedure that improve end-task performance. We now aggregate these improvements and evaluate their combined im- pact. We call this configuration RoBERTa for Robustly optimized BERT approach. Specifically, RoBERTa is trained with dynamic masking (Section[4.1) FULL-SENTENCES without NSP loss (Sec- tion 4.2 large mini-batches (Section and a larger byte-level BPE (Section 4.4).
Even without large scale parallel hardware, large hatch training can improve training efficiency through gradient accumulation - i.c., accumulating gradients from multiple mini- mini-batches before each optimization step. 'You et al. (019) train BERT with even larger batch sizes, up 32K sequences. We leave further explo- ration of the limits of large hatch training future work.
5

---

Under review as a conference paper at ICLR 2020
Model
data
batch size
steps
SQuAD (v1.1/2.0)
MNLI-m
SST-2
RoBERTa
with BOOKS WIKI
16GB
8K
100K
93.6/87.3
89.0
95.3
+ additional data (3.22
160GB
8K
100K
94.0/87.7
89.3
95.6
+ pretrain longer
160GB
8K
300K
94,4/88.7
90.0
96.1
+ pretrain even longer
160GB
8K
500K
94.6/89.4
90.2
96.4
BERT
with BOOKS WIKI
13GB
256
IM
90.9/81.8
86.6
93.7
XIAGEtLARGE
BOOKS WIKI
13GB
94.0/87.8
88.4
94.4
+ additional data
126GB
2K
500K
94.5/88.8
89.8
95.6
Table 3: Development set results for RoBERTa as we pretrain over more data (16GB + 160GB of text) and pretrain for longer (100K 300K + 500K steps). Each row accumulates improvements from the rows above. RoBERTa matches the architecture and training objective of BERT Results for BERT and XLNet, are from Devlin et al. (2019) and Yang et al. (2019), re- spectively. Complete results on all GLUE tasks can be found in Appendix
Additionally, we investigate two other important factors that have been under-emphasized previ- work: (1) the data used for pretraining, and (2) the number of training passes through the data. For example, XLNet (Yang et al., 2019) was pretrained using 10 times more data than BERT, with batch size eight times larger for half many optimization steps, thus seeing four times as many sequences in pretraining compared to Devlin et al. (201)).
To help disentangle the importance of these factors from other modeling choices (e.g., g., the pretraining objective), we begin by training RoBERTa following the BERT architecture (L = 24, H 1024, A 16, 355M parameters). We pretrain for 100K steps over a comparable BOOKCORPUS plus WIKIPEDIA dataset as was used in Devlin et al. (201)). We pretrain our model using 1024 V100 GPUs, which takes approximately one day per 100K steps.
When controlling for training data, we observe that RoBERTa provides large improvement over the originally reported BERT results, reaffirming the importance of the design choices we explored in Section
Results We present our results Table 3
Next, we combine this data with the three additional datasets described in Section 3.2 We train RoBERTa over the combined data with the same number of training steps as before (100K). In total, we pretrain over 160GB of text. We observe further improvements in performance across all downstream tasks, validating the importance of data size and diversity pretraining
Finally, we pretrain RoBERTa for significantly longer, increasing the number of pretraining steps from 100K to 300K, and then further to 500K. We again observe significant gains downstream task performance, and the 300K and 500K step models outperform XLNetLARGE across most tasks. We note that even our longest-trained model does not appear to overfit our data and would likely benefit from additional training.
5.1 GLUE RESULTS
For GLUE, consider finetuning settings. In the first setting (single-task, dev), finetune RoBERTa separately for each of the GLUE tasks, using only the training data for the corresponding task. We consider limited hyperparameter sweep with batch sizes {16, 32) and learning rates (1e-5, 2e-5, 3e-5). with linear warmup for the first 6% of steps followed by linear decay O. We finetune for 10 epochs with early stopping based on each task's dev set. The rest of the hyperparameters remain the same as during pretraining. In this setting, we report the median development set results for each task over five random initializations, without model ensembling.
In the second setting (ensembles, lest), we compare RoBERTa to other approaches on the test set via the GLUE leaderboard. While many submissions to the GLUE leaderboard depend on multi-task
A careful analysis disentangling data size and diversity is needed, but left future work.

---

Under review as a conference paper at ICLR 2020
MNLI
QNLI
QQP
RTE
SST
MRPC
CoLA
STS
WNLI
Avg
Single-task single models dev
BERT LARGE 86,6/-
92.3
91.3
70.4
93.2
88.0
60.6
90.0
XLNE 89.8/-
93.9
91.8
83.8
95.6
89.2
63.6
91.8
RoBERTa 90.2/90.2
94.7
92.2
86.6
96.4
90.9
68.0
92.4
91.3
Ensembles on test (from leaderboard as of July 25, 2019)
ALICE
88,2/87.9
95.7
90.7
83.5
95.2
92.6
68.6
91.1
80.8
86.3
MT-DNN
87.9/87.4 4
96.0
89.9
86.3
96.5
92.7
68.4
91.1
89.0
87.6
XL.Net
90.2/89.8
98.6
90.3
86.3
96.8
93.0
67.8
91.6
90.4
88.4
RoBERTa
90.8/90.2
90.2
88.2
96.7
92.3
67.8
92.2
89.0
88.5
Results on GLUE. All results are based on a layl architecture. BERT and XLNettARGE results from Devlin et al. (2019) and Yang et al.) (2019). respectively. RoBERTa results on the dev set are a median over five runs. RoBERTa results on the test set are ensembles of single-task models. For RTE, STS and MRPC finetune starting from the MNLI model.
Table 4:
SQuAD 1.1
SQuAD 2.0
SQuAD 2.0
Model
Model
EM
F1
EM
F1
EM
F1
Single models on dev, w/o data augmentation
Single models on test (as of July 25,
2019)
BERT laRGe
84.1
90.9
79.0
81.8
XLNet,
86.31
89.1
XLNetLARGE
89.0
94.5
86.1
88.8
RoBERTa
86.8
89.8
RoBERTa 88.9
94.6
86.5
89.4
XL.Net SG-Net Verifier
87.01
89.91
Table 5: Results on SQuAD. f indicates results that depend on additional external training data. RoBERTa uses only the provided SQuAD data both dev and test settings. BERTLARGS and XLNetLARGE results are from Devlin al. (2019) and Yang et al. (2019), respectively.
finetuning, our submission depends only on single-task finetuning. For RTE, STS and MRPC we finetune starting from the MNLI single-task model, following Phang al. (2018). We explore slightly wider hyperparameter space, described in Appendix C and ensemble between 5 and models per task. Two the GLUE tasks require task-specific finetuning approaches to achieve competitive leaderboard results; these approaches are described in Appendix
Results We present our results in Table In the first setting (single-task, dev), RoBERTa achieves state-of-the-art results on all 9 of the GLUE task development sets. Crucially, RoBERTa uses the same masked language modeling pretraining objective and architecture LARGE yet consis- tently outperforms both BERT and XLNettARGE. This raises questions about the relative im- portance of model architecture and pretraining objective, compared more mundane details like dataset size and training time that we explore in this work. A more comprehensive comparison of the BERT and XLNet pretraining objectives is needed, but left future work.
In the second setting (ensembles, test), we submit RoBERTa to the GLUE leaderboard and achieve state-of-the-ate results on out tasks and the highest average score to date. Notably, RoBERTa does not depend on multi-task finetuning, and expect future work may further improve these results by incorporating more sophisticated multi-task finetuning procedures.
5.2 SQUAD RESULTS
We adopt much simpler approach for SQuAD compared to past work. While BERT (Devlin al. 2019) and XLNet (Yang et al. 2019) augment their training data with additional QA datasets, we only finetune RoBERTa using the provided SQuAD training data. We also for all layers, in contrast to the custom layer- wise learning rate scheduled used by Yang et al. 2019
single learning rate
For SQuAD v1.1 follow the same finetuning procedure as Devlin et al. (2019). For SQuAD v2.0, we additionally classify whether a given question is answerable; we train this classifier jointly with the span predictor by summing the classification and span loss terms.

---

Under review as a conference paper at ICLR 2020
Model
Accuracy
Middle
High
Single models on test (as of July 25,
2019)
BERT LARGE
72.0
76.6
70.1
XLNet 81.7
85.4
80.2
RoBERTa
83.2
86.5
81.3
Table 6: Results the RACE test set. BERT LARGE and XLNettARGE results from Yang et al. (0119)
Results We present results in Table 5 On the SQuAD v1.1 development set, RoBERTa matches the state-of-the-art set by XLNet. On the SQuAD v2.0 development set, RoBERTa sets new state- improving over XLNet by 0.4 points (EM) and 0.6 points (F1).
We also submit RoBERTa to the public SQuAD 2.0 leaderboard. Most of the top systems build upon either BERT (Devlin et al. (211)) or XLNet (Yang al. (219) and therefore rely on additional external training data. Our single RoBERTa model outperforms all but one of the single model submissions, and is the top scoring system among those that do not rely on additional external data.
5.3 RACE RESULTS
In RACE, systems are provided with a passage of text, associated question, and must classify which of four candidate answers is correct. We modify RoBERTa for this task by concatenating each candidate answer with the corresponding question and passage. We encode each of these four sequences and pass the resulting (CLS) representations through a fully-comnected layer, which is used to predict the correct answer. We truncate question-answer pairs that are longer than 128 tokens and, if needed, the passage so that the total length is at most 512 tokens.
Results are presented in T RoBERTa achieves state-of of- oef-the-art accuracy across all settings.
RELATED WORK
Pretraining methods have been designed with different training objectives, including language mod- eling (Dai & Le 2015 Peters al. 2018 Howard & Ruder 2018), machine translation (McCann 2017). and masked language modeling (Devlin et al. [219 Lample & Conneau (019). Many recent papers have used basic recipe of finetuning models for each end task (Howard Ruder
et al.
2018
Radford et al., 2018), and pretraining with some variant of masked language model objec-
tive.
However, newer methods have improved performance by multi-task fine tuning (Dong et al,,
2019 incorporating entity embeddings (Sun et al., 2019). span prediction (Joshi et al. 2019)) and
multiple variants of autoregressive pretraining (Song et al. 2019 Chan et al.) 2019 Yang et al.) 2019). Performance is also typically improved by training bigger models more data (Devlin et al. 2019 Baevski et al., 2019 Yang et al.) 2019; Radford et al. 2019). Our goal was to repli- cate, simplify, and better tune the training of BERT, reference point for better understanding the relative performance of all of these methods.
CONCLUSION
We evaluate number of design decisions when pretraining BERT models, demonstrating that per- formance can be substantially improved by training the model longer, with bigger batches over more data; removing the next sentence prediction objective; training on longer sequences; and dynami- cally changing the masking pattern applied to the training data. We additionally use a novel dataset, NE-NEWs, and release models and code for pretraining and finetuning anonymous URL.
Our improved pretraining procedure, which we call RoBERTa, achieves state-of-the-art results on GLUE, RACE, SQuAD, SuperGLUE and XNLI. These results illustrate the importance of these previously overlooked design decisions and suggest that BERT's pretraining objective remains com- petitive with recently proposed alternatives.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Eneko Agirre, Lluís Mårquez, and Richard Wicentowski (eds.). Proceedings of the Fourth Interna- tional Workshop on Semantic Evaluations (SemEval-2007), 2007.
Alexei Baevski, Sergey Edunov, Yinhan Liu, Luke Zettlemoyer, and Michael Auli. Cloze-driven pretraining of self-attention networks. arXiv preprini arXiv: 2019.
Roy Bar Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. The second PASCAL recognising textual entailment challenge. 2006.
Luisa Bentivogli, Ido Dagan, Hoa Trang Dang, Danilo Giampiccolo, and Bernardo Magnini. The fifth PASCAL recognizing textual entailment challenge. 2009.
Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. A large anno- tated corpus for learning natural language inference. In Empirical Methods in Natural Language Processing (EMNLP), 2015.
William Chan, Nikita Kitaev, Kelvin Guu, Mitchell Stern, and Jakob Uszkoreit. KERMIT: Genera- tive insertion-based modeling for sequences. arXiv preprint 1906.0900.01604 2019.
Christopher Clark, Kenton Lee, Ming-Wer Meng-Wei Chang, Tom Kwiatkowski. Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of NAACL-HLT 2019, 2019.
Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel R. Bowman, Holger Schwenk, and Veselin Stoyanov. Xnli: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods Natural Language Processing. Association for Computational Linguistics, 2018.
Ido Dagan, Oren Glickman, and Bernardo Magnini. The PASCAL recognising textual entailment challenge. Machine learning challenges. evaluating predictive uncertainty, visual object clas- sification, and recognising tectual entailment, pp. 177-190. Springer, 2006.
Andrew M Dai and Quoc V Le. Semi-supervised sequence learning. In Advances in Neural Infor- mation Processing Systems (NIPS), 2015.
De Marneffe, Mandy Simons, and Judith Tonhauser. The CommitmentBank: In- vestigating projection in naturally occurring discourse. 2019. To appear in proceedings of Sinn und Bedeutung 23. Data can found at
Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. BERT; Pre-training of deep bidirectional transformers for language understanding. In North American Association for Com- putational Linguistics (NAACL), 2019.
William Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the International Workshop on Paraphrasing, 2005.
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao- Wuen Hon. Unified language model pre-training for natural language understanding and generation. arXiv preprint arXiv:1905.03190 2019.
Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. The third PASCAL recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL workshop on textual entailment and paraphrasing, pp. 1-9. Association for Computational Linguistics, 2007.
Aaron Gokaslan and Vanya Cohen. Openwebtext corpus. http::/wwet http://wwb....iiie.e....o... owg.ssiv...////veee.... http::/skylionot .i git.huitthggg. i /ttpe//eebeeetttorpps..pppp http://o......g.aooooooo///eennr 2019
Felix Hamborg. Norman Meuschke, Corinna Breitinger, and Bela Gipp. news-please: A generic news crawler and extractor. In Proceedings of the 15th International Symposium of Information Science, 2017.
Dan Hendrycks and Kevin Gimpel. 1oXiv:1006.086 2016.
Gaussian error linear units (gelus). arXiv preprint

---

Under review as a conference paper at ICLR 2020
Matthew Honnibal and Ines Montani. spaCy 2: Natural language understanding with Bloom em- beddings, convolutional neural networks and incremental parsing. To appear, 2017.
Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. 2018.
arXiv preprint
Shankar Iyer, Nikhil Dandekar, and Kornl Csernai. First quora dataset release: Question aetaaa..uura..... www.gu.w./o.aa.... wtt./irsttt///eeeeeeeeeeee Sarrs-t-uor---aaassttt Relsseee-eeeeeeeeeee Questiin
pairs. Pairs, 2016.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. SpanBERT: Improving pre-training by representing and predicting spans. arXiv preprint 1907. 2019.
Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. Look- ing beyond the surface: A challenge set for reading comprehension over multiple sentences. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 252-262, 2018.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
Vid Kocijan, Ana-Maria Cretu, Oana-Maria Camburu, Yordan Yordanov, and Thomas Lukasiewiez. A surprisingly robust trick for winograd schema challenge. arXiv preprint arXiv:1905.06290 2019.
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. Race: Large-scale reading comprehension dataset from examinations. arXiv preprint 17X.v:/704.4000 2017.
Guillaume Lample and Alexis Conneau. Cross-lingual language model pretraining. arXiv preprint arXiv:1901.07291, 2019.
Hector J Levesque, Ernest Davis, and Leora Morgenstern The Winograd schema challenge. In AAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning, volume 46, pp- 47, 2011.
Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. Improving multi-task deep neu- ral networks via knowledge distillation for natural language understanding. arXiv preprint arXiv:1904.09482, 2019a.
Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. Multi-task deep neural networks for natural language understanding. arXiv preprint arXiv: 1911111190 2019b.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. International Confer- ence on Learning Representations, 2019. URL httptht:://ppereeeeeeee wet.iiewwweeeeeeeeeeeeeee.eeeeee fort//foum?? id=
Bkg6RiCqY7
Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. Learned in translation:
Contextualized word vectors. In Advances in Neural Information Processing Systems (NIPS), pp- 6297-6308, 2017.
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed precision training. International Conference on Learning Representations, 2018.
Sebastian Nagel. Co-news. http::///eeb..aarrrreeooor///rrr httpttppppp//ommmmcraca.l. available,
org/2016 1//20016111000 0a11///11////eeee aaaataset-aaillallee 2016.
Myle Out, Sergey Edunov, David Grangier, and Michael Auli. Scaling neural machine translation.
In Proceedings of the Third Conference on Machine Translation (WMT), 2018.
Myle Ott. Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng. David Grangier, and Michael Auli. FAIRSEQ: fast, extensible toolkit for sequence modeling. In North American Association for Computational Linguistics (NAACL)): System Demonstrations, 2019.
10

---

Under review as a conference paper at ICLR 2020
Matthew Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In North American Association for Computational Linguistics (NAACL). 2018.
Jason Phang, Thibault Fvry, and Samuel R. Bowman. Sentence encoders stilts: Supplementary training on intermediate labeled-data tasks. arXiv preprint 1rXiv:18/1.01088. 2018.
Mohammad Taher Pilehvar and Jose Camacho-Colladde WiC: The word-in-context dataset for
evaluating context-sensitive meaning representations. In Proceedings of NAACL-HLI, 2019.
Adam Poliak, Aparajita Haldar, Rachel Rudinger, J. Edward Hu, Ellie Pavlick, Aaron Steven White, and Benjamin Van Durme. Collecting diverse natural language inference problems for sentence representation evaluation. In Proceedings of EMNLP, 2018.
Alec Radford, Karthik Narasimhan, Time Salimans, and Ilya Sutskever. Improving language under- standing with unsupervised learning. Technical report, OpenAI, 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models unsupervised multitask learners. Technical report, OpenAI, 2019.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ ques- tions for machine comprehension of text. In Empirical Methods in Natural Language Processing (EMNLP), 2016.
Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't 't know: Unanswerable questions for squad. In Association for Computational Linguistics (ACL), 2018.
Melissa Roemmele, Cosmin Adrian Bejan, and Andrew Gordon. Choice plausible alternatives: An evaluation of commonsense causal reasoning. In 2011 AAAI Spring Symposium Series, 2011 1.
Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. Gender bias in coreference resolution. In Proceedings of NAACL-HLT, 2018
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Association for Computational Linguistics (ACL), pp. 1715-1725, 2016.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Empirical Methods in Natural Language Processing (EMNLP), 2013.
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie- Yan Liu. MASS: Masked sequence to sequence pre-training for language generation. International Conference on Machine Learning (ICML), 2019.
Yu Stephanic Sun. Shuohuan Wang. Yukun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xinlun Tian, Danxiang Zhu, Hao Tian, and Hua Wu. ERNIE: Enhanced representation through knowledge integration. arXiv preprint arXiv: 999.000222..... 2019.
Trieu H Trinh and Quoc V Le. A simple method for commonsense reasoning. arXiv preprint 18X:::1006.0287, 2018
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, 2017.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. SuperGLUE: A stickier benchmark for general pprppse language understanding systems. arXiv preprint 1905.00537. 2019a.
Alex Wang. Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations (ICLR), 2019b.
Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman, Neural network acceptability judgments. arXiv preprint 1805. /2471, 2018. 2018.
11

---

Under review as a conference paper at ICLR 2020
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for tence understanding through inference. North American Association for Computational Lin- guistics (NAACL), 2018
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint wrXiv:1906.082577 2019.
Yang You, Jing Li, Jonathan Hseu, Xiaodan Song, James Demmel, and Cho-Jui Hsieh. Reducing bert pre-training time from 3 days to 76 minutes. arXiv preprint NXXVv:1904000962 2019.
Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. Defending against neural fake news. arXiv preprint arXiv: 19X. 19999005..666
2019.
Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. ReCoRD: Bridging the gap between human and machine commonsense reading comprehension. arXiv preprint 1810.12885, 2018.
Yukun Zhu, Ryan Kiros, Richard Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In arXiv preprint arXiv:1506.0672: 2015.
12

---

Under review as a conference paper at ICLR 2020
A
STATIC vs. DYNAMIC MASKING
Masking
SQuAD 2.0
MNLI-m
SST-2
reference 76.3
84.3
92.8
Our reimplementation:
static
78.3
84.3
92.5
dynamic 78.7
84.0
92.9
Table 7: Comparison between the published BERT BASK results from Devlin et al. (019 our reimplementation with either static dynamic masking. We report F1 for SQuAD and accuracy for MNLI-m and SST-2. Reported results are medians over random initializations (seeds). Reference results are from Yang et al. (2019). We find that reimplementation with static masking performs similar to the original BERT model, and dynamic masking is comparable or slightly better than static masking.
B TASK-SPECIFIC MODIFICATIONS FOR GLUE
Two the GLUE tasks require task-specific finetuning approaches to achieve competitive leader- board results:
Recent submissions on the GLUE leaderboard adopt pairwise ranking formulation for the QNLI task, in which candidate answers are mined from the training set and compared to one another, and single (question, candidate) pair classified as positive (Liu al., 2019b a Yang et al., 2019). This formulation significantly simplifies the task. but is not directly comparable to BERT (Devlin cet al. 2019). Following recent work, we adopt the ranking approach for our test submission, but for direct comparison with BERT all reported development set results are based pure classification approach.
QNLI
WNLI We found the provided NLI-format data to be challenging to work with. Instead we use the reformatted WNLI data from SuperGLUE (Wang et al. 2019a), which indicates the span of the query pronoun and referent. We then finetune RoBERTa using variation of the approach from Kocijan et al. (201)). In particular, for a given input sentence, we first use spaCy (Honnibal & Montani 2017 to extract additional candidate noun phrases from the sentence, and then finetune our model so that it assigns higher scores to positive referent phrases than for any of the generated negative candidate phrases.
In contrast to Kocijan et al. 2019), who finetune BERT using margin ranking loss between (query. candidate) pairs, we instead query and all mined candidates. This reduces the number of hyperparameters that need to be tuned and in practice produces more stable results on the development Our best model achieved 92.3% development set accuracy, compared to 90.2% accuracy for the margin loss approach.
single
entropy loss
the log-probabilities for the
One unfortunate consequence of our overall approach is that we can only make use of the positive training examples, which excludes over half of the provided training data 100
C FULL RESULTS ON GLUE
In Table we present the full of development results for RoBERTa all 9 GLUE datasets We present results for a LARGE configuration with 355M parameters that follows BERT BERTLARGE. LARGE. as well as a BASE configuration with 125M parameters that follows BERT
While we only use the provided WNLI training data, our results could potentially be improved by aug- menting this with additional pronoun disambiguation datasets.
"The GLUE datasets are: CoLA (Warstadt et al. 2018). Stanford Sentiment Treebank (SST) (oocher et al.
2013). Microsoft Research Paragraph Corpus (MRPC) (Dolan & Brocketi 2005). Semantic Textual Similar-
ity Benchmark (STS) (Agirre et al. (2007). Quora Question Pairs (QQP) [yyrr al. 2016). Multi-Genre NLI
(MNLJ) (Willims 2018), Question NI.I (QNLI) (Rajpurkar al. 2016), Recognizing Textual Entail-
13

---

Under review as a conference paper at ICLR 2020
MNLI
QNLI
QQP
RTE
SST
MRPC
CoLA
STS
RoBERTàaAse all data + 500k steps
87.6
92.8
91.9
78.7
94.8
90.2
63.6
91.2
ROBERTaLARGE
with BOOKS WIKI
89.0
93.9
91.9
84.5
95.3
90.2
66.3
91.6
additional data (.2.2
89.3
94.0
92.0
82.7
95.6
91.4
66.1
92.2
pretrain longer 300K
90.0
94.5
92.2
83.3
96.1
91.1
67.4
92.3
pretrain longer 500k
90.2
94.7
92.2
86.6
96.4
90.9
68.0
92.4
Table 8:
Development set results on GLUE tasks for various configurations of RoBERTa. All results are a median over five runs.
D PRETRAINING HYPERPARAMETERS
Hyperparam
RoBERTaLARGE
RoBERTagas TRASE
Number of Layers
24
12
Hidden size
1024
768
FFN inner hidden size
4096
3072
Attention heads
12
Attention head size
64
64
Dropout
0.1
0.1
Attention Dropout
0.1
0.1
Warmup Steps
24k
24k
Peak Learning Rate
4e-4
6e-4
Batch Size
8k
8k
Weight Decay
0.01
0.01
Max Steps
500k
500k
Learning Rate Decay
Linear
Linear
Adam
le-6
le-6
Adam 81
0.9
0.9
Adam 82
0.98
0.98
Gradient Clipping
0.0
0.0
Table 9: Hyperparameters for pretraining RoBERTa and SoBERTaRASE:
E FINETUNING HYPERPARAMETERS
Hyperparam
RACE
SQUAD
GLUE
SuperGLUE
Learning Rate
le-5
1.5e-5
(1e-5, 2e-5, 3e-5)
(1e-5, 2e-5, 3e-5)
Batch Size
48
{16, 32)
Weight Decay
0.1
0.01
0.1
0.1
Max Epochs
2
10
{10, 50) Linear
Learning Rate Decay
Linear 0.06
Linear
Linear 0.06
Warmup ratio
0.06
0.10
Table 10: Hyperparameters for finetuning RoBERTaLARCE on RACE, SQuAD and GLUE. We select the best hyperparameter values based on the median of random seeds for each task.
ment (RTF) (Dagan al. 2006 Bar Haim al. 2006 Giampiccolo et al. 2007 Bentivogli al. 2009 and
Winograd NI. (WNLI) Levesque al. 2011).
14

---

Under review as a conference paper at ICLR 2020
F RESULTS ON SUPERGLUE
BoolQ
CB
COPA
MultiRC
ReCoRD
RTE
WiC
WSC
Avg
Single-task single models on dev
+ERT++ 80.1
96.4/95.0
78.0
70.7/24.7
70.6/69.8 89.5/89.0
82.3
74.9
68.3
74.6
RoBERTa 86.9
98.2/-
94.0
85.7/-
86.6
75.6
Ensembles on test (from leaderboard as of August
12, 2019)
BERT
77.4
75.7/83.6 6
70.6
70,0/24.1
72.0/71.3
71.7
69.6
64.4
69.0
BERT++ 79.0
84.8/90.4
73.8
70.0/24.1
72.0/71.3
79.0
69.6
64.4
71.5
Outside Best
80.4
84.4
70.4/24.5
74.8/73.0
82.7
RoBERTa
87.1
90.5/95.2
90.6
84.4/52.5
90.6/90.0
88.2
69.9
89.0
84.6
Human (est.)
89.0
95.8/98.9
100.0
81,8/5199
91.7/91.3
80.0
100.0
89.8
Table 11: Results on SuperGI.UE. All results are based on 24-layer architecture, RoBERTa results on the development set are median over five runs. RoBERTa results on the test set are ensembles of single-task models. Averages are obtained from the SuperGLUE leaderboard.
We also evaluate RoBERTa on the SuperGLUE benchmark (Wang et al., 2019a). which consists of 8 natural language understanding tasks 12] We largely follow the same setup for SuperGLUUE did for GLUE, with several task-specific modifications:
BoolQ and MultiRC: we follow the same input format as the Wang et al. (2019a) baseline.
CB: we finetune starting from the MNLI model, following Phang et al. (2018).
COРA: we concatenate the premise and each alternative with because and so markers for cause and effect questions, respectively. This input format more closely matches the pretraining data format and provides better results in practice.
ReCoRD: during training adopt pairwise ranking formulation with negative and positive entity for each (passage, query) pair. At evaluation time, we pick the entity with the highest score for each question.
WiC: we input the pair sentences as normal. We then feed the concatenation of the representa- tions of the two marked words and the /CLS) token to the classification layer.
RTE and WSC: we reused our submission to the GLUE leaderboard.
In Table II we present RoBERTa results on the 8 SuperGLUE datasets. RoBERTa achieves state- of-the-art results on the development and test sets for BoolQ, CB, COPA, MultiRC and ReCoRD and the highest average score to date on the SuperGLUE leaderboard.
G RESULTS ON XNLI
Machine transiation baselines
TTRANSILATE-TESS
TEST))
(MLM+TLM)
79.0
70.8
XLM-en
88.8
81.4
82,3
80.1
80.3
80.9
76.2
76.0
75.4
72.0
71.9
75.6
70.0
65.8
65.8
76.2
RoBERTa
82.9
78.3
77.5
66.8
Table 12: Results on XNLI (Conneau et al. 2018) for RoBERTa in the TRANSLATE-TEST"
setting. We report macro-averaged accuracy (A) using the provided English translations of the XNLI test sets. RoBERTa achieves state of the art results on all 15 languages.
1he SuperGLUE datasets are: BoolQ Clark al. 2011)). CommitmentBank (CB) DD Marneffe et al. 201T). Multi-Sentence Reading Com- Commonsense Reasoning (ReCoRD) (Zhang et al. (2018), Recognizing Textual Entailment (RTE) (Dagan et al. 2006 Bar Haim et al.
2019 Choice of Plausible Altemntives (COPA) (Roemmele
prehension (MultiRC) (Khashabi et al.] 2018). Reading Comprehension
2006 Giampiccolo et al. 2007 Bentivogli et al. 2009). Words in Context (WiC) Pilehvarr & Camacho
Collados (209)) the Winograd Schema Challenge (WSC) [Levesque et al. 2011), and Winogender Schema
Diagnostics (Poliak et al.
2018 Rudinger al. 2018
15


Reviews:
[{"review_id": "r1gzpOvAFH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a detailed replication study of the BERT pre-training model considering alternative design choices such as dynamic masking, removal of next sentence prediction loss, longer training time, larger batch sizes, additional training data, training on longer single document or cross-document sequences etc. to demonstrate their efficacy on several benchmark datasets and tasks by achieving the new state-of-the-art results. Overall, the paper is very well-written and the experimental setups are reasonable and thoroughly presented that would benefit the community for future research and exploration. However, I am not sure if the paper presents a case of adequate novelty in terms of ideas as many of them are rather obvious and the current state-of-the-art models could also improve considerably using similar experimental setups, which authors also acknowledged in footnote 2.\n\nOther comments:\n\n- Section 3.1: please clarify how exactly setting Beta2=0.98 would improve stability when training with large batch sizes.\n\n- It's not clear what exactly was the motivation for proposing full-sentences and doc-sentences input formats. Please explain.\n\n- Although the presented metrics show that the removal of NSP loss helps, however, no explanation was provided based on qualitative evaluation as to why this is the case. Some task-specific examples would have been nice to discuss the effects of NSP loss.\n\n- Section 5: Please provide details on why you think the longest-trained model does not appear to overfit. ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "Byx4-_ZZcS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a replication study of BERT pretraining and carefully measures the impact of many key hyperparameters and training data size. It shows that BERT was significantly undertrained and propose an improved training recipe called RoBERTa. The key ideas are: (i) training longer with bigger batches over more data, (ii) removing NSP, (iii) training over long sequences, and (iv) dynamically changing the masking pattern. The proposed RoBERTa achieves/matches state-of-the-art performance on many standard NLU downstream tasks. \n\nThe in-depth experimental analysis of the BERT pretraining process in this paper answers many open questions (e.g., the usefulness of NSP objective) and also provide some guidance in how to effectively tweak the performance of pretrained model (e.g., large batch size). It also further demonstrates that the BERT model, once fully tuned, could achieve SOTA/competitive performance compared to the recent new models (e.g., XLNet). The main weakness of the paper is that it is mainly based on further tuning the existing BERT model and lacks novel contribution in model architecture. However, the BERT analysis results provided in this paper should also be valuable to the community. \n\nQuestions & Comments:\n•\tIt is stated that the performance is sensitive to epsilon in AdamW. This reminds us of the sensitivity of BERT pretraining to the optimizers. Since one of the main contributions of this paper is the analysis of the BERT pretraining process, more experimental analysis on the optimizer should also be included.\n•\tIt is stated that (page 7) the submission to GLUE leaderboard uses only single-task finetuning. Is there any special reason for restraining it to single-task finetuning if earlier results demonstrates multi-task finetuning is better? Of course, it is valuable to see the great performance achieved by single-task finetuning for RoBERTa. But there should be no reason that it is restricted to be so. An additional experimental results with multi-task finetuning should also be added.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "ByeB-Xqatr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper is a replication study of BERT for training large language models. Its main modification is simple: training longer with more data. Significantly improvements have been reported, and the work achieves on-par or higher accuracy over a large set of downstream tasks compared to XLNet, which is a state-of-the-art autoregressive language model. \n\nPros:\n+ The paper incorporates robust optimization into BERT training with more data, and shows that together it significantly improves BERT's performance on downstream tasks.\n+ The experimental results show that RoBERTa can significantly advance the baseline BERT model and achieve on-par or new state-of-the-art accuracy on a large range of downstream tasks.\n\nCons:\n- While the replication study is well appreciated, the novelty contribution of the paper is marginally incremental as the model structure is largely unchanged from BERT. The other techniques applied also are somewhat trivial.\n- Very little can be deduced from the experiments, as performance is often improved by training over more data. \n\nOverall, I believe this paper comes at the right time and is addressing an interesting problem. The paper is well-organized and well-written. The contribution of the paper comes mostly from carefully taking into account several additional design choices and show that they could help train BERT with more data and can achieve SoTA performance on downstream tasks. Those modifications can be summarized as: (1) large-batch training with batch size 8k; (2) no mixed sequence length training only used 512 for the entire run; (3) no next sentence prediction; (4) dynamic masking instead of static; (5) larger byte-level BPE vocab (which increases BERT-large size by 20M parameters). Although they are interesting, a major concern is that it is difficult to find one thing that would have catapulted it over others to ensure publication.\n\nQuestion:\nDo you plan to release the datasets used for training in this work?", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper conducts an extensive study of training BERT and shows that its performance can be improved significantly by choosing a better training setup (e.g., hyperparameters, objective functions). I think this paper clearly offers a better understanding of the importance of tuning a language model to get the best performance on downstream tasks. However, most of the findings are obvious (careful tuning helps, more data helps). I think the novelty and technical contributions are rather limited for a conference such as ICLR. These concerns are also shared by all the reviewers. The review scores are borderline, so I recommend to reject the paper.
Decision: Reject