The MultiBERTs: BERT Reproductions for Robustness Analysis

Experiments with pre-trained models such as BERT are often based on a single checkpoint . While the conclusions drawn apply to the artifact tested in the experiment ( i.e. , the particular instance of the model ) , it is not always clear whether they hold for the more general procedure which includes the architecture , training data , initialization scheme , and loss function . Recent work has shown that repeating the pre-training process can lead to substantially different performance , suggesting that an alternative strategy is needed to make principled statements about procedures . To enable researchers to draw more robust conclusions , we introduce MultiBERTs , a set of 25 BERT-Base checkpoints , trained with similar hyper-parameters as the original BERT model but differing in random weight initialization and shuffling of training data . We also define the Multi-Bootstrap , a non-parametric bootstrap method for statistical inference designed for settings where there are multiple pre-trained models and limited test data . To illustrate our approach , we present a case study of gender bias in coreference resolution , in which the Multi-Bootstrap lets us measure effects that may not be detected with a single checkpoint . The models and statistical library are available online,1 along with an additional set of 140 intermediate checkpoints captured during pretraining to facilitate research on learning dynamics . 1 INTRODUCTION . Contemporary natural language processing ( NLP ) relies heavily on pretrained language models , which are trained using large-scale unlabeled data ( Bommasani et al. , 2021 ) . BERT ( Devlin et al. , 2019 ) is a particularly popular choice : it has been widely adopted in academia and industry , and aspects of its performance have been reported on in thousands of research papers ( see , e.g. , Rogers et al. , 2020 , for an overview ) . Because pre-training large language models is computationally expensive ( Strubell et al. , 2019 ) , researchers often rely on the release of model checkpoints through libraries such as HuggingFace Transformers ( Wolf et al. , 2020 ) , which enable them to use large-scale language models without repeating the pre-training work . Consequently , most published results are based on a small number of publicly released model checkpoints . While this reuse of model checkpoints has lowered the cost of research and facilitated head-to-head comparisons , it limits our ability to draw general scientific conclusions about the performance of a particular class of models ( Dror et al. , 2019 ; D ’ Amour et al. , 2020 ; Zhong et al. , 2021 ) . The key issue is that reusing model checkpoints makes it hard to generalize observations about the behavior of a single model artifact to statements about the underlying pre-training procedure which created it . Pre-training such models is an inherently stochastic process which depends on the initialization of the model ’ s parameters and the ordering of training examples ; for example , D ’ Amour et al . ( 2020 ) report substantial quantitative differences across multiple checkpoints of the same model architecture on several “ stress tests ” ( Naik et al. , 2018 ; McCoy et al. , 2019 ) . It is therefore difficult to know how much of the success of a model based on the original BERT checkpoint is due to BERT ’ s design , and how much is due to idiosyncracies of a particular artifact . Understanding this difference is critical if we are to generate reusable insights about deep learning for NLP , and improve the state-of-the-art going forward ( Zhou et al. , 2020 ; Dodge et al. , 2020 ; Aribandi et al. , 2021 ) . 1http : // [ anonymized ] This paper describes MultiBERTs , an effort to facilitate more robust research on the BERT model . Our primary contributions are : • We release MultiBERTs , a set of 25 BERT-Base checkpoints to facilitate studies of robustness to parameter initialization and order of training examples ( §2 ) . Releasing these models preserves the benefits to the community of a single checkpoint release ( i.e. , low cost of experiments , apples-to-apples comparisons between studies based on these checkpoints ) , while enabling researchers to draw more general conclusions about the BERT pre-training procedure . • We present the Multi-Bootstrap , a non-parametric method to quantify the uncertainty of experimental results based on multiple pre-training seeds ( §3 ) , and provide recommendations for how to use the Multi-Bootstrap and MultiBERTs in typical experimental scenarios . We implement these recommendations in a software library . • We illustrate the approach with a practical use case : we investigate the impact of counterfactual data augmentation on gender bias , in a BERT-based coreference resolution systems ( Webster et al. , 2020 ) ( §4 ) . Additional examples are provided in Appendix E , where we document challenges with reproducing the widely-used original BERT checkpoint . The release also includes an additional 140 intermediate checkpoints , captured during training for 5 of the runs ( 28 checkpoints per run ) , to facilitate studies of learning dynamics . Our checkpoints and statistical libraries are available at : http : // [ anonymized ] . Additional Related Work . The MultiBERTs release builds on top of a large body of work that seeks to analyze the behavior of BERT ( Rogers et al. , 2020 ) . In addition to the studies of robustness cited above , several authors have introduced methods to reduce BERT ’ s variability during finetuning ( Zhang et al. , 2021 ; Mosbach et al. , 2021 ; Dodge et al. , 2020 ; Lee et al. , 2020 ; Phang et al. , 2018 ) . Other authors have also studied the time dimension , which motivates our release of intermediate checkpoints ( Liu et al. , 2021 ; Hao et al. , 2020 ; Saphra & Lopez , 2019 ; Chiang et al. , 2020 ; Dodge et al. , 2020 ) . Similarly to §3 , authors in the NLP literature have recommended best practices for statistical testing ( Koehn , 2004 ; Dror et al. , 2018 ; Berg-Kirkpatrick et al. , 2012 ; Card et al. , 2020 ; Søgaard et al. , 2014 ; Peyrard et al. , 2021 ) , many of which are based on existing tests to estimate the uncertainty of test sample . In concurrent work , Deutsch et al . ( 2021 ) considered bootstrapping methods similar to the Multi-Bootstrap , in the context of summarization metrics evaluation . Also in concurrent work , the Mistral ( Karamcheti et al. , 2021 ) project released a set of 10 GPT-2 models with intermediate checkpoints at different stages of pre-training . Our work is complementary , focusing on BERT , introducing a larger number of pre-training seeds , and presenting a methodology to draw robust conclusions about model performance . 2 RELEASE DESCRIPTION . We first describe the MultiBERTs release : how the checkpoints were trained and how their performance compares to the original BERT on two common language understanding benchmarks . 2.1 TRAINING . Overview . The MultiBERTs checkpoints are trained following the code and procedure of Devlin et al . ( 2019 ) , with minor hyperparameter modifications necessary to obtain comparable results on GLUE ( Wang et al. , 2019 ) ; a detailed discussion of these differences is provided in Appendix E. We use the BERT-Base , Uncased architecture with 12 layers and embedding size 768 . We trained the models on a combination of BooksCorpus ( Zhu et al. , 2015 ) and English Wikipedia . Since the exact dataset used to train the original BERT is not available , we used a more recent version that was collected by Turc et al . ( 2019 ) with the same methodology . Checkpoints . We release 25 models trained for two million steps each , each training step involving a batch of 256 sequences . For five of these models , we release 28 additional checkpoints captured over the course of pre-training ( every 20,000 training steps up to 200,000 , then every 100,000 steps ) . In total , we release 165 checkpoints , about 68 GB of data . Training Details . As in the original BERT paper , we used batch size 256 and the Adam optimizer ( Kingma & Ba , 2014 ) with learning rate 1e-4 and 10,000 warm-up steps . We used the default values for all the other parameters , except the number of steps , which we set to two million , and sequence length , which we set to 512 from the beginning with up to 80 masked tokens per sequence.2 We follow the BERT code and initialize the layer parameters from a truncated Normal distribution , using mean 0 and standard deviation 0.02 . We train using the same configuration as Devlin et al . ( 2019 ) 3 , with each run taking about 4.5 days on 16 Cloud TPU v2 chips . Environmental Impact Statement . We estimate compute costs at around 1728 TPU-hours for each pre-training run , and around 208 GPU-hours plus 8 TPU-hours for associated fine-tuning experiments ( §2.2 , including hyperparameter search and 5x replication ) . Using the calculations of Lacoste et al . ( 2019 ) 4 , we estimate this as about 250 kg CO2e for each of our 25 models . Counting the 25 runs each of CDA-incr and CDA-full from §4 , associated coreference models ( 20 GPU-hours per pretraining model ) , and additional experiments of Appendix E , this gives a total of about 12.0 metric tons CO2e before accounting for offsets or clean energy . Based on the report by Patterson et al . ( 2021 ) of 78 % carbon-free energy in Google Iowa ( us-central1 ) , we estimate that reproducing these experiments would emit closer to 2.6 tons CO2e , or slightly more than two passengers on a round-trip flight between San Francisco and New York . By releasing the trained checkpoints publicly , we aim to enable many research efforts on reproducibility and robustness without requiring this cost to be incurred for every subsequent study . 2.2 PERFORMANCE BENCHMARKS GLUE Setup . We report results on the development sets of the GLUE tasks : CoLA ( Warstadt et al. , 2019 ) , MNLI ( matched ) ( Williams et al. , 2018 ) , MRPC ( Dolan & Brockett , 2005 ) , QNLI ( v2 ) ( Rajpurkar et al. , 2016 ; Wang et al. , 2019 ) , QQP ( Chen et al. , 2018 ) , RTE ( Bentivogli et al. , 2009 ) , SST2 ( Socher et al. , 2013 ) , and SST-B ( Cer et al. , 2017 ) . In all cases we follow the same approach as Devlin et al . ( 2019 ) . For each task , we fine-tune BERT for 3 epochs using a batch size of 32 . We run a parameter sweep on learning rates [ 5e-5 , 4e-5 , 3e-5 , 2e-5 ] and report the best score . We run the procedure five times for each of the 25 models and average the results . 2Specifically , we keep the sequence length constant ( the paper uses 128 tokens for 90 % of the training then 512 for the remaining 10 % ) to expose the model to more tokens and simplify the implementation . As we were not able to reproduce original BERT exactly using either 1M or 2M steps ( see Appendix E for discussion ) , we release MultiBERTs trained with 2M steps under the assumption that higher-performing models are more interesting objects of study . 3We train with TensorFlow ( Abadi et al. , 2015 ) version 2.5 in v1 compatibility mode . 4https : //mlco2.github.io/impact/ SQuAD Setup . We report results on the development sets of SQuAD versions 1.1 and 2.0 ( Rajpurkar et al. , 2016 ; 2018 ) , using a setup similar to that of Devlin et al . ( 2019 ) . For both sets of experiments , we use batch size 48 , learning rate 5e-5 , and train for 2 epochs . Results . Figures 1 and 2 show the distribution of the MultiBERTs models ’ performance on the development sets of GLUE and SQuAD , in comparison to the original BERT checkpoint.5 On most tasks , original BERT ’ s performance falls within the same range as MultiBERTs ( i.e. , original BERT is between the minimum and maximum of the MultiBERTs ’ scores ) . However , original BERT outperforms all MultiBERTs models on QQP , and under-performs them on SQuAD . The discrepancies may be explained by both randomness and differences in training setups , as investigated further in Appendix E. To further illustrate the performance variability inherent to pre-training and fine-tuning , we analyze the instance-level agreement between the models in Appendix C .

Summary: Many tasks in contemporary NLP begin by building off of a large language model.  This can cast the downstream task as a sort of fine-tuning experiment, whereby the results are heavily influenced by conditioning on the starting point of a single pre-trained version of an LLM.  In this work, the authors take BERT as an example, and ask how much a specific artifact as a draw from the distribution over (model weights, initialization scheme, training data, loss function) affects downstream tasks built upon it.    The authors provide a wide variety of BERT models that are varied in their training and initalization.  They define a bootstrap procedure for the scenario where multiple instantiations of base models are available, and tie their findings together in a case study of gender bias in coreference resolution.