The MultiBERTs: BERT Reproductions for Robustness Analysis
Authors: Thibault Sellam
Steve Yadlowsky
Ian Tenney
Jason Wei
Naomi Saphra
Alexander D'Amour
Tal Linzen
Jasmijn Bastings
Iulia Raluca Turc
Jacob Eisenstein
Dipanjan Das
Ellie Pavlick
Venue: ICLR 2022 Spotlight (2022)
Keywords: Pre-trained models
BERT
bootstrapping
hypothesis testing
robustness

Abstract: Experiments with pre-trained models such as BERT are often based on a single checkpoint. While the conclusions drawn apply to the artifact tested in the experiment (i.e., the particular instance of the model), it is not always clear whether they hold for the more general procedure which includes the architecture, training data, initialization scheme, and loss function. Recent work has shown that repeating the pre-training process can lead to substantially different performance, suggesting that an alternative strategy is needed to make principled statements about procedures. To enable researchers to draw more robust conclusions, we introduce MultiBERTs, a set of 25 BERT-Base checkpoints, trained with similar hyper-parameters as the original BERT model but differing in random weight initialization and shuffling of training data. We also define the Multi-Bootstrap, a non-parametric bootstrap method for statistical inference designed for settings where there are multiple pre-trained models and limited test data. To illustrate our approach, we present a case study of gender bias in coreference resolution, in which the Multi-Bootstrap lets us measure effects that may not be detected with a single checkpoint. The models and statistical library are available online, along with an additional set of 140 intermediate checkpoints captured during pre-training to facilitate research on learning dynamics.

Published as a conference paper at ICLR 2022
THE MULTIBERTS: BERT REPRODUCTIONS FOR ROBUSTNESS ANALYSIS
Thibault Sellam, Steve Yadlowsky, Ian Tenney, Jason Wei, Naomi Saphra3 Alexander D' A'mour, Tal Linzen, Jasmijn Bastings, Iulia Ture, Jacob Eisenstein, Dipanjan Das, and Ellie Pavlick
(tsellam, yadlowsky, iftenney, epavlick) (gopllkkkkgooole.ccooooo Google Research
ABSTRACT
Experiments with pre-trained models such as BERT are often based on single checkpoint. While the conclusions drawn apply to the artifact tested in the ex- periment (i.e., the particular instance of the model), it is not always clear whether they hold for the more general procedure which includes the architecture, training data, initialization scheme, and loss function. Recent work has shown that repeat- ing the pre-training process can lead to substantially different performance, sug- gesting that alternate strategy is needed make principled statements about procedures. To enable researchers to draw more robust conclusions, we intro- duce the MultiBERTS, set of 25 BERT-Base checkpoints, trained with similar byper-parameters as the original BERT model but differing in random weight ini- tialization and shuffling of training data. We also define the Multi-Bootstrap. a non parpaaraettrcc bootstrap method for statistical inference designed for settings where there are multiple pre-trained models and limited test data. To illustrate our approach, we present a case study of gender bias in coreference resolution, in which the Multi-Bootstrap lets us measure effects that may not be detected with single checkpoint. We release our models and statistical library," along with an additional set of 140 intermediate checkpoints captured during pre-training to facilitate research on learning dynamics.
INTRODUCTION
Contemporary natural language processing (NLP) relies heavily on pretrained language models, which are trained using large-scale unlabeled data (Bommasani et al., 2021). BERT (Devlin al., 2019) is a particularly popular choice: it has been widely adopted in academia and industry, and aspects of its performance have been reported on in thousands of research papers (see, e.g., Rogers et al., 2020, for an overview). Because pre-trac pre-training large language models is computationally ex- pensive (Strubell et al., 2019), researchers often rely on the release of model checkpoints through libraries such HuggingFace Transformers (Wolf et al., 2020), which enable them to use large-scale language models without repeating the pre-training work. Consequently, most published results are based on small number of publicly released model checkpoints.
While this reuse model checkpoints has lowered the cost of research and facilitated head-to-head comparisons, it limits ability to draw general scientific conclusions about the performance of particular class models (Dror et al., 2019; D' Amour et al., 2020; Zhong et al., 2021). The key issue is that reusing model checkpoints makes it hard generalize observations about the behavior of a single model artifact to statements about the underlying pre-training procedure which created it. Pre-training such models is an inherently stochastic process which depends the initializa- tion of the model's parameters and the ordering of training examples; for example, D' Amour et al.
Equal contribution.
Work done Google Al resident.
t Work done during an internship at Google. hhtp:://gee gl e e/mul m iiberts

---

Published as a conference paper at ICLR 2022
(2020) report substantial quantitative differences across multiple checkpoints of the same model chitecture on several "stress tests" (Naik et al., 2018; McCoy et al., 2019). It is therefore difficult to know how much of the success of model based on the original BERT checkpoint is due to BERT'S design, and how much is due to idiosyncracies of a particular artifact. Understanding this differ- ence is critical if are generate reusable insights about deep learning for NL.P, and improve the state-of-the-e going forward (Zhou et al., 2020; Dodge et al., 2020; Aribandi et al., 2021).
This paper describes the MultiBERTS, an effort to facilitate more robust research the BERT model. Our primary contributions are:
We release the MultiBERTS, set of 25 BERT-Base, Uncased checkpoints to facilitate stud- ies of robustness to parameter initialization and order training examples ($2). Releas- ing these models preserves the benefits to the community of single checkpoint release (i.e., low cost of experiments, apples-to-apples comparisons between studies based on these checkpoints), while enabling researchers to draw more general conclusions about the BERT pre-training procedure.
We present the nulti-Boottstap. non-parametric method to quantify the uncertainty of ex- perimental results based on multiple pre-training seeds ($3), and provide recommendations for how to the Multi-Bootstie and MultiBERTS in typical experimental scenarios. We implement these recommendations in a software library.
We illustrate the approach with a practical use case: we investigate the impact of coun- terfactual data augmentation on gender bias, in a BERT-based coreference resolution sys- tems (Webster et al., 2020) (š4). Additional examples are provided in Appendix E, where we document challenges with reproducing the widely-used original BERT checkpoint.
The release also includes an additional 140 intermediate checkpoints, captured during training for 5 of the runs (28 checkpoints per run), to facilitate studies learning dynamics. Our checkpoints and statistical libraries are available at: http: : / /go.. gle/mu mmmummltte berts..
Additional Related Work. The MulTiBERTS release builds on top of large body of work that seeks to analyze the behavior BERT (Rogers et al., 2020). In addition to the studies of robust- ness cited above, several authors have introduced methods to reduce BERT'S 's variability during fine- tuning (Zhang et al., 2021; Mosbach et al., 2021; Dodge et al., 2020; Lee et al., 2020; Phang et al., 2018). Other authors have also studied the time dimension, which motivates diate checkpoints (Liu et al., 2021; Hao et al., 2020; Saphra & Lopez, 2019; Chiang et al., 2020; Dodge et al., 2020). Similarly to $3, authors in the NLP literature have recommended best practices for statistical testing (Koehn, 2004; Dror et al., 2018; Berg-KKrkpatrick et al., 2012: Card et al., 2020; Søgaard et al.. 2014; Peyrard al., 2021), many of which are based on existing tests to estimate the uncertainty of test sample. In concurrent work, Deutsch et al. (2021) considered bootstrapping methods similar to the Multi-Bootstrap, in the context of summarization metrics evaluation. Also in concurrent work, the Mistral project (Karamcheti et al., 2021) released set of 10 GPT-2 models with intermediate checkpoints at different stages of pre-training. Our work is complementary, cusing BERT, introducing a larger number of pre-training seeds, and presenting methodology to draw robust conclusions about model performance.
release of interme-
2 RELEASE DESCRIPTION
We first describe the MultiBERTS release: how the checkpoints were trained and how their perfor- mance compares to the original BERT on two common language understanding benchmarks.
2.1 TRAINING
Overview. The MultiBERTs checkpoints are trained following the code and procedure of Devlin et al. (2019), with minor hyperparameter modifications necessary to obtain comparable results on GLUE (Wang et al., 2019); detailed discussion of these differences is provided in Appendix E. We use the BERT-BE BERT-Base, Base, Uncased architecture with 12 layers and embedding size 768. We trained the models on combination of BooksCorpus (Zhu al., 2015) and English Wikipedia. Since the
2

---

Published as a conference paper at ICLR 2022
COLA
(OLL
MRPG
DNU
RTE
SST-2 jaco!
STS. ITS-BIM
ccreckpcint
2
20 21 0 2
Figure 1: Distribution of the performance GLUE dev sets (Wang al., 2019), averaged across fine-tuning runs for each checkpoint. The dashed line indicates the performance of the original BERT release.
exact dataset used train the original BERT is not available, used more recent version that was collected by Turc et al. (2019) with the same methodology.
Checkpoints. We release 25 models trained for two million steps each, each training step involv- ing a batch of 256 sequences. For five of these models, we release 28 additional checkpoints captured over the course of pre-training (every 20,000 training steps up to 200,000, then every 100,000 steps). In total, we release 165 checkpoints, about 68 GB of data.
Training Details. As in the original BERT paper, we used batch size 256 and the Adam opti- mizer (Kingma & Ba, 2014) with learning rate le-4 and 10,000 warm-up steps. We used the default values for all the other parameters, except the number of steps, which we set to two million, and se- quence length, which
set to 512 from the beginning with up to 80 masked tokens per sequence." We follow the BERT code and initialize the layer parameters from a truncated normal distribution, using mean 0 and standard deviation 0.02. We train using the same configuration as Devlin et al. (2019)3, with each run taking about 4.5 days on 16 Cloud TPU v2 chips.
Environmental Impact Statement. We estimate compute costs at around 1728 TPU-hours for each pre-train run, and around 208 GPU-hours plus 8 TPU-hours for associated fine-tuning ex- periments ($2.2, including hyperparameter search and 5x replication). Using the calculations of Luccioni et al. (2019)*, we estimate this as about 250 kg CO2e for each of our 25 models. Counting the 25 runs each of CDA-iner and CDA-full from $4, associated coreference models (20 GPU-hours per pretraining model), and additional experiments of Appendix E, this gives total of about 12.0 metric tons CO2e before accounting for offsets or clean energy. Based on the report by Patterson et al. (2021) of 78% carbon-fire -fren energy in Google Iowa (us-centrall), we estimate that reproducing these experiments would emit closer to 2.6 tons CO2e, or slightly more than two passengers on a round-trip flight between San Francisco and New York. By releasing the trained checkpoints pub- licly, we aim to enable many research efforts on reproducibility and robustness without requiring this cost to be incurred for every subsequent study.
2.2 PERFORMANCE BENCHMARKS
GLUE Setup. We report results on the development sets of the GLUE tasks: CoLA (Warstadt et al., 2019), MNLI (matched) (Williams al., 2018), MRPC (Dolan & Brockett, 2005), QNLI (v2) (Rajpurkar et al., 2016; Wang et al., 2019), QQP (Chen et al., 2018), 8), RTE (Bentivogli et al., 2009), SST-2 (Socher et al., 2013), and SST-B (Cer et al., 2017). In all cases we follow the same approach as Devlin et al. (2019). For each task, we fine-tune BERT for 3 epochs using a batch
Specificallyy keep the sequence length constant (the paper 128 tokens for 90% of the training then 512 for the remaining 10%) expose the model more tokens and simplify the implementation. As we were not able to reproduce original BERT exactly using either 1M or 2M steps (see Appendix for discussion), we release MultiBERTS trained with 2M steps under the assumption that higher-performing models are more interesting objects of study.
3 WWe use https: pestps:/ggte //gittiit huu. wtw..hhw..oooe.eeeeeeeeeee cww/goog////oooooooooooooooooooo ee reweerch/beeeeeeeeeeeeeeeeeee with TensorFlow (Abadi et al., 2015) version 2.5 in vl compatibility mode.
*tttps: /ml co2. github. i0/ / /pact//
3

---

Published as a conference paper at ICLR 2022
size of 32. We parameter sweep learning rates [5e-5, 4e-5, 3e-5, 2e-5] and report the best score. We rum the procedure five times for each the 25 models and average the results.
SQuAD Setup. We report results on the development sets of SQuAD versions 1.1 and 2.0 (Rajpurkar et al., 2016; 2018),
using a setup similar to that of Devlin et al. (2019). For both sets of experiments, we use batch size 48, learning rate 5e-5, and train for 2 epochs.
Results. Figures 1 and 2 show the distribution of the MultiBERTS models' performance on the development sets of GLUE and SQuAD, in comparison to the original BERT checkpoint. On most tasks, original BERT'S performance falls within the same range as MultiBERTs (i.e., original BERT is between the minimum and maximum of the MultiBERTS' scores). However, original BERT outperforms all MuTtiBERTS models on QQP, and under-performs them on SQuAD. The discrepancies may be explained by both randomness and differences in train- ing setups, as investigated further Appendix
Figure 2: Performance distribution on the dev sets of SQuAD v1.1 and v2.0 (Rajpurkar al., 2016; 2018).
To further illustrate the performance variability inherent to pre-training and fine-tuning, we analyze instance-level agreement between models Appendix
HYPOTHESIS TESTING USING MULTIPLE CHECKPOINTS
The previous section compared MultiBERTS with the original BERT, finding many similarities but also some differences (c.g., in the case of SQuAD). To what extent can these results be explained by random noise? More generally, how can we quantify the uncertainty of a set of experimental results when there multiple sources of randomness?
In parallel to the MultBERTSs release, we propose a more principled and standardized method compare training procedures. We recommend non-parametrie bootstrapping procedure, the "Multi-Bootstrap"" which enables us to make inference about model performance in the face of multiple sources of uncertainty: the randomness due to the pre-training seed, the fine-tuning seed, and the finite test data. The main idea is to use the average behavior over seeds as means of summarizing expected behavior in an ideal world with infinite samples.
Although we present Multi- BoltiiBBoottrapp in the context of analyzing the MultiBERTs, the method could be applied in all setups that involve a of checkpoints pre-trained with the same method, a finite test set, and (possibly) multiple rounds of fine-tuning The multi-Bootstrap is implemented as Python library, included with the MultiBERTS release.
3.1 INTERPRETING STATISTICAL RESULTS
The Multi- provides an estimate of the amount of remaining uncertainty when summariz- ing the performance over multiple seeds. The following notation will help us state this precisely. We assume access to model predictions fz) for each instance æ in the evaluation set. We consider randomness arising from:
1. The choice of pre-training seed S M
2. The choice fine-tuning seed T N
3. The choice test sample (X, Y) D
The Multi-Bootstran procedure allows to account for all of the above. Specifically, MultiBERTS enables us to estimate the variance due to the choice of pre-training seed (1), which would not be possible with single artifact. Note that multiple fine-tuning runs not required in order to the procedure.
'We used https: /ttp/a/ttaaeeeeeeeeeeeeeeeaeeee. googll...oll... googleapis sociiscoo com http/bbrrttoddee// s s/2020002_200mmmee 1.- 1--12_ 768 A- 12. zip, linked from https: / //gi htgggttttu...///gg com/ researeeeaee reasearch. /bert.
4

---

Published as a conference paper at ICLR 2022
For each pre-training seed let fz( denote the learned model's prediction input features and let L(s) denote the expected performance metric of fa on a test distribution D over features X and labels Y. For example, the accuracy would be [(s) B[ fx) We can use the test sample (which we will assume has ny examples) to estimate the performance for each of the seeds in MultiBERTS, which we denote as L(s).
The performance Lss) depends on the seed, but we are interested in summarizing the model over all seeds. A natural summary is the average over seeds, E [LS),, which we will denote by 8. Then, using n, independently sampled seeds,
compute an estimate @ as
0
[(S)). 1=1
Because 9 is computed under a finite evaluation set and finite number of seeds, it is necessary to quantify the uncertainty of the estimate. The goal of Multi-Bootstrap is to estimate the distribution of the error in this estimate, about 8, such as whether it is above some threshold of interest. Below, we describe a few common experimental designs in NLP that can be studied with these tools.
6, in order to compute confidence intervals and test hypotheses
Comparison to a Fixed Baseline. In many use cases, we want to compare BERT'S behavior to that of single, fixed baseline. For instance, does BERT encode information about syntax as a feature-engineered model would (Tenney et al., 2019; Hewitt & Manning, 2019)? Does it encode social stereotypes, and how does compare to human biases (Nadeem et al., 2021)? Does encode world knowledge, similarly to explicit knowledge bases (Petroni et al., 2019)? Does another model such RoBERTa (Liu al., 2019) outperform BERT on common tasks such as those from the GLUE benchmark?
Design 1:
In all these cases, we compare MultiBERTS some external baseline of which we only have single estimate (e.g., random or human performance), or against an existing model that is not derived from the MultiBERTS checkpoints. We treat the baseline as fixed, and assess only the uncertainty that arises from MultiBERT'' random seeds and the test examples.
Design 2: Paired Samples. Alternatively, we might seek to assess the effectiveness of a specific intervention model behavior. In such studies, an intervention is proposed (e.g.. representation learning via a specific intermediate task, or a specific architecture change) which can be applied to any pre-trained BERT checkpoint. The question whether the procedure results in an improvement over the original BERT pre-training method: does the intervention reliably produce the desired effect, or is the observed effect due to the idiosyncracies of a particular model artifact? Examples of such studies include: Does intermediate tuning on NLI after pre-training make models more robust across language understanding tasks (Phang al., 2018)? Does pruning attention heads degrade model performance on downstream tasks (Voita et al., 2019y? Does augmenting BERT with information about semantic roles improve performance on benchmark tasks (Zhang et al., 2020)?
We refer to studies like the above as paired since each instance of the baseline model fa (which does not receive the intervention) can be paired with an instance of the proposed model I' (which receives the stated intervention) such that Sa and I'. are based on the same pretrained checkpoint produced using the same seed. Denoting Oy and 0y' as the expected performance defined above for the bascline and intervention model respectively, our goal is to test hypotheses about the true difference in performance 0,p 1p-0; 0f using the estimated difference δ = 00 -
In a paired study, well as the correlation between the two. Together, these allow to approximate the distribution of the overall estimation error 8 5 = O y.) (0, rr)) between the estimate 8 and the truth 5. With this, we can compute confidence intervals for 6, the true average effect of the intervention on performance over seeds, and test hypotheses about as well.
allows us to estimate both of the errors A, O and - fy, as
Design 3: Unpaired Samples. Finally, we might seek to compare number of seeds for both the intervention and baseline models, but may not expect them to be aligned in their dependence on the seed. For example, the second model may use a different architecture that they cannot be built
5

---

Published as a conference paper at ICLR 2022
from the same checkpoints, the models may be generated from entirely separate initialization schemes. We refer to such studies as unpaired. Like in a paired study, the Multi-Boot Bootstrap allows us to estimate the errors Of- Of and 0 Oyr; however, in an unpaired study, we cannot estimate the correlation between the errors. Thus, we assume that the correlation zero. This will give a conservative estimate of the error (0, - 0r) (Ø, 0y), as long as 8 - f, and o 0y are not negatively correlated. Since there is little reason to believe that the random seeds used for two different models would induce a negative correlation between the models' performance, we take this assumption to be relatively safe.
Hypothesis Testing. Given the measured uncertainty, we recommend testing whether or not the difference is meaningfully different from some arbitrary predefined threshold (i.e., 0 in the typical case). Specifically, we are often interested in rejecting the null hypothesis that the intervention does not improve over the baseline model, i.e.,
Ho 6 <
in statistically rigorous way. This can be done with the Multi-Bootstrap procedure described below.
3.2 MULTI-BOOTSTRAP PROCEDURE
The Multi-Bootstren is non con-parametric bootstrapping procedure that allows us to estimate the dis- tribution of the eтror 0 - 0 over the seeds and test instances. The algorithm supports both paired and unpaired study designs, differentiating the two settings only the way the sampling is performed.
To keep the presentation simple, we will assume that the performance T.(s) is an average of per- example metric (ar, y, fs) over the distribution D of (X, Y), such as aceuracy or the log likelihood, and ((s) is similarly an empirical average with the observed n test examples,
L(s) = Xx, and i(s)= = Σ Ya.
We note that the mapping D - L(s) is linear in D, which is required for our result in Theorem 1. However, conjecture that this is artifact of the proof; like most bootstrap methods, the method here likely generalizes to any performance metric which behaves asymptotically like a linear map- ping of D, including AUC, BLEU score (Papineni et al., 2002), and expected calibration error.
Building on the rich literature on bootstrap methods (e.g., Efron & Tibshirani, 1994), the Multi- Bootstrap is new procedure which accounts for the way that the combined randomness from the seeds and test set creates error in the estimate 6. The statistical underpinnings of this approach have theoretical and methodological connections to inference procedures for two-sample tests (Van der Vaart, 2000), where the samples from each population are independent, However, in those settings, the test statistics naturally differ as a result of the scientific question at hand.
In our procedure, we generate bootstrap sample from the full sample with replacement sep- arately over both the randomness from the pre-training seed and from the test set (X, That is, we generate sample of pre-training seeds (Si, S2...., randomly with replacement from the pre-training seeds, and we generate a test set sample ((X;, (X2, Y2), Y2.....; (Xū, Yt)) with each (X, pair drawn randomly with replacement from the full test set. Then, we compute the bootstrap estimate 6* as
with each Sÿ drawn
D*S)), where (=(s)= 7s
Tix Σ (X).
To illustrate the procedure, we present a minimal Python implementation in Appendix A. For suf- ficiently large n, and n., the distribution of the estimation error 0 0 is approximated well by the distribution of 0~ - O over re-draws of the bootstrap samples, stated precisely in Theorem 1. Theorem 1. Assume that E[ Y,Js)] < ac. Furthermore, assume that for each 8, (X,, < co, and for almost every (x,y) pair, E[[ (,J) x= Y y] < co. Let = n2 + n2 and assume that 0 < <p, = na/n < stays fixed (up to rounding error) as n + xo.
Then, there exists < 0² 30 such that vn(0 0) $ G with G N(0, Furthermore, conditionally ((X1, Yi), (X2, Y2), ....,,,,, vū 0) $ G.

---

Published as a conference paper at ICLR 2022
The proof of Theorem 1 is in Appendix B, along with a comment the rate of convergence for the approximation error. The challenge with applying existing theory to our method is that while the seeds and data points each marginally iid, the observed losses depend on both, and therefore not iid. Therefore, we need to handle this non-iid structure in our method and proof.
For nested sources of randomness (e.g., if for each pre- tre-training seed 8, we have estimates from multi- ple fine-tuning seeds), we average over all of the inner samples (fine-tuning seeds) in every bootstrap sample, motivated by Field & Welsh (2007)'s s recommendations for bootstrapping clustered data.
Paired Samples (design 2, continued). In paired design, the Multt-Boottsrapp proce- dure can additionally tell us the joint distribution of 0g. 0p and 0f - 0,- To do so, one must use the same bootstrap samples of the seeds S2........ Si.)) and test examples Yr,)) for both models. Then, the correlation between the errors 0 0y fr and 8 - yy is well approximated by the correlation between the bootstrap errors 0j. f7. and 07.
(X), Y2), .y)..... (X.,
In particular, recall that we defined the difference in performance between the intervention f and the baseline f to be ö, and defined its estimator to be ö. With the Multi-Bootsse Bootstrap, we can estimate the bootstrapped difference
8 j 0j.
With this, the distribution of the estimation error is well approximated by the distribution of 8- 3 over bootstrap samples.
Unpaired Samples (design 3, continued). For studies that do not match the paired format, we adapt the Multi-Bootstrap procedure that, instead of sampling a single pre-training seed that is shared between f and f", we sample pre-training seeds for each one independently. The remainder of the algorithm proceeds as in the paired case. Relative to the paired design discussed above, this additionally assumes that the errors due to differences in pre- pre-training seed between 0y. 0 and 0, ff are independent.
Comparison to Fixed Baseline (design 1, continued). Often, we do not have access to multiple estimates of Lss), for example, when the baseline f against which we are comparing is an estimate of human performance for which only mean accuracy was reported, or when f is the performance of a previously-publishhes model for which there only exists a single artifact or for which we do not have direct access to model predictions. When we have only a point estimate 8,- = Ī(St) of 0 for compute a confidence
the baseline with a single seed S1, we recommend using interval around 0 and reporting where the given estimate of baseline performance falls within that distribution. An example of such case is Figure in which the distribution of BERTS performance is compared to that from the single checkpoint of the original BERT release. In general such results should be interpreted conservatively, as we cannot make any claims about the variance of the bascline model.
Hypothesis Testing. A valid p-value for the hypothesis test descrihed in Equation 1 is the fraction of bootstrap samples from the above procedure for which the estimate 8 is negative.
4 APPLICATION: GENDER BIAS IN COREFERENCE SYSTEMS
We present study to illustrate how MultBEETTs and the Multi-Bootstrap can help draw more robust conclusions about model behavior.
The use case is based on gendered correlations. For particular measure of gender bias, we take single BERT checkpoint and measure value of 0.35. We then apply an intervention, foo, designed to reduce this correlation, and measure 0.25. In effort do even better, we create whole new checkpoint by applying the foo procedure from the very beginning of pre-training. On this checkpoint, measure 0.3. How does one make sense of this result?
7

---

Published as a conference paper at ICLR 2022
Winogender blas correlation (r) by pretrain seed
1.0
CDA-Incr
0.8
0.6
correetton
0.4
2
3
6
7
10 11
12 Pretraining Seed
13
24
15
16
17
18
19
20
21
22
23
24
Figure 3: Bias correlation on Winogender for each pre-training seed. Each box represents the distri- bution of the score over five training runs of the coreference model. Dark boxes represent each base MultiBERTS checkpoint, while lighter boxes (CDA-incr) are the corresponding checkpoints after 50k steps of additional pretraining with CDA. Some seeds better than others on this task (for example, seed 23), but the CDA-ince consistently reduces the bias correlation for most seeds.
As a concrete example, we analyze gender bias in coreference systems (Rudinger et al., 2018) and showing how MultiBERTs and the tion, counterfactual data augmentation (CDA). We follow a set-up similar to Webster et al. (2020), which augments the BERT pretraining data with counterfactual sentences created by randomly swap- ping English binary-genders pronouns and other words such as occupation terms (e.g., doctor, nurse). We compare our baseline MultiBERTs models to two strategies for CDA. In the first (CDA-iner), we continue pre-training each MultiBERTs model for an additional 50K steps on the counterfactual data of Webster et al. (2020). In the second, we train BERT models from scratch (CDA-full) on the same dataset.
can help understand the effect of interven-
pronouns. The goal is to weaken the correlation between gendered
The Winogender dataset consists of template sentences covering 60 occupation terms and instan- tiated with either male, female, or neutral pronouns. We follow Webster et al. (2020) and train gold-mention coreference system using a two-layer feedforward network that takes span representa- tions from a frozen BERT encoder input and makes binary predictions for mention-refeee The model is trained on OntoNotes (Hovy et al., 2006) and evaluated on the Winogender examples for both per-sentence accuracy and bias score, defined the Pearson correlation between the per- occupation bias score (Figure 4 of Rudinger et al. 2018) and the occupational gender statistics from the U.S. Bureau of Labor Statistics." For each pre-training run, train five coreference models, using the same encoder but different random seeds to initialize the classifier weights and to shuffle the training data.
pairs.
4.1 PAIRED ANALYSIS: CDA-INCR VS. BASE
We investigate the impact of the intervention on performance and bias. Overall accuracy is fairly consistent across pre-training seeds, at 62.6&11.2% for the base model, with only a small and not statistically significant change under CDA-iner (Table 1). However, considerable variation in bias correlation, with r values between 0.1 and 0.7 depending on pre- training seed.? The range for CDA-iner overlaps somewhat, with values between 0.0 and 0.4; how- ever, because the incremental CDA is an intervention on each base checkpoint, we can look at the individual seeds and see that most cases there appears be significant improvement. paired Multi-Boue nulti-Bootstrap allows us to quantify this and further account for noise due to the finite evaluation
shown in Figure there
We use the occupation data as distributed with Winogender dataset, https:/ //ggt//ggtttbb// com/
rudi ngge/ //w//ww nogender- schemas.
"Some of this variation is due to the classifier training, but on this task there is large intrinsic contribution from the pretraining seed. See Appendix for detailed analysis.
8

---

Published as a conference paper at ICLR 2022
sample of 60 occupations. The results shown in Table 1, which show that CDA-iner significantly
reduces bias by 8 = -0.162 with p = 0.001.
Accuracy
Bias Corr. (r)
Base
8
0,626
0,423
CDA-iner
0.623
0.261
Avg. Diff.
8 p -0- 8
-0.004
-0.162
p-value
0.210
0.001
Table I: Paired Multi-Bootstrap results for CDA intervention over the base MultiB ERTS checkpoints on Winogender. Accuracy is computed by bootstrapping over all 720 examples, while for bias corre- lation we first compute per-occupation bias scores and then bootstrap over the 60 occupation terms. For both, we use 1,000 bootstrap samples. lower value of indicates less gender-occupation bias.
Accuracy
Bias Corr. (r)
Seeds
Examples
CDA-iner
0
0,623
0.256
0,264
0.259
CDA-full
0p
0.622
0.192
0.194
0.193
Avg. Diff.
8 = 0p 0f
-0.001
-0.064
-0.070
-0.067
p-value
0.416
0.132
0.005
0.053
Table 2: Unpaired Multi- Multi-Boottstra results comparing CDA-full to CDA-iner on Winogender. Ex- amples are treated as in Figure 1. The "Seeds" column bootstraps only over pre-training seeds while using the full set of 60 occupations, while the "Examples" column bootstraps over examples, averaging over all pre-trainn pre-training seeds. For all tests
1,000 bootstrap samples.
4.2 UNPAIRED ANALYSIS: CDA-FULL VS. CDA-INCR
We can also test we get any additional benefit from running the entire pre- pre-training with counterfactually augmented data. Similar to MultiBERTS, 2M steps on the CDA dataset. & Because these are entirely new checkpoints, independent from the base MultiBERTS runs, of examples but samples pretraining seeds independently for CDA-incr and CDA-full. As shown in Table 2, overall accuracy does not change appreciably (0.622 vs. 0.623, p 0.416), while bias correlation seems to decrease but not significantly (0.256 vs 0. 192, 5 -0.064 with 0.132).
trained 25 CDA-full checkpoints for
unpaired version of the Multi-Bootstrap. which uses the same
As ablation, also experiment with sampling over either only seeds (taking the of examples, ie. occupations, as fixed), or over examples (taking the set of 25 seeds as fixed). As shown in Table we find lower p-values (0.005 and 0.053) in both cases sroes-showingg that failing to account for finite samples along either dimension could lead to overconfident conclusions.
In Appendix E, we present two additional examples: paired study where we increase pre- training time from 1M to 2M steps, as well as an unpaired comparison to the original bert-base lett--bas-unccsser uncased checkpoint.
CONCLUSION
To make progress on language model pre-training, it is essential to distinguish between the properties of specific model artifacts and those of the training procedures that generated them. To this end, we have presented research on BERT, and the Multi-Bootstrap. non-parametric statistical method to estimate the uncertainty of model comparisons across multiple training seeds. We demonstrated the utility of these resources by showing how to quantify the effect of an intervention to reduce a type of gender bias in coreference systems built on BERT. We hope that the release of multiple checkpoints and the use of principled hypothesis testing will become standard practices in research on preitrained language models.
resources: the MultiBERTS, set of 25 model checkpoints support robust
Following Webster al. (2020), 20 masks per sequence instead of the 80 from Devlin al. (2019).

---

Published as a conference paper at ICLR 2022
REFERENCES
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving. Michael Isard, Yangging Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg. Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wat- tenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-scale machine learn- ing on heterogeneous systems, 2015. URL https:/ /twp//wwww.... www.ornnnorrooo org/ Software available from tensorflow.org.
Vamsi Aribandi, Yi Tay, and Donald Metzler. How reliable model diagnostics? In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 1778-1785, Online, August 2021. Association for Computational Linguisties. doi: 10. 1 URL attps://acllnt
org/2021. mndings-acl- 155.
Luisa Bentivogli, Ido Dagan, Hoa Trang Dang, Danilo Giampiccolo, and Bernardo Magnini. The fifth PASCAL recognizing textual entailment challenge. In TAC. National Institute of Standards and Technology, 2009.
Taylor
David Burkett, and Dan Klein. An empirical investigation of statistical significance in nlp. In Proceedings of the 2012 Joint Conference Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pp. 995-1005, 2012.
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Ste- fano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Pe- ter Henderson, John Hewitt, Daniel E. Ho. Jenny Hong. Kyle Hsu, Jing Huang, Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Kohd, Mark Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Ben Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, Julian Nyarko, Giray Ogut, Laurel Orr, Isabel Papadim- itriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Rob Reich, Hongyu Ren, Frieda Rong. Yusuf Roohani, Camilo Ruiz, Jack Ryan, Christopher Ré, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishnan Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang. William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. On the opportunities and risks of foundation models, 2021.
Francesco Cantelli. Sulla probabilità come limite della frequenza. Rendiconti della Reale Accademia dei Lincei, 26(1):39-45, 1917.
Dallas Card, Peter Henderson, Urvashi Khandelwal, Robin Jia, Kyle Mahowald, and Dan Jurafsky. With little power comes great responsibility. Proceedings of the 2020 Conference on Em- pirical Methods in Natural Language Processing (EMNLP), pp. 9263-9274, Online, November 2020. Association for Computational Linguistics. doi: 10. 11200///11122200mmmmmmmalall 745. URL https: http:///aaaaatthogog.ooo.. org/ 2020 emnlp-main, 745.
Daniel Cer, Mona Diab, Eneko Agirre, lñigo Lopez-Gazpio. and Lucia Specia. SemEval-2011 task I: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 1 11th International Workshop on Semantic Evaluation ComEvval22117 pp. 1-14, Vancouver, Canada, August 2017. Association for Computational Linguistics. doi: 10.18653/v1/S17--1 2001. URL https: //www a cl web. org oww.bborg/nttooooogoooooooooooo hor /0tpy///1/200111
10

---

Published as a conference paper at ICLR 2022
Zihan Chen, Hongbo Zhang, Xiaoji Zhang, and Leqi Zhao. Quora question pairs. University of Waterloo, 2018.
Cheng-Han Chiang, Sung-Feng Huang, and Hung-yi Lee. Pretrained language model embryol-
ogy: The birth of ALBERT. In Proceedings of the 2020 Conference on Empirical Methods
in Natural Language Processing (EMNLP), pp. 6-288. Online, November 2020. Associ-
ation for Computational Linguistics. doi: https//////22.mmmmmmm cwnlp-mmin.555. URL https:
hww.aaatttooooooooooraeeeo emnip-
William G Cochran. Sampling techniques. John Wiley & Sons, 2007.
Alexander D' D'Amour, Katherine Heller, Dan Moldovan, Ben Adlam, Babak Alipanahi, Alex Beu- tel, Christina Chen, Jonathan Deaton, Jacob Eisenstein. Matthew Hoffman, et al. Under- specification presents challenges for credibility in modern machine learning. arXiv preprint arXiv:201 1.03395, 2020.
Daniel Deutsch, Rotem Dror, and Dan Roth. A statistical analysis of summarization evaluation met- rics using resampling methods. Transactions of the Association for Computational Linguistics, 9: 1132-1146, 2021. doi: 10. 162/tall 00417. URL https: //aa/caanthhlooo tacl-1.6 67.
2021.
Jacob Devlin, Ming-Wei. Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers). pp. 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-11 1423. URL https: / /www. wwwwwwwwbbb....................w borg/anthologgg / 1423.
Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannanch Hajishirzi, and Noah Smith. Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping. arXiv preprint arXiv:2002.06300; 2020.
William B. Dolan and Chris Brockett. Automatically constructing corpus sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (ZWP2005), 2005. URL
https:/ / /wwwww.... aclweb. www.oorgg/atttggggoeeeee anthology 100-5002.
Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. The hitchhiker's guide to testing statistical significance in natural language processing. In Proceedings the 56th Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers). pp. 1383-1392, 2018,
Rotem Dror, Segev Shlomov, and Roi Reichart. Deep dominance how properly compare deep neural models. In Proceedings of the 57th Annual Meeting of the Association for Computa- tional Linguistics, pp. 2773-2785, Florence, Italy, July 2019. Association for Computational Lin- guistics. doi: 10. 1/118633/1//11-12666 URL https:/ /www///w.o..mom/.oo acl web. herg//anholl ogyy P19-122 119-.
Bradley Efron and Robert J Tibshirani. An introduction to the bootstrap. CRC Press, 1994.
M Émile Borel. Les probabilités dénombrables et leurs applications arithmétiques. Rendiconti del Circolo Matematico di Palermo (1884-1940), 271):2477- 1909.
Nasrollah Etemadi. An elementary proof of the strong law of large numbers. Zeitschrift für Wadhscheennicckeeissseeei und verwandte Gebiete, 55(1): 19-122, 1981.
Christopher A Field and Alan H Welsh. Bootstrapping clustered data. Journal the Royal Statistical Society: Series B (Statistical Methodology), 69(3):369-390, 2007.
Yaru Hao, Li Dong, Furu Wei, and Ke Xu. Investigating learning dynamics of BERT fine-tuning. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computa- tional Linguistics and the 10th International Joint Conference Natural Language Processing, pp. 87-92, Suzhou, China, December 2020. Association for Computational Linguistics. URL https: a antho alllhhhhloyyy..ooooooooo logy. wwwoorgg/2000 aacl -mae-mmaiii 11

---

Published as a conference paper at ICLR 2022
John Hewitt and Christopher D. Manning. A structural probe for finding syntax in word representa- tions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Pa- pers), pp. 4129-4138, Minneapolis, Minnesota, June 2019. Association for Computational Lin- guisties. doi: 10.18653/11/X19-141999 URL https:: /wwww a a web. org/ aeagg//nttht hoathhoooooo ogy/ 119-1
Eduard Hovy, Mitchell Marcus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel. Ontonotes:
The 90% solution. In Proceedings of the Human Language Technology Conference of the NAACL, Companion Volume: Short Papers, NAACL-Short '06, pp. 57-60, Stroudsburg, PA, USA, 2006. Association for Computational Linguistics. URL http: : /al acm. wwt.ooog//t/.oooooooooooooe wwt//iianniinnnnn cfm?
1d=1614049 1614064.
Siddharth Karamcheti, Laurel Orr, Jason Bolton, Tianyi Zhang, Karan Goel, Avanika Narayan, Rishi Bommasani, Deepak Narayanan, Tatsunori Hashimoto, Dan Jurafsky, Christopher D. Manning. Christopher Potts, Christopher Ré, and Percy Liang. Mistral a journey towards reproducible lan- guage model training, 2021. URL https: /eattps://gittee //github. com/ httssssaafodddrffee aa....iaaaa..ll...............lm
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Philipp Koehn. Statistical significance tests for machine translation evaluation. In Proceedings of the 2004 conference on empirical methods in natural language processing, pp. 388-395, 2004.
Cheolhyoung Lee, Kyunghyun Cho, and Wanmo Kang. Mixout: Effective regularization to finetune large-scale pretrained language models. In International Conference Learning Representa- tions, 2020. URL https::////.. http///pppeeeeeeeeeeeeeeeeeeeeee /
net 2 wwwoormmii-HHgaaaae...
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1907:1997.1199// 2019.
Zeyu Liu, Yizhong Wang. Jungo Kasai, Hannaneh Hajishirzi, and Noah A. Smith. Probing across time: What does RoBERTa know and when? In Findings of the Association for Computational Linguistics: EMNLP 2021. pp. 820-842, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021 1/t.65//1/221..nninnneeeee/ findings-emnlp.71. URL htttps://aclattholl
org/ 2021. fi Endings-emt 1p.71. 71.
Sasha Luccioni, Victor Schmidt, Alexandre Lacoste, and Thomas Dandres. Quantifying the car- bon emissions of machine learning. In NeurlPS 2019 Workshop on Tackling Climate Change with Machine Learning, 2019. URL https:/ /:::://www /wwww c www.iattteteeeeeeeeeeeeaeaee
neurips2019/22.
Nuno Luzia. simple proof of the strong law of large numbers with rates. Bulletin of the Australian Mathematical Society, 97(3):513-517 2018.
R. Thomas McCoy, Junghyun Min, and Tal Linzen. BERTs of a feather do not generalize to- gether: Large variability in generalization across models with similar test set performance. In Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Net- works for NLP. pp. 217-227 Online, November 2020. Association for Computational Lin- guisties. doi: 10. N..18853/v1/2222 1tt.://///000...aahooooopo 1.21. URL https: // //www ac web org/ anthology/ (ttpogy//20000 2200 b. www.lckkoooooocccoc 1..221
Tom McCoy, Ellie Pavlick, and Tal Linzen. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In Proceedings of the 57th Annual Meeting of the As- sociation for Computational Linguistics, pp. 1928-3448, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10. 10.18653/w 1/13//11PP111///1 1334. URL https: / /www/wwww...o aclweb. org/ htss/anthologyy/Pe htwphhhooooo/11--1111111
Marius Mosbach, Maksym Andriushchenko, and Dietrich Klakow. On the stability of fine- tuning BERT: Misconceptions, explanations, and strong baselines. In International Confer- ence on Learning Representations, 2021. URL /ttps//eee//eeeeii..e.m net meteeffrormmiio
nzpLWnVAyah.
12

---

Published as a conference paper at ICLR 2022
Moin Nadeem, Anna Bethke, and Siva Reddy. StereoSet: Measuring stereotypical bias in pretrained language models. In Proceedings of the 59th Annual Meeting of the Association for Computa- tional Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pp. 5356-5371, Online, August 2021. Association for Computational Linguisties. doi: 10.
URL https:
ogy. org/
2021. aelllacc-lons. 1ocl-long.44 416.
Aakanksha Naik, Abhilasha Ravichander, Norman Sadeh, Carolyn Rose, and Graham Neubig. Stress test evaluation for natural language inference. Proceedings of the 27th International Conference Computational Linguistics, pp. 2340-2353, Santa Fe, New Mexico, USA, Au- gust 2018. Association for Computational Linguistics. URL https: hwwp:///wwwwwwwbb..............b .rrg anthology/C18-111 /0
Kishore Papineni. Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic
evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Associa- tion for Computational Linguistics, pp. 311-318, Philadelphia, Pennsylvania, USA, July 2002.
Association for Computational Linguistics. doi: 10.3 10.3115/1073083.107333... URL https:
// /acllantholoyyyo..oooooooo wtg/p/2-1100 1040.
David Patterson, Joseph Gonzalez, Quoc Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David So, Maud Texier, and Jeff Dean. Carbon emissions and large neural network training. arXiv preprint arXiv: 10350, 2021.
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. Language models as knowledge bases? In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Confer- ence on Natural Language Processing (EMNLP-IJCNLP), pp. 2463-2473, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/1//D19.1 1250. URL https:/ www. www.wb. org/ wttphhooooy//19-12220
Maxime Peyrard, Wei Zhao, Steffen Eger, and Robert West. Better than average: Paired evaluation of NLP systems. In Proceedings of the 59th Annual Meeting of the Association for Computa- tional Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume Long Pupers), pp. 2301-2315, Online, August 2021. Association for Computational Linguistics. doi: 10. 22118633vv1/2222 acl-- 179. URL https: http://aaaaattooolgggooooooo org// 2021 aal- 179. 179.
Jason Phang, Thibault Févry, and Samuel R Bowman. Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks. arXiv preprint
2018.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference Empirical Methods in Natural Language Processing, pp. 2383-2392, Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10.18653/1/D11--11 1264. URL https: /wwww/ww...... aclweb.
org/ ggg/anthologyy/// /D16 1264.
Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't know: Unanswerable questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 784-789, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10. 1/11866//11/118-12222 URL https: // /aaclanthology.orret 110-1181121121
TA Ramasubban. The mean difference and the mean deviation of some discontinuous distributions. Biometrika, 45(3/4):549-556, 1958.
Eric Rieders. Marcinkiewicz-type strong laws for partially exchangeable arrays. Journal of Multi- variate Analysis, 38(1):114-140, 1991.
Anna Rogers, Olga Kovaleva, and Anna Rumshisky. A primer in BERTology: What we know about how BERT works. Transactions of the Association for Computational Linguistics, 8:842-866 866, 2020. doi: 16.1/62/ a 00349. URL https: :/t::///wwwww/// awwweb...w..b.o................. hwtp:///t..///aoogo///////////// tacl-1.55
13

---

Published as a conference paper at ICLR 2022
Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. Gender bias in coreference resolution, In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, New Orleans, Louisiana, June 2018. 8. Association for Computational Linguistics.
Naomi Saphra and Adam Lopez. Understanding learning dynamics of language models with SVCCA. In Proceedings of the 2019 Conference of the North American Chapter the Associa- tion for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 3257-3261 Minneapolis, Minnesota, June 2019. Association for Computational Lin- guistics. doi: 10. 10.18663/1/N11-1332)) URL http:: /aasttss:/aclanteen http://aaaaaaantn..ooooooo
1329.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, sentiment
and Christopher Potts. Recursive deep models for semantic compositionality treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1631-1642, Seattle, Washington, USA, October 2013. Association for Computa- tional Linguistics. URL https::////........ /ww..ww....... ac web. org/
Anders Søgaard, Anders Johannsen, Barbara Plank, Dirk Hovy, and Hector Martínez Alonso. What's in p-value in NLP? In Proceedings of the Eighteenth Conference Computational Natural Language Learning, pp. 1-10, Ann Arbor, Michigan, June 2014. Association for Com- putational Linguistics. doi: 10.3115/v1/W44- 1601. URL https: htt///aaaaaatttgyyy..aoa.a org/
W14-1601
Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in NLP. In Proceedings of the 57th Anmual Meeting of the Association for Com- putational Linguisties, pp. 3645-3650, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10. 10.18653/11/199111 1355. URL https: www. c web. org/ ant ogy/ P19-1355.
lan Tenney, Dipanjan Das, and Ellie Pavlick. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguisties, pp. 4593 Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/ v1/P19-1452. URL https: www. c web. org/ http//anthooooyy/oaoee /01-1////.1111121111111111111111
Julia Ture, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Well-read students learn better: On the importance of pre-training compact models. arXiv preprint arXiv:1908.08800 2019.
Aad W Van der Vaart. Asymptotic statistics, volume Cambridge University Press, 2000.
Aad W van der Vaart, Aad van der Vaart, Adrianus Willem van der Vaart, and Jon Wellner. Weak convergence and empirical processes: with applications to statistics. Springer Science & Busi- ness Media, 1996.
Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov, Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguisties, pp. Italy, July 2019. Association for Computational Linguistics. doi: 1111665311/119115555 URL /P1 https:/ www. acl web. www.brr.....eee.............e... rrgg/entholl ogy / 119 101191150 1580.
Florence,
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of International Conference on Learning Representations, 2019.
Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625-641, March 2019. doi: 10. 10.1162/tacLaa.00290 URL https: http:///aaaaaahhccooooo www.ooyy..o.roogooooooo .oo. 1 1040.
Kellie Webster, Xuezhi Wang, Ian Tenney, Alex Beutel, Emily Pitler, Ellie Pavlick, Jilin Chen, Ed Chi, and Slav Petrov. Measuring and reducing gendered correlations in pre-trained models. arXiv preprint 2011::2010.00032, 2020.
14

---

Published as a conference paper at ICLR 2022
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sen- tence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo- gies, Volume (Long Pupers), pp. 1112-1122 New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653/1//18--10000 URL https: / /wwwwww..... ac web.
/ //aaa/thhoooy/g188/ 1101.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowie7, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gug- ger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-an natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38 45, Online, October 2020. As- sociation for Computational Linguistics. URL https:://///.c /www www. acl wew........... org anthology/ 2020. smanp-ddemce 6.
Tianyi Zhang, Felix Wu, Arzoo Katiyar, Kilian Q Weinberger, and Yoav Artzi. Revisiting few- sample BERT fine-tuning. International Conference Learning Representations, 2021. URL
https://openrev///
net. hor.rrrrreeee.ettttooorrood----
Zhuosheng Zhang, Yuwei Wu, Hai Zhao, Zuchao Li, Shuailiang Zhang, Xi Zhou, and Xiang Zhou. Semantics-aware BERT for language understanding. Proceedings of the AAAI Conference on Artificial Intelligence, 34005)9628-9635, Apr. 2020, doi: 10.1609/aaii v34i05 v34105.6510. URL
attps:///jss aaai. org/ index.nix.. wwtdddex.pppppeeoomm AAA./attiteeeeaaaeaeeeeaeeeeeeea
6510.
Ruiqi Zhong, Dhruba Ghosh, Dan Klein, and Jacob Steinhardt. Are larger pretrained language models uniformly better? comparing performance the instance level. In Findings of the As- sociation for Computational Linguistics: ACL-IJCNLP 2021, pp. 3813-3827, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021 http:///222.....nnaaaieaee acl.34. URL https: / ac /at./aaaannntogyy....o...o....oa org/ 2021. findings-aci. 334.
Xiang Zhou, Yixin Nie, Hao Tan, and Mohit Bansal. The curse of performance instability in analysis datasets: Consequences, source, and suggestions. Proceedings of the 2020 Conference on Em- pirical Methods in Natural Language Processing (EMNLP), pp. 2215-8228, Online, November 2020. Association for Computational Linguistics. doi: 10. 36633/v12020.emm/// main.659. URL https: a antho ogy- - org/ 2020. emnlp-ma i 65
Y. Zhu, R. Kiros, R. Zemel, R. Salakhutdinov, R. Urtasun, A. Torralba, and S. Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In 2015 IEEE International Conference on Computer Vision (ICCV), pp. 19-27, Dec 2015. doi:
10.1109/1CC222055.11.
15

---

Published as a conference paper at ICLR 2022
A MINIMAL IMPLEMENTATION OF THE MULTI-BOOTSTRAP
Below, we present a simplified Python implementation of the Multi-Bootstrap algorithm presented in Section 3.2. It describes single-sided version the procedure, which could be used, e-g., to test that model's performance is greater than O. The input is matrix of predictions where row indices correspond to test examples and column indices random seeds. The functions returns an array of Nooot samples [o...... nssool
def mult wiccostttrrpppppo. (predi ctions, labels, retric_fun, nboct)
Generates bootstrap samples of model's performance.
Input:
predictions: 2D Numpy array with the predictions for different seeds. labels: 1D Numpy array with the labels. metric_fun: Python function. Takes a pair of arrays input, and returns metric or loss.
nboot: Number of bootstrap samples to generate.
Output:
12
Numpy array with nboot samples.
13
14
# Checks the data format.
n_samples, n_seeds predi ons shape assert labels. shape (n samples,
17
18
thetas = np. zeros (nboot)
for boot_ix in range (nboot):
samples n_samples test examples and n_seeds pre-training seeds.
22
x_smpless np. random, choice (n_samples, size=n_samples, replace-True) s samples random, craicc (n_seedss size-n_seeds, replace=True)
23
Computes the metric over the bootstrapping samples. predictiens Inp. ix_ (x_samples, s_samples) [ampled_labele labels ( (x_samples)
26
27
28
sampled_metrie metrics = metric_fun (sampled wet_ppredccionnn jl, sampled_labes labels) for in range (n_seeds)
31
]
32
Averages over the random seeds. thetas [boot_ix np (sampled metrics)
35
36
return thetas
We provide the complete version of the algorithm on our repository http://get -l gle/ mu multiberts. Our implementation optimized and supports all the experiment designs described in Section 3, including paired and unpaired analysis well multiple fine-tuning runs for each pretraining seed.
16

---

Published as a conference paper at ICLR 2022
B PROOF OF THEOREM
Before giving the proof, we define some useful notation that will simplify the argument considerably. We let Dr be the empirical measure over the ny observations (Z₁ = (X₁, Y.)) and M, be the empirical measure over the n, observations (S,) For a function f V - and a distribution
P over V, we will use the shorthand Pf to denote the expectation of f under P,
Pf Bv-P)((V)).
For example, this allows us to write
T2 Zi, J8y).
DM En-nn (Z,ss) and 0 = Dr
Th
f=1
For the bootstrapped distributions, let D: denote the distribution over the bootstrap data sam- ples (71, 7/ Zn,)) and Mr denote the distribution over the bootstrapped seed samples, (ST. S2: .,, 5x ) both conditional on the observed samples (Z:) and (Sy) Note that the empirical average over bootstrapped sample
fs;)
T2
can be written as
na The 1 n1
A,B, The
where A, is the number of times Zi appears in the bootstrapped sample (Z))== and By is the number of times Sj appears in the bootstrapped sample (Si) With this in mind, we will abuse notation, and also denote D as the distribution over the A, and M, as the distribution over the By. Finally, we will use E' and Var to denote the expectation and variance of random variables defined with respect to D or Mt conditional on D,, and Mn-
We will use P to denote the distribution P = D x M. Throughout, all assertions made with respect to random variables made without note about their probability of occurrence hold P-almost surely.
Proof. The challenge with applying existing theory to our method is that because the performance metric (E(Z,, f over the n, observations for a given seed S, all depend on the same Sj. they not independent. Similarly for the performance we need to handle this non-iid structure in our proof for the mulli-bootstrap.
given observation, over seeds. Therefore,
There are conceptually three steps our proof that allow to do just that. The first is to show that i has an asymptotically linear representation as
vn(O 0) = Vn(D,, DYME+ Vn(MM - M)DE + op(().
(2)
The second is to show that conditional on Dn, and M,, the multi-bootstrapped statistic 0- - D M has an asymptotically linear representation as
Vn 0) vn(D; - D.)Me + VM(M, -
where D and M are multiplier bootstrap samples coupled to the bootstrap D and M which we define formally in the beginning of Step The third step use standard results for the multiplier bootstrap of the mean of iid data to show that the distributions of the above linearized statistics converge to the same limit.
Because we have assumed that fs) < co, fs) S < oc, and E[e (l) fs) 2| < co,
Fubini's theorem allows us to switch the order of integration over Z and S needed.
We will assume that DM subtracting ML to the bootstrap expression gives
0. This is without loss of generality, because adding and
Vm 0)
Vi(D; M,
=VTDD,M, DDM--MM++ DME D,.,,)
=Vn(D,M,, DME) D,,,,(E -DM)),
17

---

Published as a conference paper at ICLR 2022
so if we prove that the result holds with the mean zero assumption, it will imply that the result holds for with a nonzero mean.
This theorem guarantees consistency of the Multi-Bootsiin estimates. One question that comes up is whether possible to get meaningful / tight rates of convergence for the approximation. Unfortunately, getting Op(1/1 convergence found in many bootstrap methods (Van der Vaart, 2000) is difficult without the use of Edgeworth expansions, by which the Multi- Multi--oottstra is not well- adapted analysis. That said, many of the remainder terms already have variance of order O(1/n), or could easily be adapted to the same, suggesting an Op(1/ vn) convergence, The main difficulty, however, is showing rates of convergence for the strong law on separately exchangeable arrays (see the proof of Lemmas 2, 4-5). Showing weaker notion of convergence, such as in probability, may perhaps allow one show that the remainder is Op(1/vn), however the adaptation of the aforementioned Lemmas is nontrivial.
Recalling that 0 A Dn e and A a DME, expand vn( f) as follows,
Step 1
VT VDnDaM CAMM,-DMML DME) - vT((D, &M,L + DM,L - DME)
-Vnl(Dn D)M,E+ + D(M, - M)))
-Vr((Dn D)M, + +DDn - -D)M - -(Dn- (Dn-D)ME D(M, M)0)
=VT((Dn D)Me+ (Dn DM - -M) + D(M, Mye)
The following lemma shows that Vn(DD, - DD,M, - M) is lower order term.
Lemma 1. Under the assumptions of Theorem 1, VT((Dn - D)(M, - MYe - op().
Therefore,
1 M.DDMML-DMM V1 Ps
(Tn - D)Me= 1 VT.(MM M)De+ VPs
One of the challenges with working with the bootstrap sample D and Mn is that the induced per-sample weights {A and }= do not have independent components, because they each follow a multinomial distribution over fgg items and n, items, respectively. However, they are close enough to independent that we can define a coupled set of random variables ( )- and {B that do have independent components, but behave similarly enough to {A:} and [B, J that using these weights has a negligible effect on distribution of the bootstrapped estimator, as described concretely below.
Step 2
First, we discuss the coupled multiplier bootstrap sample D; and M;. The creation of this sequence, called "Poissonization" is standard technique for proving results about the empirical bootstrap that require independence of the bootstrap weights (van der Vaart et al., 1996). We describe this for n as the idea is identical for M,. Because our goal is to couple this distribution D; we define on the same sample space, and extend the distribution p*, expectation E and variance Var" to be over Da and M.. conditionally on D,, and M,, as with D and M,,
To construct the distribution De from the empirical distribution D, and a bootstrap sample D start with the distribution D; and modify it as follows: We draw Poisson random variable N with mean nz. If Nn_ > Rz. then we sample N,, - Rr iid observations from D,,, with replacement, and add them to the bootstrap sample initialized with D to produce the distribution De If No, < Nay we sample nz Nr., observations from D",, without replacement, and remove them from the bootstrap sample to produce the distribution D If Nn A; is the number of times the i-th sample is included in D; similarly define Ai as the number of times the i-th sample is included in D;- Note that by the properties of the Poisson distribution, AJ Poisson(1), and {Ap] 1 independent. Note that the natural normalization for D; would be Nn.. However, it will be useful to maintain the normalization by ny, so abusing notation, for a function f(z), we will say that D
n2. then Do D Recalling that
(Z,))
7h 21=1
18

---

Published as a conference paper at ICLR 2022
Define 8 as the following empirical estimator of f under the distribution D x MR.
0- DrM
Ass.)
The
Lemma 2 shows that vn 0-) - op- (1), and so vn(00- 0) vn((0- 00) + Op-
Lemma 2. Under the assumptions of Theorem I, and that DMC O, vn(0" 00) op- (1).
With this, the expansion of vn(00 0) begins mutatis mutandis the same as in Step 1, get that
1
vn(O - 0)
(D - D,)M,l++ vo(Da Da) M,)e
VI
Ps
(M M.)D..e
Ps
As with Step 1, we provide Lemma 3 showing that the remainder term vn(D; DM) - M.)e will be lower order.
Lemma 3. Under the assumptions of Theorem I, vn(Da - (M, - M,) -1-
Therefore,
1 VI- Ps
Vn(D, D,,,n)-
n Dr Mn)
(M M,)Dnl+ (1).
VPa
Then, to write - in terms of Vig (M; -M;-M,, ( DE as wanted in Eq. (3), instead of VTg (M; D,L,,,L, we must additionally show that the functional has enough continuity that the error term (M, M,)(D,, D) lower order. The following lemma shows exactly this.
Lemma 4. Under the assumptions of Theorem conditionally the sequences Zi, Zz.... and S₁ S2.S.....
(a) vn(D,, - Dn)(M, M)e = op- (1), and
(b) vn(D, D)(ME - -M,)e 0p-(1).
Altogether, these imply that
(D, - DD)Me+ (M, M,) op- (1). Vn(D;,Me D,M,f)= VT - ps
vps
Step Noting that ME(.,f [ finite variance (similarly for DE(Z, -)), and recalling that the n, samples used for D,, and n, samples for M,, satisfy n n11(1 Ps) and = n. ns/P.:: for < Ps 1, the conventional central limit theorem shows that for some positive semi-definite matrix Σ 2x2 and G ~ (O0, Σ),
real-valued random variable with
(Dn (Dx DDaDe Vn (M -M,- MyDe ) )
( (D, D)Me T.P (
) SG.
P. VM, M)DE
Note that D, and M,, are independent, so is, in fact, a diagonal matrix.
Additionally, the conditional multiplier CLT (van der Vaart et al., 1996, Lemma 2.9.5, pg. 181) implies that conditionally on Z1, Zz.... and S1, S2....,
(D: Dn)ME (M; - M, ) S.G.
Vnl
Finally, applying the delta method (see Theorem 23.5 from Van der Vaart (2000)) along with the results from Steps and 2 shows that the distributions of /n(0 0) and vn(00 0) converge to (0, o?), where o² En/(1 ps) + Eza/Ps.
19

---

Published as a conference paper at ICLR 2022
B.1 PROOF OF LEMMA 1
Fix e > O. Note that E((D, - D) - M) = s0 by Chebyshev's inequality,
P (IVTTD, - D) MM) > c) Varlyn(DD (),,, M)
6²
Therefore, it suffices to show that 7 Var( Var(vn(D, D,, M)) O. To do so, apply the law of total variance, conditioning on Dn, and bound the resulting expression by C/n.
Var( (D - (M,, - M) = nEIVar (Dn, (M,, M)e D.)l
+ nnVar(B((DD,
DM, - MMe D.I))
nEI[aar( (Dn D)(M, - -M)e D.))
nIR[Var((M,, M) -DJ D.))
E
Var((nn Ds.(: D..)
n
E
n
Var Var((Dn - D)e(: fs.) D.))
n2
1
E
E
- ((2, - Eie(Z, sil
11
Ps
71.
i=1
27
E
(Z, - Es)
Ps
1=1
fs) E(((,, -1) S:))
P&T
E
(e(Zis E(E( |1]))
P&n
E (((Z), - { < C 0.
Pa Paym
B.2 PROOF OF LEMMA 2
First, note the following representation for 0- - -
1
1
ng 1
00
A,B,((Zi,,s)
Thr
Th.
This
j=1
i=1
j=1
(B)
B
(A,
AP)
n
A,E(Ze, fsy)
ByE(Z:: fs,).
n2
n2
n
j=l
1
Let >> O. Noting that E-[4]]
EE" [12] 0, applying Chebyshev's inequality gives
Var (0- -0)
Var" (I1 + Var (Iz)
Pa (vr - Øo E)
2n
2²
2²
The arguments for each term are mutatis mutandis the same, and so we proceed by showing the proof for Iz.
suffices show that n Var" (41 - and Var (Iz)
By the law of total variance.
Var (72) Var Var'(E"(Ta (B)) + E [Var* VTa (B, )=
20

---

Published as a conference paper at ICLR 2022
Because [A,], [A[4]] and {B 1 A,, A. it follows that E [By])=1] 0. Taking the remaining term and re-organizing the sums in 1z.
|
Var Var"(I2) E"
Var*
Σ (A T2e
1
nr Ai)
1 Σ
Bs,
(4)
T j=1
i=1
Next, we apply the law of total variance again, conditioning on Nu = E. First,
N
1
Re
E [/2 No.. [B,]==1
Byl(Zi:sss)
f1
The
1hss
f=1
and
Var (E'(Iz) Nu.. (Bi]fsal {B,}}=]]
Byl(Zz fs,)
Then, conditionally on Nr, (and {B,}). I2 is the (centered) empirical average N,, n| samples from finite population of size n, rescaled by IN, ml/n.. Therefore, applying Theorem 2.2 of Cochran (2007) gives the conditional variance as
IN nz
1 aa
By Bfl(Zi.s.) 1=1
Bye(Z fs.)
n2
n1
11
1h
i=1
To take the expectation over N notice that because E* [NN = Rr,, this is the mean absolute de- viation (MAD) of N,,- Using the expression for the MAD of a Poisson variable from Ramasubban (1958) gives
nh cxp(-nx)) nz))
E N nzl 2n
ng!
and using Stirling's approximation, this is bounded by C Cy for some 0 < C < xc.
Combining this with the above term for the variance of the conditional expectation, have
Var*
Σ (A: 19) 1=1
B Ss:)
(B,)
Byl( (g8(Z4, Jss)
Thg
11g
j=1
1=1
V2
(5)
L.5
Noting that EE* [B3] [B, 1. get the following bound:
Var"(T2)
Var
<
El (Z,,5s))
+ 1. p2
nr
Th. i=1
The
j=1
where
2
2
1
E( ((Z,, fsy)
nl.
1
1
Zz. fs,)
The
P2
1
n2
n
iml
Because of the assumption that DMC 0, the SLLN adapted separately exchangeable arrays (Rieders, 1991, Theorem 1.4) implies that
The
lim
Σ
((Zr.ss)
21

---

Published as a conference paper at ICLR 2022
almost surely. Therefore, the first term of (5) is o(1/n).
Note that 2 the empirical variance of the conditional expectation E(Z,, fs,) given {2,1 Therefore, the law of total variance shows that
V2 2
(Zi. - T1a
(Zis fs,)
1=
By the SLLN adapted to separately exchangeable arrays (Rieders, 1991, Theorem 1.4), both of the terms converge almost surely to DMC 00 and (DME)². respectively. and therefore,
2
Rx
lim nVar' (T, & lim n n-000
(Z,,
Ze Js,)
12
T2
Thg
n 155
B.3 PROOF OF LEMMA 3
As with Lemma I, the main idea of the proof is to apply Chebyshev's inequality, and show that the variance tends to zero. Indeed, choosing arbitrary O.
P (IvR(D) (M 2 Var (Vn(D; (M - M.,e)
Therefore, suffices to show that the variance in the above display goes zero. To do this, we start by re-writing the expression in terms of Aï and Bg. and then apply the law of total variance.
1
Var (Vr(DD (M) - Mn)e) = Vaa
() - 1),(2, fs,)) (A.
T2
1=1 ==
= n Var
E
(4 - 1)(B) 1)((Z,,fs)) (AP1
ny
i=
R2
The
+ nF
Var
- 1)((,, ((4)))
ng
j=1
Because {B are independent of {.4. 1- and have mean 1, the conditional expectation in the first term is 0 almost surely. Expanding out the second term, using that Var (Bj) 1, and that the {B are uncorrelated.
n.
Var*
(A - 1)(3) 1)8(2,,);;) (1,
n.
1=1 1=1
1 n,
(B) - 1) 1 Σ
1 nr
nE*
Var
(A 1),(Z,,fs,)
14 14
112
72 i=1
j=1
27
1 7,
1 n
nE*
(A 1)((Zs.fs))
11
Thg
Thp
=n
(A - 1)(A 1)8(Z,, (s
22
22

---

Published as a conference paper at ICLR 2022
Now, noting that Var" (Ag) 1,, and that the (41) are uncorrelated, this simplifies to
R2
nE
Σ (Ai - (2,
(Z, fs,).
T i=1
Teg Rog
Because D M/ (Z, fs) oc, the SLLN adapted separately exchangeable arrays (Rieders, 1991, Theorem 1.4) implies that this converges almost surely to O.
B.4 PROOF OF LEMMA
We prove (a) of the Lemma, as (b) follows from applying Fubini's theorem and following mutatis mutandis the same argument. Without loss of generality, we will assume that E(Z,, fs,) 2 0. Because Var( Z,, fs,)) )) < oc,
always decompose ((:,-) into positive and negative part, and show that the result holds for each individually.
Once again, prove (a) by turning to Chebyshev's inequality. Fix > 0, and observe that
Pa (IVn(D; - D.)(M, M)e > 2) Var (Vn(DR - (A) - M))
e2
so it is sufficient to show that Var (Vn(D2 - -N) M)) + Writing the above in terms of AG, have
Var (Vn(Dn, - (M,, - M))
Vπ Σ
1 Σ
= Var*
(Ag -1)
(s,) -11((,,,ss) 21]
T1e i=1
11 j=1
n2
Var (A--1)
e(Zi, fs,) Ble(Z,,fs,) 24]
n2
11g j=1
2
71
1 ns
-B) fs,) Zd]
1=1
Now, we want show that the last display converges almost surely to O. Notice that each term within the outer sum will obviously converge due to the SLLN. Showing that the outer sum also converges almost surely is technically difficult, but conceptually follows the same argument used to prove the SLLN (specifically, we follow the one done elegantly by Etemadi (1981); Luzia (2018) provides a more detailed account of this proof technique that is helpful for developing deeper understanding).
We show the following version of almost sure convergence: that for any > 0.
P
n
E(Zi, fs,) -I((Z,, 1/1
> >ci. = =0,
where i.o. stands for infinitely often.
Define the shorthand Liş = eZZi, Js,) and let Liş = Liglthi < 1/} be a truncated version of Lig. The proof of Theorem 2 of Etemadi (1981) implies that P(Ly * Lig i..)) = 0, because the implies the assumption used in Etemadi (1981), and independence of (Lay)is Ji,j is not needed for this result. Therefore,
assumption Var(L,;)
1
Σ
Lij
Lig
0, and Thx t=1
E[L Z1 -E
Zz]
2 0.
na
nu j=1
t=1
==1
23

---

Published as a conference paper at ICLR 2022
Together, these imply that if we prove that the truncated sum converges, ie.,
2
n2 Σ E[L
Li -
Zz]
- O,
(6)
Th j=1
this is sufficient to show that the un-truncated version converges almost surely.
To prove (6), we show two things: first, that there is a subsequence kn such that (6) holds when restricted to the subsequence, and then we show that the sequence is a Cauchy sequence, which together imply the result.
Let a > and let ki a". For convenience, denote kn as the number of data samples and kns as the number of seed samples when knn + Knn kn total samples are drawn. We will ignore integer rounding issues, and assume kiz = (1 - P. Ja" and kn, - ppa".
The following lemma shows that the subsequence defined by kn converges almost surely. Lemma 5. Let a > 1, and kn = =aa Under the ussumptions of Theorem I and that Lij 2
2
kas
Σ j=1
Lis
EE(Ls
Zıl
> cio =0.
Ki me i=1
We now must show that the sequence (6) is a Cauchy sequence. Note that the SLLN implies that
z $ [[E [21²1,
Th
and the LLN for exchangeable arrays (Rieders, 1991, Theorem 1.4) implies that
EL
Zd *
E[E[ 1.1...
1=1
1=1
Therefore,
2
knz
Lie
a.s.
E Lij 2.17).
Rin Kits
2
Notice that because Lig 2 O, the sum En=
Ee= Lij
is monotone increasing in n, and ng-
With this in mind, for any m O, let be such that kin m knt1. Then, by the montonicity,
2
(1-p.)m
EEpET Lij)
kinnt)o
k(n-
2
kin
Lis
==1
kut1 kin
Σ
Lis
1ip+1
lin
pº11 po)m3
kn+1
Ø=1
From (7), the left hand side converges to 1 [Li a ['E[E[Las 2. Because a is arbitrary, this proves that the sequence
7.1%, and the right hand side converges to
Σ E (1-p.lm p. Lnj
12(1 Pa)m
almost surely Cauchy. Together with Lemma 5, this implies (6).
B.5 PROOF OF LEMMA 5
We will show that
RnD Laj
P
B[
Zi]
>
< 0c.
Knek
2
R=1
j=1
24

---

Published as a conference paper at ICLR 2022
This, along with the first Borel-Cantelli lemma (Émile Borel, 1909; Cantelli, 1917) implies the result.
Applying Markov's inequality and using the fact that Lag and Lin are independent conditional on Z, gives
2
oc
Kin
Rnz
P
Lij - B[Les [s]
>6
knohins nos
7
27
oc
Rone
Nince
Σ 5iml
Lis -B(Lis [i]
kin nis
Rin kina
(Eis - EiLss 2,1)
king ns
n=1
im j=1
0c
Rir kine
E[L
kingk
n=l
1=1
where the last line follows from the law of total variance. To simplify the remaining algebra, we will use a S b to denote that there is some constant 0 < c < oo such that a < cb. Continuing, we have
Run
Kug
bus
B[Tig]
1=
2= 1=1
n=1
E[L
2n
n=
20
max (i/(1 - //D. E[L3g1
max (1, 1, B[[2j]]
1=1
max B[L
where (i) is shorthand for ni,, = log,, maxfi/(1-pa)) (.pp).j/ppp. is the first n such that kuz 2 i and kns 2 j.
Now, define Q as the distribution of Ln induced by Z1 and Sj. Additionally, split the inner sum into two pieces, one for when j < and max(i, i and one for when j and max(i,j) j-
20
8 2 dQ(x)
x2 (1
max max 33 |LL3311
Σ
dQ +
x² dQ(x))
/=
25

---

Published as a conference paper at ICLR 2022
switching the order of the indices over and k, using that S h S ij and the constraints on relative to i.
dQ()))
ΣΣ dd((z)
k/i)
*
x dQ(()) +
x² dQ(x)
1
k-1
k= j=max ft/k/1)
jo k-1
k/i)
x² dQ(()) +
Σ
x2 PdQ(x)
13
max (i, k/ij²
k-1
k=1
Switching the order of summation over i and k, and separating out the terms where k/i < i and k/i i,
k/i)
x² dQ(x)) +
a² dQ()))
max ki,k/ A-1
x² PQ(()
Vk+1
k/i)
Σ
kー
x² (dQ(a))
vk
b-1
x²
dQ(z))
k-1 v
1.5 dQ(xx) 00.
26

---

Published as a conference paper at ICLR 2022
C INSTANCE-LEVEL AGREEMENT OF MULTIBERTS ON GLUE
present additional performance experiments to complement Section
Table shows per-example agreement rates GLUE predictions between pairs of models pre- trained with a single seed ("same") and pairs pre-trained with different seeds ("diff"); in all cases, models fine-tuned with different seeds. With the exception of RTE, see high agreement (over 90%) on test examples drawn from the same distribution as the training data, and note that agreement is 1-2% lower average for the predictions of models pre-trained on different seeds compared to models pre-trained on the same seed. However, this discrepancy becomes significantly more pro- nounced if look out-of-domain "challenge sets" which feature different data distribution from the training set. For example, if we evaluate our MNLI models on the anti-sterotypical examples from HANS (McCoy al., 2019), we see agreement drop from 88% 82% when comparing across pre-trainnn training seeds. Figure 4 shows how this can affect overall accuracy, which can vary over range of nearly 20% depending on the pre-training seed. Such results underscore the need evaluate multiple pre-training runs, especially when evaluating a model's ability to generalize outside of its training distribution.
pre-
Same
Diff.
Same
Diff.
CoLA
91.5%
89.7%
1.7%
MNLI
93.6%
90.1%
3.5%
HANS (all)
92.2%
88. 7%
4. %
HANS (neg)
88.3%
81.9%
6.4%
MRPC
91.7%
90.4%
1.3%
QNLI
95.0%
93.2%
1.9%
QQP
95.0%
94.1%
0.9%
RTE
74.3%
73.0%
1.3%
SST-2
97.1%
95.6%
1.4%
STS-B
97.6%
96.2%
1.4%
Table 3: Average per-example agreement between model predictions each task. This is computed as the average "accuracy" between the predictions of two runs for classification tasks, or Pearson correlation for regression (STS-B). B). We separate pairs of models that the pre-training seed but different fine-tuning seeds (Same) and pairs that differ both in their pre-training and fine-tuning seeds (Diff). HANS (neg) refers to only the anti-stereotypical examples (non-entailmennt), which exhibit significant variability between models (McCoy et al., 2020).
0.5
Accuracy (non-entalle led examples)
0.4
0.3
0.0
2
3
7
8
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
Pretraining
Figure 4: Accuracy of MNLI models on the anti-stereotypical (non-entailment) examples from HANS (McCoy et al., 2020), grouped by pre-training seed. Each column shows the distribution of five fine-tuning runs based the same initial checkpoint.
27

---

Published as a conference paper at ICLR 2022
Winogender bias correlation (r) by pretrain seed
Base
Bias carrelation
0.2
0.0
-0.2
0
1
3
4
5
7
8
10 11
12
13
14
15
16
17
18
19
20
21
22
23
24
Pretraining
Figure 5: Bias correlation Winogender for each pre-training seed. Each box represents the dis- tribution of the score over five training runs of the coreference model over each MultiBERTS base checkpoint. This is the same data Figure 3, but showing only the base checkpoints.
Bles variation by pretrain seed, base wiextra seeds
0.8
Bias correlation (n)
0.6
0.4
0.2
Bias (r)
Seed 0
0
0.368
Seed
0,
0.571
Avg. Diff.
8 8= 0g 0f
0.203
-0.2
p-value
0.009
Pretreining
Figure 6: Bias correlation on Winogender for five pretraining seeds, with 25 coreference runs seed.
Table 4: Unpaired
on Wino-
gender bias correlation, comparing pretraining seed ( to pretraining seed 1.
D CROSS-SEED VARIATION
Figure shows variation in Winogender bias correlation (S4) between each MultiBERTS pretraining seed. Each box shows the distribution over five runs, and some of the variation between seeds may simple be due variation in training the coreference model. then look at the distribution of this per-seed average score, we 0.4 What if pretraining didn't matter? we ignore the seed and randomly sample sets of five runs from this set with replacement, get scores of 0.45+00.05 telling us that most of the variance only be explained by differences between the pretraining checkpoints.
average the scores for each seed
We can confirm this by taking subset of our pretraining seeds and training additional 25 randomly- initialized coreference models. Figure 6 shows the result: seeds 0, 2, 3, and appear closer together Winogender metric.
than in Figure 5, but seed clearly has different properties with respect We can confirm this with an unpaired multibootstrap analysis, taking seed 0 as base and seed I as experiment: observe significant effect of å 0.203 0.009), shown in Table 4.
28

---

Published as a conference paper at ICLR 2022
MNLI
RTE
MRPC
9 (IM steps)
0.837
0.644
0.861
0r (2M steps)
0.844
0.655
0.860
8 8= 0r 0g
0.007
0.011
-0.001
p-value
(Hp that 0)
0.000
0.141
0.564
Table 5: Expected scores (accuracy), effect sizes, and p-values from Multi-Bootstrap selected GLUE tasks. We pre-select the best fine-tuning learning rate by averaging over runs; this 3e-5 for checkpoints at IM steps, and 2e-5 for checkpoints at 2M pre-training steps. All tests use 1000 bootstrap samples, in paired mode on the five seeds for which both IM and 2M steps are available.
E CASE STUDY: MULTIBERTS vs. ORIGINAL BERT
As an additional example of application, we discuss challenges in reproducing the performance of the original BERT checkpoint, using the Multi- Multi-Bootstrap procedure.
The original bert-basee bat--baae-uncased checkpoint appears to be an outlier when viewed against the distribution of scores obtained using the MultiBERTs reproductions. Specifically, in reproducing the training recipe of Devlin et al. (2019), we found it difficult to simultaneously match performance of hyperparameters.. Devlin et al. (2019) reports training for 1M
on all tasks using single steps. However, as shown in Figure and 2, models pre-trained for 1M steps matched the original checkpoint on SQuAD but lagged behind on GLUE tasks; if pre-training continues to 2M steps, GLUE performance matches the original checkpoint but SQuAD performance is significantly higher.
The above observations suggest two separate but related hypotheses (below) about the BERT pre- training procedure.
On most tasks, running BERT pre-training for 2M steps produces better models than IM steps.
2. The MultiBERTS training procedure outperforms the original BERT procedure on SQuAD.
Let us use the Multi-Boun Bolti-Bootttrap to test these hypotheses.
E. HOw MANY STEPS PRETRAIN?
Let I be the predictor induced by the BERT pre-training procedure using the default 1M steps, and let f' be the predictor resulting from the proposed intervention of training to 2M steps. From glance at the histograms in Figure 8, can that MNLI appears to be a case where 2M is generally better, while MRPC and RTE appear less conclusive. With the MultiBERTS, we can test the significance of the results. The results are shown in Table 5. We find that MNLI conclusively performs better (ō 0.007 with < 0.001) with 2M steps; for RTE and MRPC we cannot reject the null hypothesis of no difference (p- 0.14 and 0.56 respectively).
As an example of the utility of this procedure, Figure 7 shows the distribution of individual samples of L for the intervention f' and baseline from this bootstrap procedure (which we denote as L' and L. respectively). The distributions overlap significantly, but the samples are highly correlated due to the paired sampling, and we find that individual samples of the difference (L' L) are nearly always positive.
E.2 DOES THE MULTIBERTS PROCEDURE OUTPERFORM ORIGINAL BERT ON SQUAD?
To test our second hypothesis, i.c., that the MultiBERTS procedure outperforms original BERT on SQuAD, must use the unpaired Multi-Bootstrap procedure. In particular, we are limited to the case in which we only have point estimate of L'SS), because we only have single estimate of the performance of our baseline model f' (the original BERT checkpoint). However, the Multi- Bootstrap procedure still allows us to estimate variance across our MultiBERTS seeds and across the examples in the evaluation set. On SQuAD 2.0, we find that the MultiBERT models trained for 2M
29

---

Published as a conference paper at ICLR 2022
steps outperform original BERT with 95% confidence range of 1.9% to 2.9% and p 0.001 for the null hypothesis, corroborating our intuition from Figure 2.
0.855
0,012
0,010
0.835
0.838
Figure 7: Distribution of estimated performance on MNLI across bootstrap samples, for runs with 1M 2M steps. Individual samples of T(S,, (X, Y)) and ('(S, (X, (X,Y)) the left, deltas L',S, (x, Y)) L(S, (x, Y)) shown on the right. Bootstrap experiment is run as in Table 5, which gives 8 0.007 with p 0.001.
CoLA (acc)
MNLI (acc)
10
0.835
0.840
0.845
MRPC (acc)
QNLI (acc)
10
0.80
0.82
0.84
0.86
0.88
0.905
0.910
0.955
OOP (acc)
RTC (acc)
100
SST-2 (acc)
STS-B (r)
0.910 0.913 0.920
0.925
0.880 0.805 0.890
0.895
Figure 8: Distribution of the performance GLUE dev sets, showing only runs with the best selected learning rate for each task. Each plot shows 25 points (5 fne-tuning x 5 pre-training) for each of the IM and 2M-step versions of each of the pre-training for which release intermediate checkpoints ($2).
30


Reviews:
[{"review_id": "jATwwhSnQ1e", "reviewer": "Reviewer_H2Mp", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper presents an interesting approach, a novel contribution that might have some applications. From the methodological point of view, it is well-written, there is a good review of related works, a description of the MultiBERTs release, and an application to reduce a type of gender bias in coreference systems built on BERT. There are also comprehensive supplementary materials with proofs of theorems and lemmas. It seems to be a novel approach, it's hard to assess its significance but the presented application suggests there the impact might be important. If the article is accepted, there are some typos that should be fixed before publication:\n- p. 2: seeks to to analyze -> seeks to analyze\n- p. 2:  the uncertainty the test sample ->  the uncertainty of the test sample\n- p. 6: it looks that in one case, to mark a distribution, D, is used, in another case, it is P\n- p. 8: statiatically -> statistically\n\n\n\n\n", "summary_of_the_paper": "The paper presents MultiBERTs, a set of 25 model checkpoints to support robust research on BERT, and the Multi-Bootstrap, a non-parametric statistical method to estimate the uncertainty of model comparisons across multiple training seeds. It demonstrates the utility of these resources by showing how to quantify the effect of an intervention to reduce a type of gender bias in coreference systems built on BERT. ", "summary_of_the_review": "The paper presents an interesting approach, a novel contribution that might have some applications. From the methodological point of view, it is well-written, there is a good review of related works, a description of the MultiBERTs release, and an application to reduce a type of gender bias in coreference systems built on BERT. There are also comprehensive supplementary materials with proofs of theorems and lemmas. It seems to be a novel approach, it's hard to assess its significance but the presented application suggests that the impact might be important. Therefore, from the methodological point of view, the paper is marginally above the acceptance threshold, but if it is accepted, there are some minor typos that should be fixed before publication.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "J8cRdOa0fIQ", "reviewer": "Reviewer_KA6v", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper presents MultiBERTs, a set of 25 model checkpoints, and the Multi-Bootstrap, a non-parametric method to estimate the model uncertainty. The experiments verified the proposed method on a case study of gender bias in coreference resolution. \n\nPaper Strengths:\n- It is great to see the models and statistical library are available online (165 checkpoints). I appreciate the authors can provide the CO2 information on model training, which is environment friendly.\n- This is an empirical analysis paper, which is useful for the community but the computational cost seems very high. Pretrained model has been widely used and show impressive performance. The method in this paper is novel and theoretically sound, which is helpful for understanding the large models.\n- The proposed method is rigorous in terms of mathematics and statistics. Even though the techniques are simple, but they can be categorized into three designs and formulated as a formal problem.\n ", "summary_of_the_paper": "The paper presents MultiBERTs, a set of 25 model checkpoints, and the Multi-Bootstrap, a non-parametric method to estimate the model uncertainty. The experiments verified the proposed method on a case study of gender bias in coreference resolution. ", "summary_of_the_review": "This paper is different from classical analysis paper in NLP. The authors claim to make progress on language model pre-training and it is essential to distinguish between the properties of specific model artifacts and those of the training procedures that generated them. I can buy this claim to some degree but still reserve my recommendation for acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "5MywpyGPUdz", "reviewer": "Reviewer_LeNr", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The paper provides various pre-trained BERT models with many checkpoints, which could enable future researches on reproducibility and robustness.\n- The proposed Multi-Bootstrap procedure could give a reasonable estimate of the distribution of the error over the seeds and test instances, under both paired and unpaired scenarios.\n- The provided code for Multi-Bootstrap is easy to understand.\n\nOverall, I feel it is still expensive for common NLP tasks to adopt MultiBERTs and MultiBootStrap methods to draw conclusions, which may hinder it from a wide range of applications.", "summary_of_the_paper": "This paper releases MultiBERTs, a set of 25 BERT base checkpoints to facilitate studies of robustness to parameter initialization and order of training examples. It also proposes the Multi-Bootstrap method to quantify the uncertainty of experimental results based on multiple pre-training seeds.", "summary_of_the_review": "The paper provides adequate models and a good estimation method for future researches on robustness.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "-Rx_BptTm4l", "reviewer": "Reviewer_67rn", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors address the question of using artifactual large language models as the basis of experimentation in NLP research.  In particular, they examine the extent to which results of finetuning or transfer learning can be attributed to the use a single best checkpoint of drawn from a distribution over training data, loss, random seed, and architecture, or  for common building blocks (here they consider BERT).\n\nStrengths:\n\n- The biggest strength of the work as I read it is the multi-bootstrap.  This is the most likely element to be reused in future work\n- I commend the openness of the authors; their code, their checkpoints, and the clarity of their writing made this paper a pleasure to review\n- The inclusion of a case study in section 4 brought the whole of the contributions together, and highlighted the value of both multiple independent samples as well as the bootstrap framework used to assess the contribution of different possible choices.  \n\nWeaknesses:\n\n- I hope that the resources of the MultiBERT models will be reused in NLP research, but fear that since LLM research is progressing at a frenetic pace, BERT is no longer as relevant as it once was even a mere 12 months ago, being supplanted on several tasks by prompt-based language models (e.g T5, GPT-3) or smaller but more versatile reimplementations (e.g GPT-J).  This is beyond the authors control however, and does not much diminish their contribution here.\n\n- On page 5, it was not clear to me why in the paired samples design that estimating $\\hat{\\delta} - \\delta$ represents the *overall error*.  It would help here to explain (in a footnote if need be) how the overall error is represented in terms of $\\delta$ or $\\hat{\\delta}$.\n\nQuestions:\n\n- This may be already established in other bootstrap results, but what is the sample efficiency of the convergence in distribution for Theorem 1?\n\n- As the authors detail in their environmental impact statement of section 2.1, the cost of producing the MultiBERT checkpoints even in favourable energy generation conditions conditions is not negilible.  Could the multi-bootstrap be adapted to probing smaller sets of models that themselves are resampled (e.g. under repeated dropout mask as in MC-Dropout) to approximate posterior distributions over the model parameters?  The samples would no longer be IID, but such a scheme would admit many more usecases for the multi-bootstrap.\n\n- In section 4, CDA-full sounds like a much more drastic intervention.  The MultiBERTS in CDA-incr are trained on much larger corpii, but CDA-full MultiBERTS are trained from initialization only on Webster et al’s data?  Won’t this introduce a confound whereby the CDA-full models are less proficient overall compared to the CDA-incr models?\n", "summary_of_the_paper": "Many tasks in contemporary NLP begin by building off of a large language model.  This can cast the downstream task as a sort of fine-tuning experiment, whereby the results are heavily influenced by conditioning on the starting point of a single pre-trained version of an LLM.  In this work, the authors take BERT as an example, and ask how much a specific artifact as a draw from the distribution over (model weights, initialization scheme, training data, loss function) affects downstream tasks built upon it.  \n\nThe authors provide a wide variety of BERT models that are varied in their training and initalization.  They define a bootstrap procedure for the scenario where multiple instantiations of base models are available, and tie their findings together in a case study of gender bias in coreference resolution.", "summary_of_the_review": "It's a sound paper, well overdue in the NLP literature, and reminiscent of other sober reflection papers in the vein of [Melis et al. 2017, Narang et al. 2021], though the latter concentrate mainly on testing the benefit architectural improvements.  Going beyond these papers however, the authors build and provide a bootstrap procedure to evaluate specific hypotheses.  \n\nWhile I'm not sure that ICLR is the best venue for this work, it is clearly important and deserves to be showcased.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper is a resource and numerical investigation into the variability of BERT checkpoints. It also provides a bootstrap method for making investigations on the checkpoints.

All reviewers appreciate this contribution that can be expected to be used by the NLP community.
Decision: Accept (Spotlight)