Revisiting Few-sample BERT Fine-tuning
Authors: Tianyi Zhang
Felix Wu
Arzoo Katiyar
Kilian Q Weinberger
Yoav Artzi
Venue: ICLR 2021 Poster (2021)
Keywords: Fine-tuning
Optimization
BERT

Abstract: This paper is a study of fine-tuning of BERT contextual representations, with focus on commonly observed instabilities in few-sample scenarios. We identify several factors that cause this instability: the common use of a non-standard optimization method with biased gradient estimation; the limited applicability of significant parts of the BERT network for down-stream tasks; and the prevalent practice of using a pre-determined, and small number of training iterations. We empirically test the impact of these factors, and identify alternative practices that resolve the commonly observed instability of the process. In light of these observations, we re-visit recently proposed methods to improve few-sample fine-tuning with BERT and re-evaluate their effectiveness. Generally, we observe the impact of these methods diminishes significantly with our modified process. 

Published as a conference paper at ICLR 2021
REVISITING FEW-SAMPLE BERT FINE-TUNING
Tianyi Zhang As
Felix Wu
Arzoo Katiyar
Kilian Q. Weinberger! Yoar Artzi
Penn State University t t2.88staanfrd.eee (fwu, kwe nberger, yoav) @asapp. com arzoo@psu. edu
1 ASAPP Inc.
Stanford University
'Cornell University
ABSTRACT
This paper is study of fine-tuning of BERT contextual representations, with focus on commonly observed instabilities in faw-sample scenarios. We identify several factors that cause this instability: the common of non-standard optimization method with biased gradient estimation; the limited applicability of significant parts of the BERT network for down-stream tasks; and the prevalent practice of using a pre-determined, and small number of training iterations. We empirically test the impact of these factors, and identify alternative practices that resolve the commonly observed instability of the process. In light of these observations, we re-visit recently proposed methods improve few-sample fine-tuning with BERT and re-evaluate their effectiveness. Generally, we observe the impact of these methods diminishes significantly with our modified process.
INTRODUCTION
Fine-tuning self self-supervised pre-trained models has significantly boosted state-of-the-an -rr performance on natural language processing (NLP) tasks (Liu) 2019 Yang al., 20[9a] Wadden al., 2019 Zhu Guu et al. 2020). One of the most effective models for this process is BERT (Devlin 2019). However, despite significant success, fine-tuning remains unstable, especially when using the large variant of BERT (BERT on small datasets, where pre-training stands provide the most significant benefit. Identical learning processes with different random seeds often result in significantly different and sometimes degenerate models following fine-tuning, even though only few, seemingly insignificant aspects of the learning process are impacted by the random seed (Phang As result, practitioners resort multiple random trials for model selection. This increases model deployment costs and time, and makes scientific comparison challenging (Dodge et al., (2220).
et al. 2020:
al.
al. 2018 Lee et al. 2020 Dodge al. 2020 [1]
This paper is a study of different aspects of the few-sample fine-tuning optimization process. Our goal is to better understand the impact of common choices with regard to the optimization algorithm, model initialization, and the number of fine-tuning training iterations. We identify suboptimalities in common community practices: the use of non-standard optimizer introduces bias in the gradient estimation; the top layers of the pre- rre-trained BERT model provide bad initialization point for fine- tuning: and the use of a pre-determined, , but commonly adopted number of training iterations hurts convergence. We study these issues and their remedies through experiments on multiple common benchmarks, focusing on faw-sample fine-tuning scenarios.
Once these suboptimal practices are addressed, we observe that degenerate runs are eliminated and performance becomes much more stable. This makes it unnecessary to execute numerous random restarts proposed in Dodge et al. (020).. Our experiments show the remedies we experiment with for each issue have overlapping effect. For example, allocating more training iterations can eventually compensate for using the non-standard biased optimizer, even though the combination of a bias-corrected optimizer and re-initializing some of the pre-trained model parameters can reduce fine-tuning computational costs. This empirically highlights how different aspects of fine-tuning influence the stability of the process, at times in similar manner, In the light of our observations, we re-evaluate several techniques (Phang et 2018 Lee al. 2020 Howard & Ruder 2018 that
Equal contribution, Work done at ASAPP.
Fine-tuning instahility is also receiving significant practitioner attention. For example:
https//gthhh..oomm/llaaa////ssss and http://wwwwwooooooo/oooooooonn/n

---

Published as a conference paper at ICLR 2021
recently proposed increase few-sample fine fine-tuning stability and show significant decrease in their impact. Our work furthers the empirical understanding of the fine-tuning process, and the optimization practices outline identify impactful avenues for the development of future methods.
2
BACKGROUND AND RELATED WORK
BERT The Bidirectional Encoder Representations from Transformers (BERT: Devlin al. 2019 model is a Transformer encoder (Vaswani et al. 2017) trained on raw text using masked language modeling and next-sentence prediction objectives. It generates an embedding vector contextualized through a stack of Transformer blocks for each input token. BERT prepends special [CLS] ] token to the input sentence or sentence pairs. The embedding of this token is used as a summary token for the input for classification tasks. This embedding is computed with an additional fully-connected layer with a tanh non-lincarity, commonly referred to as the pooler, to aggregate the information for the [CLS] embedding.
The common approach for using the pre-trained BERT model is to replace the original output layer with a new task-specific layer and fine-tune the complete model. This includes learning the new output layer parameters and modifying all the original weights, including the weights of word embeddings, Transformer blocks, and the pooler. For example, for sentence- levell classification, an added linear classifier projects the [CLS] embedding to an unnormalized probability vector over the output classes. This process introduces two sources of randomness: the weight initialization of the new output layer and the data order in the stochastic fine-tuning optimization. Existing work (Phang) (ct al. 2018; Lee et al. 2020 Dodge et al., 2020) shows that these seemingly benign factors can influence the results significantly, especially on small datasets (i.e., < 10K examples). Consequently, practitioners often conduct many random trials of fine-tuning and pick the best model based on validation performance (Devlin et al. (211)).
Fine-tuning
Fine-tuning Instability The instability of the BERT fine-tuning process has been known since its introduction (Devlin al., 2019), and various methods have been proposed to address it. Phang et al. (2018) show that fine-tuning the pre-trained model on large intermediate task stabilizes later fine-tuning on small datasets. [ee et al. (2020) introduce a new regularization method to constrain the fine-tuned model to stay close to the pre- tre-trained weights and show that it stabilizes fine-tuning, Dodge jet al. (2020) propose an early stopping method to efficiently filter out random seeds likely to lead to bad performance. Concurrently to our work, Mosbach et al. (2020) also show that BERT A ADAM leads to instability during fine-tuning. Our experiments studying the effect of training longer are related to previous work studying this question in the context of training models from scratch (Popel & Bojar 2018 Nakkiran et al. 2019).
BERT Representation Transferability BERT pre-trained representations have been widely stud- ied using probing methods showing that the pre-trained features from intermediate layers are more transferable (Tenney al. 2019b Liu al. 201 9a Hewitt & Manning, 2019 Hewitt & Liang 2019) or applicable (Zhang et al. (020) to new tasks than features from later layers, which change more after fine-tuning (Peters et al., 2019 Merchant et al., 2020). Our work is inspired by these findings, but focuses studying how the pre-trained weights influence the fine-tuning process. L.] et al. (020) propose to re- re-initialize the final fully-connected layer of ConvNet and show perfor- mance gain for image classification Concurrent to work, Tamkin et al. (2020) adopt similar methodology of weight re- re-initialization (Section(5) to study the transfcrability of BERT. In contrast to study, their work emphasizes pinpointing the layers that contribute the most in transfer learning, and the relation between probing performance and transferability.
3 EXPERIMENTAL METHODOLOGG
Data
We follow the data setup of previous studies (Lee et al., 2020 Phang et al. 2018 Dodge
ct al.
2020) to study few-sample fine-tuning using eight datasets from the GLUE benchmark (Wang et al. 2019b). The datasets cover four tasks: natural language inference (RTE, QNLI, MNLI), paraphrase detection (MRPC, QQP), sentiment classification (SST-2), and linguistic acceptability (CoLA), Appendix A provides dataset statistics and description of each dataset. We primarily
This concurrent work was published shortly after our study was posted.
2

---

Published as a conference paper at ICLR 2021
Algorithm 1: the ADAM pseudocode adapted from Kingma & Ba (2014), and provided for reference. 9Z. denotes the elementwise square 9: @ 9c- B1 and B2 to the power are denoted as 31 33. All operations on vectors are element-wise. The suggested hyperparameter values according to Kingma & Ba (2014) are: a = 0.001, B, = 0.9, 8g 0.999, and = 10-8
BERTADAM (Devlin et al. 2019) omits the bias correction (lines 9 10 and treats my and et
my and 00 in
Require: a: learning rate; $1, 32 E [0, 1); exponential decay rates for the moment estimates; f()): stochastic objective function with parameters 0: 0u: initial parameter vector; À E [0, 1): decoupled weight decay.
1: mo < (Initialize first moment vector)
2: Up f (Initialize second moment vector)
3: f < 0 (Initialize timestep)
4: while Og not converged do
5: 1 <- +++1
6: gi <- (0_(( (Get gradients w.r.t. stochastic objective at timestep ()
7:
ma + 31 - - + (1 - 31) - gt (Update biased first moment estimate)
8:
Vt + 32-00= 1011+/1 (1 B2) - (Update biased second raw moment estimate)
9:
My me/(1= - -8,) (Compute bias-corrected first moment estimate)
10:
D. + 01/(1 (85) (Compute borr-coree second raw moment estimate)
11: O. aa. aaaaa////eeee. + e) (Update parameters)
12: end while
13: return 02 (Resulting parameters)
focus on four datasets (RTE, MRPC, STS-B, CoLA) that have fewer than 10k training samples, because BERT fine-tuning on these datasets known be unstable (Devlin et al. 2019). We also
complement our study by downsampling all eight datasets to 1k training examples following Phang jet al. (2018). While previous studies (Lee et al.) 2020 Phang et al., 2018: Dodge et al. 2020) focus on the validation performance, we split held-out test sets for our study For RTE, MRPC, STS-B, and CoLAA, we divide the original validation set half, using one half for validation and the other for test. For the other four larger datasets, we only study the downsampled versions, and split additional Ik samples from the training set as our validation data and test on the original validation set.
Experimental Sctup Unless noted otherwise, we follow the hyperparameter setup of Lee et al. (202)). We fine-tune the uncased, 24-layer BERT model with batch size 32, dropout 0.1, and peak learning rate 2 x 10-5 for three epochs. We clip the gradients to have a maximum norm of 1. We apply linear learning rate warm-up during the first 10% of the updates followed by linear decay. We use mixed precision training using Apex to speed up experiments. We show that mixed precision training does not affect fine-tuning performance in Appendix C We evaluate ten times on the validation set during training and perform early stopping. We fine-tune with 20 random seeds to compare different settings.
4 OPTIMIZATION ALGORITHM: DEBIASING OMISSION IN BERTADAM
The most commonly used optimizer for fine-tuning BERT is BERTADAM a modified version of the ADAM first-order stochastic optimization method. It differs from the original ADAM algo- rithm (Kingma & Bai (0144) in omitting a bias correction step. This change was introduced by Devlin al. (2019), and subsequently made its way into common open source libraries, including the official implementation 5 Thugginfface's Transformers (Wolf et al. 2019 AllenNLP (Gardner et al. 2018 GluonNLP (uoo et al., 2019), jiant (Wang al. 2019c).. MT-DNN (Liu et al. 2020). and FARM As
a result, this non- non-standard implementation is widely used in both industry and research (Wang et al.
2019a [Phang al. 2018 Lee al. 2020: Dodge et al. 2020 Sun et al.
2019 Clark al. 2020
Lan et al.] Houlsby et al. 2019: Stickland & Murray [019]] Liu et al. 2019b). We observe that
the bias correction omission influences the learning rate, especially early the fine-tuning process,
and is one of the primary reasons for instability in fine-tuning BERT (Devlin et al. 2019 Phang et al.
2018
Lee al. 2020 Dodge et al. 2020).
Algorithm D shows the ADAM algorithm, and highlights the omitted line in the non-standard BERTADAM implementation. At each optimization step (lines 4 ITI. ADAM computes the exponen-
3 'The original test sets are not publicly available.
hntps://github.com wwwp://wwwooo///////eee www./o/o///ww/aaa VIDIA/apex
wtt.://www.......ooooooooo///o/a 11tp://iiaaiiiaii11111-11--1 1y#L108-L157
"The default was changed from BERTADAM to debased ADAM in commit cc07cf5a on July 11, 2019.

---

Published as a conference paper at ICLR 2021
Performance Distribution
1.0
RTE
Bius
Update Magnitude
RTT
Test Performancs
0.8
1.00
Blas in
18:0
1111
0.6
Train Loss
0:75
MNLI
0.4
Correction
0.50
0.2
Correction
Conrection
Median
0.25
No Correction
10" 10' Training Iterations (log scale)
10"
10
10'
10'
10
0.0
Outlier
23
230
RTE MRPC CoLAA STS-B
Steps
Figure 1: Bias in the ADAM up- date as function of training iter- ations. Vertical lines indicate the typical number of iterations used to fine-tune BERT on four small datasets and one large dataset (MNLI). Small datasets use fewer iterations and are most affected.
Figure 2: Performance dis- tribution box plot across 50 random trials and the four datasets with and without ADAM bias correction. Bias correction reduces the vari- ance of fine-tuning results by large margin.
Figure 3: Mean (solid lines) and range (shaded region) of training loss during fine- tuning BERT, across 50 ran- dom trials. Bias correction speeds up convergence and shrinks the range of training loss.
tial moving average of the gradients (m²) and the squared gradients (wx), where B1, B2 parameterize the averaging (lines 7+8) Because ADAM initializes m, and i2 to 0 and sets exponential decay rates 31 and 32 close to 1. the estimates of my and U2 are heavily biased towards 0 early during learning when t is small. Kingma & Ba/ (2014) computes the ratio between the biased and the unbiased estimates of my and 02 as (1 -1--8 (i) and (1 32). This ratio is independent of the training data. The model parameters are updated in the direction of the averaged gradient m, divided by the square root of the second moment vt, (line BERTADAM omits the debiasing (lines 10). and directly uses the biased estimates in the parameters update.
Figure shows the ratio the between the update using the biased and the unbiased estimation vo, as a function of training iterations. The bias is relatively high early during learning, indicating overestimation. It eventually converges to one, suggesting that when training for sufficient iterations, the estimation bias will have negligible effect Therefore, the bias ratio term is most important early during learning to counteract the vverestimation of my and U2 during early iterations. In practice,
ADAM adaptively re-scales the learning rate by V-- This correction is crucial for BERT fine-
tuning on small datasets with fewer than 10k training samples because they are typically fine-tuned with less than 1k iterations (Devlin et al.] (219)). The figure shows the number of training iterations for RTE, MRPC, STS-B, CoLA, and MNLI MNLI is the only one of this set with large number of supervised training examples. For small datasets, the bias ratio is significantly higher than one for the entire finettuning process, implying that these datasets suffer heavily from overestimation in the update magnitude. In comparison, for MNLI, the majority of fine-tuning occurs in the region where the bias ratio has converged one. This explains why fine-tuning on MNLI known to be relatively stable (Devlin et al. 2019).
We evaluate the importance of the debiasing step empirically by fine-tuning BERT with both BERTADAM and the debiased ADAM for 50 random seeds on RTE, MRPC, STS-B, and CoLA. Figure 2 summarizes the performance distribution. The bias correction significantly reduces the performance variance across different random trials and the four datasets. Without the bias correction we observe many degenerate runs, where fine-tuned models fail to outperform the random baseline. For example, on RTE, 48% of fine-tuning runs have an accuracy less than 55%, which is close to random guessing. Figure 3 further illustrates this difference by plotting the mean and the range of training loss during fine-tuning across different random trials on RTE. Figure 11 in Appendix sssows similar plots for MRPC, STS-B. and CoLA. The biased BERTADAM consistently leads to worse averaged training loss, and on all datasets to higher maximum training loss. This indicates models trained with BERTAdam are underfitting and the root of instability lies in optimization.
$ *Our experiments on the completely MNLI dataset confirm using the unbiased estimation does not improve nor degrade performance for large datasets (Appendix D) We use the PyTorch ADAM implementation http:///yyyccoooooo/c/////////ae
4

---

Published as a conference paper at ICLR 2021
RIE
MRPC
STS-B
CoLA
0.75
0.95
0.90
Exp, Test MCC.
0.70
0.92
Acc.
0.89
0.65
0.65
0.89
0.88
0.60
0.55
0.60
0.86
0.87
0.50
Correction
0.55
0.45
No Correction
0.50
20
0.80
20
0.85
20
0.40
20
of Random Trials
of Random Trials
of Random Trials
of Random Trials
Figure 4: Expected test performance (solid lines) with standard deviation (shaded region) over the number of random trials allocated for fine-tuning BERT. With bias correction, we reliably achieve good results with few (i.e., 10) random trials.
Dataset
RTE
MRPC
STS-B
CoLA
Epochs
Longer
3
Epochs
Longer
3 Epochs
Longer
3 Epochs 63.0 = 1.5
Longer 62.4 61.9 1.3
Standard Re-inil
69.5 + 2.5 72.6 1.6
72.3 + 1.
90.8 + 1.3
90.5 = 1.5 91.0
89.0 + 0.6
89.6 = 0.3
73.1
1 1.3
91.4 1 0.8
89.4 1 0.2
89.9 0.1
63.9 1.9
Dataset
RTE (Ik)
MRPC (Ik)
STS-B (Ik)
CoLAA (Ik)
Epochs
Longer
3 Epochs 80.5 + 3.3 84.6 1 1.6
Longer
Epochs
Longer
Epochs
Longer
Standard Re-init
62.5 2.8 65.6 1 2.0
65.2 1 2.1
83.8 + 2.1
84.7 + 1.4
88.0 0.4
45.9 47.6
= 1.6
48.8 +
65.8 E 1.7
86.0 1.2
87.2 0.4
88.4 0.2
1.8
Dataset
SST (Ik)
QNLI (Ik)
QQP (Ik)
MNLI (Ik)
3 Epochs
Longer
3 Epochs
Longer
3 Epochs
Longer
3 Epochs 52.2 = 4.2
Longer
Standard
89.7 1.5
90.9 4 0.5
78.6 2.0
81.4 0.9
74.0 2.7
77.4 0.8
67.5
Re-init
90.8 0.4
0.5
0.5
0.3
7.72
77.6 0.6
66.4 = 0.6
68.8 0.5
Table 1: Mean test performance and standard deviation. We compare fine-tuning with the complete BERT model (Standard) and fine-tuning with the partially re-initialized BERT (Re-init). We show results of fine-tuning for 3 epochs and for longer training (Sec()). We underline and highlight in blue the best and number statistically equivalent to it among each group of numbers. We use one-tailed Student's t-test and reject the null hypothesis when p < 0.05.
We simulate realistic setting of multiple random trials following Dodge et al. 2020). We use bootstrapping for the simulation: given the 50 fine-tuned models we trained, we sample models with replacement, perform model selection on the validation set, and record the test results; we repeat this process times to estimate mean and variance. ssows the simulated test results as function of the number of random trials. Appendix provides the same plots for validation performance. Using the debiased ADAM we can reliably achieve good results using fewer random trials; the difference in expected performance is especially pronounced when we perform less than 10 trials. Whereas the expected validation performance monotonically improves with more random trials (Dodge et al. (2020), the expected test performance deteriorates when we perform too many random trials because the model selection process potentially overfits the validation set. Based on these observations, we recommend performing moderate number of random trials (i.e., 5 or 10).
INITIALIZATION RE-INITIALIZING BERT PRE-TRAINED LAYERS
The initial values of network parameters have significant impact on the process of training deep neural networks, and various methods exist for careful initialization (Glorot & Bengio 2010 He et al.. 2015 Zhang et al. 2019: Radford et al. 2019 Dauphin& & Schoenholz 2019). During fine-tuning, the BERT parameters take the role of the initialization point for the fine-tuming tine-tuning optimization process, while also capturing the information transferred from pre-training. The common approach for BERT fine-tuning initialize layers except one specialized output layer with the pre-trained weights. We study the value of transferring all the layers in contrast to simply ignoring the information learned in some layers. This is motivated by object recognition transfer learning results showing that lower pre-trained layers learn more general features while higher layers closer to the output specialize more to the pre- tra-training tasks YYsiinski et al. 2014). Existing methods using BERT show that using the complete network is not always the most effective choice, we discuss in Section 2 Our empirical results further confirm this: we observe that transferring the top pre-trained layers slows down learning and hurts performance.
5

---

Published as a conference paper at ICLR 2021
MRFC
RTE
MRPC
0.47
1.00
Val. Accuney
Standard Re-init
0.325
0.90
Train Loss
0.75
Train Loss
0.75
00000
Standasl Re-iak
0.50
0.50
0.575
Merkan
Outlae
0.25
Standird Re Ronker
Ke-tint
Re Ko-ink col Reatit
Stand fand
Re-vint
0.25
92
34
136
238
340
Steps
Steps
Figure 5: Validation performance distribution of re-initializing different number of layers of the BERT model.
Figure 6: Mean (solid lines) and Range (shaded region) of training loss during fine-tuning BERT, across 20 random trials. Re-init leads to faster convergence and shrinks the range.
We test the transferability of the top layers using a simple ablation study. Instead of using the pre- trained weights for all layers, we re-initialize the pooler layers and the top L E N BERT Transformer blocks using the original BERT initialization, (0, 0.02²). We compare two settings: (a) standard fine-tuning with BERT, and (b) Re-init fine-tuning of BERT. We evaluate Re-init by selecting L E {I,.... 6} based mean validation performance. All experiments use the debiased ADAM
(Section with 20 random seeds.
Re-init Impact on Performance Table shows our results on all the datasets from Section We show results for the common setting of using 3 epochs, and also for longer training, which discuss Re-init consistently improves mean performance on all the datasets, showing
and study in Section [ that not all layers are beneficial for transferring. It usually also decreases the variance across all datasets. Appendix | shows similar benefits for pre-trained models other than BERT.
Sensitivity to Number of Layers Re-initialized Figure 5 shows the effect of the choice of L,, the number of blocks re-initialize, on RTE and MRPC. Figure K in Appendix | shows similar plots for the rest of the datasets. We observe more significant improvement in the worst-case performance than the best performance, suggesting that Re-init more robust to unfavorable random seed. We already see improvements when only the pooler layer is re-initialized. Re-initializing further layers helps more. For larger L though, the performance plateaus and even decreases as re-initia pre-trained layers with general important features. The best L varies across datasets.
Effect on Convergence and Parameter Change Figure G shows the training loss for both the standard fine-tuning and Re-init on RTE and MRPC. Figure [3] Appendix Appendix[[sshos shows the training loss for all other datasets. Re-init leads to faster convergence. We study the weights of different Transformer blocks. For each block, we concatenate all parameters and record the L2 distance between these parameters and their initialized values during fine-tuning. Figure 7 plots the L2 distance for four different transformer blocks as a function of training steps on RTE, and Figures [5 18 in Appendix [ show all transformer blocks four datasets. In general, Re-init decreases the 7.2 distance to initialization for top Transformer blocks (i.e., 18-24). Re-initializing more layers leads to larger reduction, indicating that Re-init decreases the fine-tuning workload. The effect of Re-init is not local; even re-initializing only the topmost Transformer block can affect the whole network. L = 3 continues benefit the bottom Transformer blocks, re-initializing too many layers (e.g.. L = 10) can increase the L2 distance in the bottom Transformer blocks, suggesting a tradeoff between the bottom and the top Transformer blocks. Collectively, these results suggest that Re-init finds a better initialization for fine-tuning and the top L layers of BERT are potentially overspecialized to the pre-training objective.
While setting
6 TRAINING ITERATIONS: FINE-TUNING BERT FOR LONGER
BERT is typically fine-tuned with a slanted triangular learning rate, which applies linear warm-up to the learning rate followed by a linear decay. This learning schedule warrants deciding the number of training iterations upfront. [Dvlinn et al. (219)) recommend fine-tuning GLUE datasets for three epochs. This recommendation has been adopted broadly for fine-tuning (Phang et al. 20188) Lee et al. 2020; Dodge et al. 2020). We study the impact of this choice, and observe that this fill--ll

---

Published as a conference paper at ICLR 2021
12 Initialization
0.90
Transformer Block 6
0.90
Transformer Block 12
0.90
Transformer Block 18
0.90
Transformer Block 24
0.72
0.72
0.72
0.54
0.54
0.54
0.54
0.36
Standard
Re init
0.36
0.36
0.18
Re-init Re-init 10
0.18
0.18
0.18
Re-init
0.00
0.00
0.00
0.00
100
150 Steps
200
50
100
150 Steps
200
250
250
50
100
150
200
250
Steps
Steps
Figure 7: L2 distance to the initial parameters during fine-tuning BERT RTE. Re-init reduces the amount of change in the weights of top Transformer blocks. However, re-initializing too many layers causes larger change in the bottom Transformer blocks.
RTB (Ik)
MRPC (Ik)
STS-B (Ik)
CoLA (Ik)
Val Perforanne
0.75
Val Perfornacee
Val Perfomancee
Val Perforancce
0.875
0.88
0.60
0.70
0.86
0.850
0.65
Re-init
Re-init 5
Re-init
Re-init
0.84
0.55
0.60
Standard
0.825
Standard
Standard
Standard
0.82
96
200 Training Iterations
400
800 1640 3200
96
200
400
W00 1610 3200
96
200
400
800 1610 3200
96
200
400
800
1610 3200
Training Iterations
Training Iterations
Training Iterations
Figure 8: Mean (solid lines) and range (shaded region) of validation performance trained with different number of iterations, across eight random trials.
three-epochs practice for BERT fine-tuning is sub-optimal. Fine-tuning BERT longer can improve both training stability and model performance.
Experimental setup We study the effect of increasing the number of fine-tuning iterations for the datasets in Section 3 For the 1k downsampled datasets, where three epochs correspond to 96 steps, we tune the number of iterations in {200, 400, 800, 1600, 3200). For the four small datasets, we tune the number of iterations in the same range but skip values smaller than the number of iterations used in three epochs. We evaluate our models ten times on the validation set during fine-tuning. This number is identical to the experiments in Sections 5 and controls for the set of models to choose from. We tune with eight different random seeds and select the best set of hyperparameters based on the mean validation performance to save experimental costs. After the hyperparameter search, we fine-tune with the best hyperparameters for 20 seeds and report the test performance.
Results Table D shows the result under the Longer column. Training longer can improve over the three-epochs setup most of the time, in terms of both performance and stability. This is more pronounced on the 1k downsampled datasets. We also find that training longer reduces the gap between standard fine-tuning and Re-init, indicating that training for more iterations can help these models recover from bad initializations. However, on datasets such as MRPC and MNLI, Re-init still improves the final performance even with training longer. We show the validation results on the four downsampled datasets with different number of training iterations in Figure & We provide a similar plot in Figure 14 Appendix G for the other downsampled datasets. We observe that different tasks generally require different number of training iterations and is difficult to identify
mne-size-fits-a solution. Therefore, we recommend practitioners to tune the number of training iterations on their datasets when they discover instability in fine-tuning. We also observe that on most of the datasets, Re-init requires fewer iterations to achieve the best performance, corroborating that Re-init provides a better initialization for fine-tuning.
REVISITING EXISTING METHODS FOR FEW-SAMPLE BERT FINE-TUNING
Instability in BERT fine-tuning, especially in fam-lample settings, is receiving increasing attention recently (Devlin et al. 2019: Phang et al. 2018 Lee et al. 2020 Dodge et al. 2020). We revisit these methods given our analysis of the fine-tuning process, focusing on the impact of using the debiased ADAM instead of BERTADAM (Section 4). Generally, we find that when these methods are re-evaluated with the unbiased ADAM they are less effective with respect to the improvement in fine-tuning stability and performance.
7

---

Published as a conference paper at ICLR 2021
Standard
Int. Task
LLRD
Mixout
Pre-trained WD
WD
Re-init
Longer
RTE
69.5 + 2.5
81.8 $18+17
69.7
4.2 3.2
71.34
1.4
69.6 2.1
69.5
+ 2.5
72.6
22.611 1.6
+ 1.9 91.0
72.3
MRPC 90.8 + 1.3
91. 1.8+10
913 1.1
90.4
+ 1.4
90.8 = 1.3
90.8
= 1.3
9L4
+ 0.8
1.3
STS-B
89.0 = 0.6
0.3
89.2
= 0.4
89.2
+ 0.4
89.0
= 0.5
89.0
= 0.6
89.4
0.2
89.6
= 0.3
CoLA 63.0 1 1.5
63.9 1.8
63.0 2.5
61.6
1 1.7
63.4 1 1.5
63.0 1 1.5
64.2
1 1.6
62.4 1.7
Table 2: Mean test performance and standard deviation on four datasets. Numbers that are statistically significantly better than the standard setting (left column) are in blue and underlined. The results of Re-init and Longer are copied from Table [ All experiments use ADAM with debiasing (Section 4 Except Longer, all methods are trained with three epochs. "Int. Task" stands for transfering via an intermediate task (MNLI).
7.1 OVERVIEW
Pre-trained Weight Decay Weight decay (WD) is a common regularization technique (Krogh & Hertz 1992). At each optimization iteration, Aw is subtracted from the model parameters, where À is a hyperparameter for the regularization strength and w is the model parameters. Pre-trained weight decay adapts this method for fine-tuning pre-trained models (Chelba & Acero 2004 Daumé III 2007) by subtracting A(w w) from the objective, where w is the pre-trained parameters. Lee et al. (2020) empirically show that pre-trained weight decay works better than conventional weight decay in BERT fine-tuning and can stabilize fine-tuning.
Mixout Mixout (Lee et al. 2020) is a stochastic regularization technique motivated by Dropout (Sni vastava et al. 2014) and DropConnect (Wan et al. 2013). At each training iteration, cach model parameter is replaced with its pre-trained value with probability p. The goal is to prevent catastrophic forgetting, and (Lee et al. (200)) proves it constrains the fine-tuned model from deviating much from the pre-trained initialization.
Layer-wise Learning Rate Decay (LLRD) LLRD (Howard Ruder 2018 a method that applies higher learning rates for top layers and lower learning rates for bottom layers. This is accomplished by setting the learning rate of the top layer and using a multiplicative decay rate to decrease the learning rate layer-by-layer from top bottom. The goal is to modify the lower layers that encode more general information less than the top layers that are more specific to the pre-training task. This method is adopted in fine-tuning several recent pre- pre-trained models, including XLNet (Yang ct al. 2019b) and ELECTRA (Clark et al. 2020).
Transferring via an Intermediate Task Phang et al. (0018) propose conduct supplementary fine-tuning on a larger, intermediate task before fine-tuning on few-sample datasets. They show that this approach can reduce variance across different random trials and improve model performance. Their results show that transferring models fine-tuned on MNLI (Williams et al.] (018)) can lead to significant improvement on several downstream tasks including RTE, MRPC, and STS-B. In contrast to the other methods, this approach requires large amount of additional annotated data.
7.2 EXPERIMENTS
We evaluate all methods RTE, MRPC, STS-B, and CoLA. We fine-tune a BERT Large model using the ADAM optimizer with debiasing for three epochs, the default number of epochs used with each of the methods. For intermediate task fine-tuning, we fine-tune BERT model on MNLI and then fine-tune for our evaluation. For other methods, we perform hyperparameter search with a similar size search space for each method, as described in Appendix H We do model selection using the average validation performance across 20 random seeds. We additionally report results for standard fine-tuning with longer training time (Section(5). 6). weight decay, and Re-init (Section 5).
Table 2 provides our results. Compared to published results (Phang et al.) 2018 Lee et al., 2020). our test performance for Int. Task (transferring via an intermediate task), Mixout, Pre-trained WD, and WD are generally higher when using the ADAM with debiasing! However, observe less pronounced benefits for all surveyed methods compared to results originally reported. At
The numbers in Table 2 are not directly comparable with previously published validation results (Phang al. 2018 Lee et al. 2020) because are reporting test performance. However, the relatively large margin between results and previously published results indicates an improvement. More important, our focus is the relative improvement, lack of improvement compared simply training longer.
8

---

Published as a conference paper at ICLR 2021
times, these methods do not outperform the standard baselines simply training longer. Using additional annotated data for intermediate task training continues to be effective, leading to consistent improvement over the average performance acToss all datasets. LLRD and Mixout show less consistent performance impact. We observe no noticeable improvement using pre-trained weight decay and conventional weight decay in improving stabilizing BERT fine-tuning in our experiments, contrary to existing work (Lee et al. (2020). This indicates that these methods potentially case the optimization difficulty brought by the debiasing omission BERT ADMM, and when we add the debiasing, the positive effects are reduced.
CONCLUSION
We have demonstrated that optimization plays vital role in the few-sample BERT fine-tuning. First, we show that the debiasing omission BERTADAM is the main cause of degenerate models on small datasets commonly observed in previous work (Phang et al. 2018 Lee al. 2020 Dodge al.) 2020). Second, we observe the top layers of the pre-trained BERT provide a detrimental initialization for fine-tuning and delay learning. Simply re-initializing these layers not only speeds up learning but also leads to better model performance. Third, we demonstrate that the common one-size-fits- three-epochs practice for BERT fine-tuning is sub-optimal and allocating more training time can stabilize fine-tuning. Finally, revisit several methods proposed for stabilizing BERT fine-tuning and observe that their positive effects are reduced with the debiased ADAM. In the future, we plan to extend our study to different pre vre-training objectives and model architectures, and study how model parameters evolve during fine-tuning.
ACKNOWLEDGMENTS
We thank Cheolhyoung Lee for his help in reproducing previous work. We thank Lili Yu, Ethan R. Elenberg, Varsha Kishore, and Rishi Bommasani for their insightful comments, and Hugging Face for the Transformers project, which enabled our work.
REFERENCES
Luisa Bentivogli, Ido Kalman Dagan, Dang Hoa, Danilo Giampiccolo, and Bernardo Magnini. The fifth pascal recognizing textual entailment challenge. In TAC 2009 Workshop, 2009.
Daniel Cer, Mona Diab, Eneko Agirre, lñigo Lopez- and Lucia Specia. Semeval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In SemEval-2017, 2017.
Ciprian Chelba and Alex Acero. Adaptation maximum entropy capitalizer: Little data can help a lot. In EMNLP, 2004.
Kevin Clark, Minh- Thang Luong, Quoc V. Le, and Christopher D. Manning, ELECTRA: Pre-training text encoders as discriminators rather than generators. ICLR, 2020.
Hal Daumé III. Frustratingly easy domain adaptation. In ACI., 2007.
Yann N Dauphin and Samuel Schoenholz. Metainit: Initializing learning by learning initialize. In NeurIPS, 2019.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, 2019.
Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah Smith. Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping. arXiv preprint arXiv:2002.06305, 2020.
William B Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In /WP, 2005.

---

Published as a conference paper at ICLR 2021
Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi. Nelson F Liu, Matthew Peters, Michael Schmitz, and Luke Zettlemoyer. Allennlp: deep semantic natural language processing platform. In NLP-OSS, 2018.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In AISTATS, 2010.
Jian Guo, He He, Tong He, Leonard Lausen, Mu Li, Haibin Lin, Xingjian Shi, Chenguang Wang, Junyuan Xie, Sheng Zha, Aston Zhang, Hang Zhang, Zhi Zhang, Zhongyue Zhang, and Shuai Zheng, Gluoncv and gluonnip: Deep learning in computer vision and natural language processing. arXiv preprint arXiv: 04433, 2019.
Kelvin Guu, Kenton Lee, Zora Tung. Panupong Pasupat, and Ming-Wei Wei Chang. Realm: Retrieval- augmented language model pre-training. arXiv preprint arXiv:2002.0890 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In ICCV, 2015.
Hewitt and P. Liang. Designing and interpreting probes with control tasks. EMNLP, 2019.
John Hewitt and Christopher D. Manning. A structural probe for finding syntax in word representa- tions. In NAACL, 2019.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. In ICML, 2019.
Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. In ACL, 2018.
Shankar Iyer, Nikhil Dandekar, and Kornel Csernai. First quora dataset release: Question pairs.
https: /ttp::///tinuuue httptiiinnrllrrreeeeeeeeeeeeeeet wwt.//o///////// com.yllyuscccmm 2017.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint 0r/i::/4/2.69880 2014.
Anders Krogh and John A. Hertz. A simple weight decay can improve generalization. In NeurlPS, 1992.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations. In ICLR, 2020.
Cheolhyoung Lee, Kyunghyun Cho, and Wanmo Kang. Mixout: Effective regularization to finetune large-scale pretrained language models. In ICLR, 2020.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. Bart: Denoising -osenuce-t--seunnceeeee pre-training for natural language generation, translation, and comprehension. arXiv preprint 1...v:1900...5. 2019.
Xingjian Li, Haoyi Xiong, Haozhe An, Chengzhong Xu, and Dejing Dou. Rifle: Backpropagation in depth for deep transfer learning through re-initializing the fully-connected layer. ICMI., 2020.
Nelson Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. Linguistic knowledge and transferability of contextual representations. arXiv preprint 19XXiv:903.0880. 2019a.
Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. Multi-task deep neural networks for natural language understanding. In ACL, 2019b.
Xiaodong Liu, Yu Wang, Jianshu Ji, Hao Cheng, Xueyun Zhu, Emmanuel Awa, Pengcheng He, Weizhu Chen, Hoifung Poon, Guihong Cao, and Jianfeng Gao. The microsoft toolkit multi-task deep neural networks for natural language understanding. arXiv preprint arXiv:2002.07972, 2020.
Yang Liu. Fine-tune BERT for extractive summarization. arXiv preprint arXiv: 1903.. 10318, 2019.
10

---

Published as a conference paper at ICLR 2021
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: Robustly Optimized BERT Pretraining Approach. arXiv preprint arXiv:1907.1199, 2019c.
Brian Matthews. Comparison of the predicted and observed secondary structure of t4 phage lysozyme. Biochimica Biophysica Acta (BBAe-nrotein Structure, 1975.
Amil Merchant, Elahe Rahimtoroghi, Ellie Pavlick, and Ian Tenney. What happens to bert embeddings during fine-tuning? arXiv preprini arXiv:2004. 10:0000.44444 14448, 2020.
Marius Mosbach, Maksym Andriushchenko, and Dietrich Klakow. On the stability of fine-tuning bert: Misconceptions, explanations, and strong baselines. arXiv preprint arXiv:2006. 04884, 2020.
Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. Deep double descent: Where bigger models and more data hurt. In ICLR, 2019.
Matthew E. Peters, Sebastian Ruder, and Noah A. Smith. To tune not to tune? adapting pretrained representations to diverse tasks. arXiv preprint arXiv:1903.05987 2019.
Jason Phang, Thibault Févry, and Samuel R Bowman. Sentence encoders stilts: Supplementary 2018
training on intermediate labeled-data tasks. arXiv preprint
Martin Popel and Ondřej Bojar. Training tips for the transformer model. The Prague Bulletin of Mathematical Linguistics, 2018.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP, 2013.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. JMLR, 2014.
Asa Cooper Stickland and Iain Murray. Bert and pals: Projected attention layers for efficient adaptation in multi-task learning. In ICML, 2019.
Chi Sun, Xipeng Qiu, Yige Xu, and Xuanjing Huang. How to fine-tune bert for text classification? In CCL. 2019.
Alex Tamkin, Trisha Singh, Davide Giovanardi. and Noah Goodman. Investigating transferability in pretrained language models. EMNLP, 2020.
Ian Tenney, Dipanjan Das, and Ellie Pavlick. Bert rediscovers the classical nlp pipeline. In ACL, 2019a.
Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Sam Bowman, Dipanjan Das, and Ellie Pavlick. What do you learn from context? probing for sentence structure in contextualized word representations. In ICLR, 2019b.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurlPS, 2017.
David Wadden, Ulme Wennberg, Yi Luan, and Hannaneh Hajishirzi. Entity, relation, and event extraction with contextualized span representations. In EMNLP-IJCNLP, 2019.
Li Wan, Matthew Zeiler, Sixin Zhang, Yann Le Cun, and Rob Fergus. Regularization of neural networks using dropconnect. In ICML,, 2013.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: stickier benchmark for general-purpose language understanding systems. In NeurIPS, 2019a.

---

Published as a conference paper at ICLR 2021
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill. Omer Levy, and Samuel Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In ICLR, 2019b.
Alex Wang, Ian F. Tenney, Yada Pruksachatkun, Phil Yeres, Jason Phang, Haokun Liu, Phu Mon Htut, Katherin Yu, Jan Hula, Patrick Xia, Raghu Pappagari, Shuning Jin, R. Thomas McCoy, Roma Patel, Yinghui Huang, Edouard Grave, Najoung Kim, Thibault Févry, Berlin Chen, Nikita Nangia, Anhad Mohananey, Katharina Kann, Shikha Bordia, Nicolas Patry, David Benton, Ellie Pavlick, and Samuel R. Bowman. jiant 1.3: A software toolkit for research on general-purpose text understanding models. http:// /jip://jiant... info/ 2019c.
Alex Warstadt, Amanpreet Singh, and Samuel Bowman. Neural network acceptability judgments. TACL, 2019.
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In ACL, 2018.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R'emi Louf, Morgan Funtowicz, and Jamie Brew. Huggingface's transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771, 2019.
Wei Yang. Haotian Zhang, and Jimmy Lin. Simple applications of BERT for ad hoc document retrieval. arXiv preprint arXiv: 10972, 2019a.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc Le. Xlnet: Generalized autoregressive pretraining for language understanding, In NeurIPS, 2019b.
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? In NeurlPS, 2014.
Hongyi Zhang, Yann N. Dauphin, and Tengyu Ma. Residual learning without normalization via better initialization. In ICLR, 2019.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. BERTScore: Evaluating Text Generation with BERT. In ICLR, 2020.
Jinhua Zhu, Yingce Xia, Lijun Wu, Di He, Tao Qin, Wengang Zhou, Houqiang Li, and Ticyan Liu. Incorporating bert into neural machine translation. In ICLR, 2020.
12

---

Published as a conference paper at ICLR 2021
RTE NLI
MRPC Puraphrase
STS-B Similarity
CoLA Acceptibility
SST-2 Sentiment
ONLI NLI
QQP Paraphrase
MNLI
Task
* of training samples
2.5k
3.7k
5.8k
8.6k
61.3k
104k
363k
392k
of validation samples
139
204
690
521
lk
1k
1k
# of test samples
139
205
690
521
1.8k
5.5k
40k
9.8k
Evaluation metric Majurity baseline (val)
Acc.
F1
SCC
MCC
Acc.
Acc.
Acc.
Acc.
$2.9
81.3
0
50.0
50.0
50.0
33.3
Majority baseline (test)
52.5
81.2
0
49.1
50.5
63.2
31.8
Table The datasets used in this work. We apply non- standaddd data splits create test sets. SCC stands for Spearman Correlation Coefficient and MCC stands for Matthews Correlation Coefficient.
A DATASETS
Table 3 summarizes dataset statistics and describes our validation/test splits. We also provide a brief introduction for each datasets:
RTE Recognizing Textual Entailment (Bentivogli et al. 2009 is binary entailment classification
task. We use the GLUE version.
MRPC Microsoft Research Paraphrase Corpus (Dolan & Brockett 2005) is binary classification task. Given a pair of sentences, a model has to predict whether they are paraphrases of each other. We use the GIUE version.
2017 a regression tasks for estimating STS-B Semantic Textual Similarity Benchmark (Cer et al. sentence similarity between a pair of sentences. We use the GLUE version.
CoLA Corpus of Linguistic Acceptability (Warstadt et al. 2019) is a binary classification task for verifying whether a sequence of words is grammatically correct English sentence. Matthews correlation coefficient (Matthews 1975) is used to evaluate the performance. We use the GLUE version.
MNLI Multi-Genre Natural Language Inference Corpus (Williams et al. 2018) is textual entail- ment dataset, where model is asked to predict whether the premise entails the hypothesis, predicts the hypothesis, or neither. We use the GLUE version.
QQP Quora Question Pairs (Iyer et al. 2017) is binary classification task to determine whether two questions are semantically equivalent (i.e., paraphrase each other). We use the GLUE version.
SST-2 The binary version of the Stanford Sentiment Treebank (Socher al., 2013 binary classification task for whether a sentence has positive or negative sentiment. We use the GLUE version.
B ISOLATING THE IMPACT OF DIFFERENT SOURCES OF RANDOMNESS
The randomness in BERT fine-tuning comes from three sources: (a) weight initialization, (b) data order, and (c) Dropout regularization (Srivastava al. 2014). We control the randomness using two separate random number generators: one for weight initialization and the other for both data order and Dropout (both of them affect the stochastic loss each iteration). We fine-tune BERT on RTE for three epochs using ADAM with 10 seeds for both random number generators. We compare the standard setup with Re-init where L This experiment is similar to Dodge use ADAM with debiasing instead of BERTADAM and control for the randomness in Dropout as well. When fixing a random seed for weight initialization, Re-init 5 shares the same initialized classifier weights with the standard baseline. Figure shows the validation accuracy of each individual run as well as the minimum, average, and maximum scores when fixing one of the random seeds. Figure I0 summarizes the standard deviations when one of the random seeds is controlled. We observe several trends. Re-init 5 usually improves the performance regardless of the weight initialization or data order and Dropout. Second, Re-init still reduces the instability when of the sourees of randomness is controlled. Third, the standard deviation of fixing the weight initialization roughly matches the one of controlled data order and Dropout, which aligns with the observation of Dodge al. (2020).
(2020). but we
13

---

Published as a conference paper at ICLR 2021
Standard Handom Seed Data Order and Deopont
Ro-init Handom Seed for Data Order
Dropoun
Min Mean Mox
5
Min Moon Mix
69.00
7466 23:00
25.36
78
78
13.01
77.04 TOLER 16.29
25.26
10.00 5720
13.36
6037
6A12 1357
73.74 75.11 73.00 76.44 76.20
Barrdom Soed for Initaalzaiio
1729 59332
13.77
Tursdom Sweel for Initaitratioo
N6.00
74
22.22 21.00
01 6604
74-78 72.32 75.29
75.40 12.10 96.25
64
64
75.09
73.36 73.22
76.00
77.54
73.50 78.30 26.20 01 75.00 76.61 72.30 77.54 73,01 75.42
Figure 9: Validation accuracy on RTE with controlled random seeds. The min, mean, and max values of controlling one of the random seeds are also included. Re-init 5 usually improves the validation accuracy.
Standard
Re-init 5
Bandom Soed for Data Order and Dropout
Rundom Seed Duta Order and Drupout
Arg:
Ang.
Su
Stdl
3.5
2.34
Rendam Seed loit mntialzztion
2.5
2.0
2.0
1.52
Rendom Seed
1.5
1.5
2.16
1.0
0.44
1.0
1:4
0.
0.50
2.54
2.07
2.52
1:30
3.72
2:21
Std
1.07
1:28
1.40
137
0:54
1:74
1:25
1:41
1.41
0.5
1.40
Figure 10: The standard deviation of the validation accuracy on RTE with controlled random seeds. We show the standard deviation of fixing either the initialization or data order and Dropout. Re-init consistently reduces the instability regardless of the sources of the randomness.
C MIXED PRECISION TRAINING
Mixed precision training can accelerate model training while preserving performance by replacing some 32-bit floating-poin point computation with 16-bit floating-point computation. We use mixed precision training in all our experiments using huggingface's 's Transformers (Wolf et al. 2019). Transformers uses O1-level optimized mixed precision training implemented with the Apex library We evaluate if this mixed precision implementation influences our results. We fine-tune BERT with 20 random trials RTE, MRPC, STS-B, and CoLA. We two-tailed (-test test if the distributions of the two methods are statistically different. Table shows the mean and standard deviation of the
14

---

Published as a conference paper at ICLR 2021
CoLA
MRPC
RTE
STS-B
Mixed precision
60.3 1.5
89.2 = 1.2
71.8 2.1
90.1 0.7
Full precision
59.9 + 1.5
88.7 + 1.4
71.4 2.2
90.1 + 0.7
Table 4: Comparing BERT fine-tuning with mixed precision and full precision. The difference between the two numbers on any dataset statistically significant.
Dev Acc. (%)
Test Acc. (%)
No bias correction
86.0 + 0.3
87.0 + 0.4
Bias correction
85.9 + 0.3
86.9 + 0.3
Table 5: Comparing BERT fine-tuning with and without bias correction on the MNLI dataset. When we have a large dataset, there is no significant difference in using bias correction or not.
test performance. The performance of mixed precision matches the single precision counterpart, and there is no statistically significant difference.
D BIAS-CORRECTION ON MNLI
The focus of this paper is few-sample learning. However, we also experiment with the full MNLI dataset. Table shows that average accuracy three random runs. The results confirm that there is no significant difference in using bias correction or not on such a large dataset. While our recommended practices do improve training on large datasets, this result shows there is no disadvantage to fine-tune such models with the same procedure as we propose for few- sample training.
E SUPPLEMENTARY MATERIAL FOR SECTION 4
Effect of ADAM with Debiasing on Convergence. Figure shows the training loss as function of the number of training iterations. Using bias correction effectively speeds up convergence and reduces the range of the training loss, which is consistent with our observation in Figure
Effect of ADAM with Debiasing the Expected Validation Performance. Figure 12 shows the expected validation performance as function of the number of random trials. Comparing to Figure 4 we observe several trends. First, using ADAM with debiasing consistently leads to faster convergence and improved validation performance, which is similar to our observation about the test performance. Second, we observe that the expected validation performance monotonically increases with the number of random trials, contrary to our observation about the test performance. This suggests that using too many random trials may overfit the validation set and hurt generalization performance.
SUPPLEMENTARY MATERIAL FOR SECTION
Effect of L on Re-init Figure 13 shows the effect of Re-init in fine-tuning on the eight downsampled datasets. We observe similar trends in Figure 13 and Figure [S] Re-init's improvement is more pronounced in the wort-case performance across different random trials. Second, the best value of L different for each dataset.
Effect of Re-init on Model Parameters We use the same setup as in Figure to plot the change the weights of different Transformer blocks during fine-tuning on RTE, MRPC, STS-B, and CoLAA
in Figures 15 18
Effect of Re-init on Other Models We study more recent pre-trained contexual embedding models We investigate whether Re-init provides better fine-tuning initialization in XLNet Yang et al. (2019b), RoBERTa Liu et al. (2019c), BART Lewis et al. (2019). and ELECTRRAL. Clark al. (2020.. XLN autoregressive language model trained by learning
beyond BERT
15

---

Published as a conference paper at ICLR 2021
RTE
MRPC
STS-B
CoLA
Correction
0.75
0.6
0.75
No Correction
Trin
0.50
0.4
0.50
0.25
0.2
0.25
0.0
23
92
161
230
34
136
238
340
54
216
378
540
80
320
560
800
Steps
Steps
Steps
Steps
Figure 11: Mcan (solid lines) and range (shaded region) of training loss during fine-tuning BERT, across 50 random trials. Bias correction speeds up convergence and reduces the range of the training loss.
RTE
MRPC
STS-B
CoLA
0.80
0.95
0.92
0.70
Exp. Val MCC.
0.74
0.92
0.65
0.60
0.68
0.89
Val
0.90
Val
0.55
0.62
0.86
0.89
0.50
Correction
0.56
0.83
0.88
0.45
No Correction
0.50
20
0.80
20
0.87
20
20
of Random Trials
# of Random Trials
of Random Trials
of Random Trials
Figure 12: Expected validation performance (solid lines) with standard deviation (shaded region) over the number random trials allocated for fine-tuning BERT. With bias correction, we can reliably achieve good results with few (i.e., 5 or 10) random trials.
RIB
MRPC
STS-8
CoLA
0.66
0.900
Val. Rafemanne
Perfoomance
Perfommacce
0
Vil. Perfomamcce
0.64
0.00
0,000
060
0.841
0000
0.89
0.800
Standard Re
Gutic
0.835
0.50
Re Spandard Feoder
Re bek
3 Reink tni
lost Revins
Spcedard Re-inss Hander
Re lest
lnst lost o Re Hest
Stundadd Be
Howder
He-mik
Re Stundard Floder de Ses
Re lant
Revins
RTF
MRPC (IR)
STS-R (IR)
CIl.A
0.62
0.53
0.36
Val. Perfommance
0.700
Val. Perfommance
Perfommance
Val. Perfommance
0.60
0,575
0.86
0,56
0.58
9.625
0.94
0.56
0.600
0.80
0.50
Standard Re Re-in Rooder Be Re boss Re Re bosk Be Bast
Stondard Re Revent Hloder
Ri lest Be bost celllst Re Ri
Standard Be Pecker Re ferans Re Re- & Re Re Re-ias
0.32
Standand Ree lnst Peoler
Re lest Re
Re dase Re Rast
Re Re
SST (1k)
(IK
QQP (Ik)
MNLI
0.90
0.82
0.70
Val. Performence
Val. Performance
0.82
Performanes
0.8)
Val. Pefformane
0.6
0.58
0.50
6.00
0.76
0.79
0.76
0.50
0.84
0.45
Standard
unit Pooler
Standadd Re-imit Re Re Fe Resiig Redese Re
Standaad Be uas Resiil Re Re Resiet a Resiig Resill
Standard unil Pooler Re Re destee Re Re Ro 6
Be Ro
Re Re Ro
Figure 13: Validation performance distribution of re-initializing different number of layers of BERT on the downsampled datasets.
all permutations of natural language sentences. RoBERTa is similar to BERT in terms of model architecture but is only pre-trained the mask language modeling task only, but for longer and
16

---

Published as a conference paper at ICLR 2021
Model
Datasel
Learning Rate
Training Epochs / Steps
Batch Size
Warmuu Ratio/ Steps
LLRD
BERT
x 10-5
3 epochs
RTE
3 10-5
800 steps
32
200 steps
XLNet
MRPC
5 x 10
800 steps
32
200 steps
STS-B
x 10
3000 steps
32
500 steps
CoLA
3 x 10
1200 steps
128
120 steps
RTE
10-5
2036 steps
steps
RoBERTa
MRPC
1 10
2296 steps
137 stops
STS-B
X 10
3598 steps
214 steps
CoLAA
1 X 10
5336 steps
16
320 steps
RTE
x 10
10 epochs
10%
0.9
ELECTRA
MRPC
X 10-5
3 epochs
32
10%
0.9
STS-B
x
10 epochs
0.9
CoLA
X
3 epochs
RTE
1 10
1018 steps
32
61 steps
BART
MRPC
2 X 10
1148 steps
64
68 sleps
STS-B
2 X 10
1799 steps
32
107 steps
CoLA
2 X 10-5
1334 steps
64
80 steps
Table 6: Fine-tuning hyper-parameters of BERT and its variants reported in the official repository of each model.
RTE
MRPC
STS-B Re-init
CoLA
Standard
Re-init
Standard
Re-init
Standard
Standard
Re-init
XLNet
71.7
12.6
80.1
1.6
92.3 4.3
94.5
86.8 1 91.8
20.4 91.7 7
51.8 1 22.5
2.1
RoBERTa
78.2
1: 12.1
83.5
5
1 1.4
94.4
0.9
94.8
0.9
1 0.3
91.8
0.2
68
4 1 2.2
67.6 1.5
ELECTRA
87
1.2
86.1
1.9
95.7
95.3
0.8
91.8
1.9
92.
=
62
1 20.4
61.3 20.1
BART 84.1
+ 2.0
83.5
= 1.5
93.5
0.9
93.7
1.2
91.7
0.3
91.8
65.4 0.3
1.9
64.9
Table 7: Average Test performance with standard deviation on four small datasets with four different pre- rreitrained models. For each setting, the better numbers are bolded and in blue if the improvement statistically significant.
more data. BART is eqquence-to-sqquene model trained denoising autoencoder. ELECTRA is a BERT-like model trained to distinguish tokens generated by masked language model from tokens drawn from the natural distribution. Together, they represent diverse range of modeling choices in pre-training, including different model architectures, objectives, data, and training strategies. We use ADAM with debiasing to fine-tune these models on RTE, MRPC, STS-B, and COLA, using the hyperparameters that are either described in the paper or in the official repository of each model. Table(6ssummarizs the
of each model for each dataset. We use the huggingface's Transformers library (Wolf et al.) 2019). The experimental setup is kept the same our other experiments. Table 7 displays the average test performance on these datasets. We observe that several models suffer from high instability these datasets and in most cases, Re-init can reduce the performance variance. We observe that for some models, like XeNNet or RoBERTaa Re-init improve the average performance and reduce variance. However, the behavior of Re-init varies significantly across different models and Re-init have less significant improvement for ELECTRAL and BARTLagge Further study of the entire model family requires significant computational resources, and we leave it as an important direction for future work.
G SUPPLEMENTARY MATERIAL FOR SECTION 6
Figure 14 plots the validation performance as a function of the number training iteration, using the same setting in Figure Similar to observations in Figure we find that training longer generally improves fine-tuning performance and reduces the gap between standard fine-tuning and Re-init. On MNLI, Re-init still outperforms standard fine-tuning.
17

---

Published as a conference paper at ICLR 2021
SST (1k)
QNLI (1k)
QQP (1k)
MNLI (IK)
Val Performance
Performance
Val Performance
Parformance
0.90
0.82
0.80
0.6
0.80
0.88
Re-init
Re-init
Re-init
Re-init
Standard
Standard
Standard
Standard
0.75
Training Iterations
Training Iterations
Training Iterations
Training Iterations
Figure 14: Mean (solid lines) and range (shaded region) of validation performance trained with different number of iterations, across eight random trials.
H EXPERIMENTAL DETAILS IN SECTION 7
The hyperparameter search space allocated for each method our experiments is:
Layerwise Learning Rate Decay (LLRD) We grid search the initial learning rate in (2x10- 12x10-5,55 10-5, 1 x 10-4} and the layerwise decay rate {0.9, 0.95).
Mixout We tune the mixout probability p E (0.1, 0.3, 0.5, 0.7,
Weight decay toward the pre-trained weight We tune the regularization strength À {10 1002, 1000).
Weight decay
We tune the regularization strength À E (10-1, 110-3 1,00, 20-00110-1)
18

---

Published as a conference paper at ICLR 2021
Transformer Block
Transformer Block 2
Transformer Block 3
Transformer Block 4
L2 Dist. to Initiaizzaiin
to Inittaizzation
1.2 Dist to Initialization
Dist. to Initialization
0.8
0.8
0.8
0.6
0.6
0.4
0.4
0.4
0.4
Stundard Re-init
0.2
Re-init Re-init
0.2
0.2
0.2
Re-init 3
0.0
0.0
0.0
0.0
150
250
150
200
150
200
250
50
100
200
250
Steps
Steps
Steps
Steps
Transformer Block
Transformer Block
Transformer Block 7
Transformer Block 8
L. Dist. Initiaizzatin
Dist. Initialization
Dist. to Initialization
Dist. to Initialization
0.8
0.8
0.8
0.8
0.6
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2
0.0
0.0
0.0
0.0
0
50
100
150
200
250
0
50
100
150
200
250
0
50
100
150
200
250
0
50
100
150
200
250
Steps
Steps
Steps
Steps
Transtormer Block
Transformer Block 10
Transformer Block 11
Transformer Block 12
La Dist. Initialization
0.8
Dist. Initialization
0.8
Dist. to Initialization
0.8
Dist to Initialization
0.8
0.6
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.2
100
150
200
250
0
100
150
200
250
0
50
100
150
200
250
50
100
150
200
250
Steps
Steps
Steps
Steps
Transformer Block 13
Transformer Block 14
Transformer Block 15
Transformer Block 16
1. Dist, Initiaiizatoo
0.8
Dist. to Initializztion
0.8
Dist. to Initialization
0.8
Dist. to Initializztiinn
0.8
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.2
0.0
0.0
0.0
150
Steps
Steps
Steps
Steps
Transformer Block 17
Transformer Block 18
Transformer Block 19
Transformer Block 20
La Dist. Initialization
0.8
to Initialization
0.8
1.2 Dist. to Initialization
0.8
Dist. to Initialization
0.8
0.6
0.6
0.6
0.6
0.4
0.2
0.2
0.2
0.2
0.0
0.0
0.0
0.0
150
200
250
150
200
250
150
200
250
50
100
200
250
Steps
Steps
Steps
Steps
Transformer Block
Transformer Block 22
Transformer Block 23
Transformer Block 24
La Dist, Initializtion
0.8
1.2 Initialization
0.8
Dist. to Initializaaion
Dist. to Initialization
0.8
0.6
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.0
0.0
0.0
50
100
150
200
250
0
50
100
1.0
200
250
0
50
100
150 Steps
200
250
50
100
150 Steps
200
250
Steps
Steps
Figure 15: L2 distance to the initialization during fine-tuning BERT on RTE.
19

---

Published as a conference paper at ICLR 2021
L2 Dist. to Initiliizatio
1.00
Transformer Block
to Inittalization
1.00
Transformer Block
Dist to Inittalization
1.00
Transformer Block
Dist. to Initialization
1.00
Transformer Block 4
0.75
0.75
0.75
0.75
0.50
0.50
0.50
0.50
Standard
0.25
Re-init
Re-init 10
0.25
0.25
0.25
Re-init 3
0.00
0.00
0.00
0.00
148
222 Steps
296
370
74
148
222
296
370
74
148
222
296
370
74
148
223 Steps
296
370
Steps
Steps
Transformer Block
Transformer Block
Transformer Block
Transformer Block
1.2 Dist. Initializatio
Dist. to Initialization
1.00
1.00
Dist. to Initialization
1.00
Dist. to Initialization
1.00
0.75
0.75
0.75
0.75
0.50
0.50
0.50
0.50
0.25
0.25
0.25
0.25
0.00
0.00
0.00
0.00
0
74
148
222
296
370
74
148
222
296
370
74
148
222 Steps
296
370
74
148
222 Steps
296
370
Steps
Steps
Transformer Block
Transformer Block
Transformer Block
Transformer Block 12
La Dist. Initiaiizaionn
1.00
Dist. Initializtiinn
1.00
Dist. Initialization
1.00
Dist. to Initialization
1.00
0.75
0.75
0.75
0.50
0.50
0.50
0.50
0.25
0.25
0
148
222
296
370
74
148
222
296
370
74
148
222
296
370
74
148
222 Steps
296
370
Steps
Steps
Steps
Transformer Block 13
Transformer Block 14
Transformer Block 15
Transformer Block 16
1.2 Dist, Initialization
1.00
1.2 Dist. Initialization
1.00
1.2 Dist. to Initialization
1.00
Dist. to Initializaatin
1.00
0.75
0.50
0.50
0.50
0.50
0.25
0.25
0.25
0.25
0.00
0.00
0.00
0.00
296
Steps
Steps
Steps
Steps
Transformer Block 17
Transformer Block 18
Transformer Block 19
Transformer Block 20
L2 Dist. Initializatio
1.00
to Initialization
1.00
1.2 Dist. to Initialization
Dist. to Initialization
0.75
0.75
0.75
0.75
0.50
0.50
0.50
0.25
0.25
0.25
0.00
0.00
0.00
0.00
148
222
296
370
74
148
222
296
370
74
148
222
296
370
74
148
222 Steps
296
370
Steps
Steps
Steps
Transformer Block 21
Transformer Block
Transformer Block
Transformer Block
Dist. Initializatinn
Dist Initialization
Dist. to Initialization
Dist. to Initialization
1.00
0.8
0.75
0.75
0.6
0.50
0.4
0.25
0.2
0.25
0.25
0.00
0.0
0.00
0.00
74
148
222
296
370
74
148
222
296
370
74
148
222
296
370
74
148
222 Steps
296
370
Steps
Steps
Steps
Figure 16: L.2 distance to the initialization during fine-tuning BERT on MRPC.
20

---

Published as a conference paper at ICLR 2021
Transformer Block
Transformer Block
Transformer Block? 3
Transformer Block 4
1.2 to Initialization
L.2 Dist. to Imittalization
Dist. to Initialization
L2 Dist. to Inittalizzatin
1.0
1.0
1.0
1.0
Standard
0.5
0.5
0.5
Re-init Re-init Re-init
0.0
0.0
0.0
0.0
110 220 330 Steps
440
550
110
220
330
440
550
110
220
330
440
550
110
220
330 Steps
440 550
Steps
Steps
Transformer Block
Transformer Block
Transformer Block?
Transformer Block
1.2 Dist. Initialization
Dist. to Initialization
Dist. to Initializzation
Dist. to Initialization
1.0
1.0
1.0
1.0
0.5
0.5
0.5
0.5
0.0
0.0
0.0
0.0
110
220
330 Steps
440
550
110
220
320
440
550
0
110
220
330
440
550
0
110
220
330 Steps
440 550
Steps
Steps
Transformer Block
Transforner Block
Transtormer Block 11
Transformer Block 12
1.2 Dist. Initialization
Dist. Imitialzation
Dist. Initialization
L.2 Dist. to Initialization
1.0
1.0
1.0
0.5
0.5
0.5
0.5
110
220
330
440
550
110
220
330
440
550
110
220
330
440
550
0
110
220
330 Steps
440 550
Steps
Steps
Steps
1.2 Dist. Initialization
Transformer Block
13
Dist. Initialization
Transforner Block 14
1.2 Dist. to Initialization
Transformer Block 15
Dist. to Initialization
Transformer Block 16
1.0
1.0
1.00
1.00
0.5
0.50
0.50
0.25
0.0
0.0
0.00
0.00
330
550
Steps
Steps
Steps
Steps
L2 Dist. to Initialization
Transtormer Block 17
Initialization
Transformer Block 18
1.2 Dist. to Initialization
Transformer Block 19
Dist. to Initialization
Transformer Block 20
1.00
1.00
1.00
0.75
0.75
0.75
0.75
0.50
0.50
0.50
0.25
0.25
0.25
0.25
0.00
0.00
0.00
0.00
110
220
330
440 550
110
220
330
440
550
110
330
440 550
110
220
330 Steps
440
550
Steps
Steps
Steps
Dist. Initiaization
Transformer Block 21
1.2 Initialization
Transforner Block
Dist. to Initialization
Transformer Block 23
Dist. to Initialization
Transformer Block 24
1.00
1.00
1.0
0.75
0.75
0.75
0.50
0.50
0.50
0.5
0.25
0.25
0.25
0.00
0.00
0.00
0.0
110
220
330
440
550
110
220
330
440
550
0
110
220
330
440
550
0
110
220
330 Steps
440
550
Steps
Steps
Steps
Figure 17: L2 distance to the initialization during fine-tuning BERT on STS-B.
21

---

Published as a conference paper at ICLR 2021
Transformer Block!
Transformer Block
Transformer Block 3
Transformer Block 4
1.2 Dist to Initiaiization
1.5
L. to Immitalization
1.5
Dist. to Initialization
1.5
Dist. to Initialization
1.5
1.0
1.0
1.0
1.0
0.5
Standard Ae-iinit
0.5
0.5
0.5
Re-init
Re-init
Re-init 3
0.0
0.0
0.0
0.0
164 328
492 656 820 Steps
164
328
192 Steps
656
820
164
328
492
656
820
0
164
528
656 820 Steps
492
Steps
Transformer Block
Transformer Block
Transformer Block
Transformer Block
1.2 Dist. Initialization
1.5
Dist. to Initialization
1.5
Dist. to Initialization
1.5
Dist. to Initialization
1.5
1.0
1.0
1.0
1.0
0.5
0.5
0.5
0.5
0.0
0.0
0.0
0.0
164
328
492 656 Steps
820
0
164
328
492 Steps
656
820
0
164
328
492
656 820
0
164
328
656 820 Steps
492
Steps
Transformer Block
Transformer Block
Transformer Block 11
Transformer Block 12
1.2 Dist. Initialization
1.5
Dist. Initiaizationn
1.5
Dist. Initialization
1.5
1.2 Dist. to Initialization
1.5
1.0
1.0
1.0
1.0
0.5
0.5
0.5
164
328 492
656
820
164
328
492
656
820
164
328
492
656
820
0
164
328
492 Steps
656
820
Steps
Steps
Steps
Transformer Bluck
13
Transformer Block 14
Transformer Block 15
Transformer Block 16
1.2 Dist. Initialization
1.2 Dist. Initaalzation
1.2 Dist. to Initialization
L. Dist. to Initialization
1.0
1.0
1.0
1.0
0.5
0.5
0.5
0.5
0.0
0.0
0.0
0.0
164
164
Steps
Steps
Steps
Steps
L.2 Initialization
Transformer Block 17
Dist. to Initializtiin
Transformer Block 18
1.2 Dist. to Initialization
Transformer Block 19
Dist. to Initialization
Transformer Block 20
1.0
1.0
1.00
1.00
0.75
0.75
0.5
0.50
0.25
0.0
0.0
0.00
0.00
164
328
492 656 Steps
820
164
328
492
656
820
164
328
492 656
820
0
164
328
492 Steps
656
820
Steps
Steps
Dist. Initalization
1.00
Transformer Block
Initialization
Transformer Block
Dist. to Initialization
1.00
Transformer Block
Dist. to Initiaiiztiion
1.00
Transformer Block 24
0.75
0.75
0.50
0.50
0.25
0.25
0.25
0.25
0.00
1.2
0.00
0.00
0.00
164
328
492 Steps
656
820
0
164
328
492 656
820
0
164
328
492
656
820
0
164
328
656 820 Steps
492
Steps
Steps
Figure 18: L2 distance to the initialization during fine-tuning BERT on CoLAA
22


Reviews:
[{"review_id": "OAGXe-tusXs", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary\nThis paper investigates fine-tuning BERT for few-sample datasets. Notably, the authors find debiasing omission in BERT-adam. They find original debiased adam is better than BERT-adam. Besides, they also find re-initializing top layers can speed up learning and achieve better performance. These two findings are interesting. Another finding fine-tuning BERT for Longer is incremental to some extend.\n\n### Strengths\n * The two findings mentioned above are notable. \n\n* The authors conduct extensive experiments to support their claims.\n\n### Weaknesses and Questions\n* Table 1 shows the results of re-init but does not show re-init how many top layers for each task.\n* I suggest the authors can investigate debiased adam and re-init on the datasets with enough samples, like MNLI or QNLI. If they can achieve slight improvement or at least do not degrade the performance, we can just conveniently use the same fine-tuning method for most datasets.\n* Lack of explaining the meaning of Int. Task.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Es4HZNa83vJ", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a few tricks to improve the stability of BERT fine-turning, which include a standard Adam optimizer (with bias correction), the top BERT layers re-initiation and longer training. It provides extensive study on the GLUE benchmark showing how important these tricks are for small tasks (such as RTE/MRPC) which have less 1K training samples. The paper is well written and provides an insightful analysis.  \n\nAlthough it provides several useful tips for practitioners, it lacks novelty: for example the adam bias correction is from the original adam paper (also pointed it out by [2]) and training longer helping the performance is also observed by [1]. Gradient clipping may also help stabilize the training and it will be great to have a discussion as well. At last, does these approaches help large tasks, such as MNLI/QQP? It will be great to have a few settings: experiments on small or large tasks.  \n\n[1] Nakkiran et al, Deep double descent: where bigger models and more data hurt.\n[2] Mosbach et al, On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "DviyGBgd1TU", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper focuses on instability issues in BERT finetuning on small datasets. They list three factors which leads to instability, and provide simple fixes for each:\n1. Lack of bias correction term in BertAdam -- Fix was to use standard Adam \n2. Using all pretrained layers for finetuning -- Reinitializing the last few layers before finetuning. \n3. Training for a predefined number of epochs -- Train for a large number of epochs.\n\nThe fixes proposed reduces the variance in the results, and in most cases also improves performance. They also show that several proposed solutions to fix training instability lose their impact when the aforementioned fixes are incorporated. \n\nOverall, I like the paper; the observation about reinitializing top layers of BERT was interesting and counter intuitive to me; and I think this will be the most important contribution of the paper. \n\nAlthough not directly related to BERT, this paper (https://arxiv.org/pdf/1804.00247.pdf) also suggests training for longer epochs. This paper should be cited here. The tasks considered in the original BERT paper had large datasets, so I think the 2-3 epoch suggestion was tuned to those. \n\nThe result about BertAdam being unstable in low data settings, was a nice contribution. I feel this algorithm was also suggested considering the large datasets considered in the BERT paper.     \n\n ", "extra_scores": {}, "extra_text": {}}, {"review_id": "-5X8P9agrUX", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Large language models (LM) architectures, such as BERT, XLNet, etc., are not generally trained from scratch, but rather used as pretrained models. Among all, BERT is one of the most widely used ones, and its use on downstream tasks mainly consists on a stage of fine-tuning, where the new layers added are trained, and the rest of parameters of the network are left unfrozen, and hence are adjusted slightly to better fit the new task. However, this step of fine-tuning BERT is known to be quite unstable, and depends on a large set of factors, especially the initialization. Since the final performance on these downstream tasks can vary notably, different approaches has been proposed to circumvent this, but still the most common solution consists simply on choosing the best performing model, from a few random initialisation, using the validation set. \n\n##### Summary\nIn the current paper, the authors aim at tackling the aforementioned problem in a more grounded way. They investigate deeper the possible causes for these instabilities, and propose methods to counteract these pitfalls. Hence, they propose three simple, yet effective, approaches to stabilise the training and ensure better performances: a modified optimiser, the use of randomly initialised top layers, and more training steps. They provide a large collection of results, compare all these solutions to previous works, and discuss differences and similarities. Thanks to the analyses carried out, the current paper results in an exhaustive study on how “safely” fine-tune BERT, and the different factors that are to be taken into account when making use of these models.\n\n##### Strong and weak points\nI would like to start with the weakest point of the paper: it actually does not present anything clearly novel, nor innovative or groundbreaking. All the solutions proposed are inspired by previous approaches, or are just slight modifications of existing methods. But, this does not mean the paper is not valuable, as I do believe it is. The instability while fine-tuning large LMs on downstream tasks is a well known problem, but yet it has not been tackle exhaustively, and I do believe there does not exist clear guidelines and/or modifications that enable easily circumventing a critical weakness of these models. But I consider this paper succeeds at precisely this important task, thanks to the extended and exhaustive study it presents, and how it proposes three simple modifications that seem to solve this pitfall on most scenarios. \n\nBesides, the paper is quite well written, and presents in a clear manner the problems with the models, some intuition about the cause of those issues, and then, the solutions to overcome them. All the solutions are sufficiently justified, and are intuitive and simple. The latter, instead of being a weak point, for this precise problem it is more an advantage, as will allow an effortless adoption. Its improved performance is ensured thanks to the large set of benchmarks, on various datasets, the authors have compiled on the current manuscript. This is indeed another strong point, as all the solutions proposed are also tested under different conditions, with more or less training steps, and different numbers of top layers randomly initialised. \n\n##### Decision, and key reasons\nI believe the paper is ready to be accepted. Overall, it is an interesting and useful paper that will help many NLP researchers, and end-users of BERT, fine-tune better models, obtain improved performances, and therefore, start from a better baseline for their endeavours. And all this, with just some simple and intuitive modifications and guidelines. All the proposed methods and suggestions are not drawn from a few bunch of tests, but rather from a large collection of simulations, for different and varied datasets, with disparate starting conditions, and run over a fair amount of random initialisations. Therefore, I believe the authors have taken their time, and simulation time, to ensure that the presented results are robust and consistent, which is something to remark also.\n\n##### Questions and additional evidence\nAlthough I believe the paper is nicely written, and compiles all the required results and tests, I would appreciate if the authors could comment further on the following points:\n* I do believe there is a reason for not performing bias-correction on BERTAdam, and therefore, introducing it back might be affecting BERT training and fine-tuning in some specific, I guess negative, way. Could the authors comment on this? Or their understanding on why the correction was removed for BERTAdam.\n* In Figure 4, you suggest that with 5 to 10 random trials, the bias correction will achieve good results. However, observing the plots for all the datasets, we realise that indeed that number of random trials may benefit more the non-corrected version, as in most of the datasets the performance is either higher, or at least comparable. And although the variance is larger, we might still ensure at least a similar result . Could you comment on this? Would not be the corrected version a better option when no random initialisations are envisaged?\n* For the re-init, when just training for 3 epochs, it surprises me that indeed we could train the last 6 layers with just this reduced amount of data and training steps. And more surprisingly, according to Figures 14-16, is that the weights for these last 6 layers are the first to stabilise, even though they started from scratch, and they are supposed to be critical for the downstream tasks. Could you comment on this? I guess my understanding is wrong, and I would appreciate therefore some further insights. \n* Also, on the Re-init scheme, you mention that the number of layers to re-initialize depends on the task. Could you in any case offer here a general rule of thumb? \n\n##### Extra feedback\nFinally, I would like to conclude listing some small typos and errors I could spot in the manuscript:\n* Page 7, after Results, the reference to the Table is wrong.\n* Page 8, table 2: I believe the result for the RTE - Int. Task is mistype. I guess it should be something around 71.8.\n* Page 14, section E, Effect of Re-init… : the reference to the figure.\n* The caption for all figures 14 to 17 is wrong, as it should read fine-tuning.\n\nThese are the ones I could find, but it is not an exhaustive list. In any case, I would like to highlight the quality of the present manuscript, in terms of clearness and writing. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper addresses some of the well-documented instabilities that can arise from fine-tuning BERT on a dataset with few samples. Through a thorough investigation, they highlight various bizarre behaviors that have a negative impact on stability: First, that BERT inexplicably uses an unusual variant of Adam that, in fact, harms behavior; and second, that people tend to undertrain BERT on some downstream tasks. Separately, they find that reinitializing some of the final layers in BERT can be helpful. Since fine-tuning BERT has become such a common way to attack NLP problems, these practical recommendations will be quite welcome to the community. These findings address issues raised by recent work, so the paper is timely and relevant. The paper has through empirical analysis and is clear to read. There is a concurrent ICLR submission with similar findings, and this paper stands on its own. Reviewers all agreed that this paper should be published.
Decision: Accept (Poster)