Revisiting Few-sample BERT Fine-tuning

1 INTRODUCTION . Fine-tuning self-supervised pre-trained models has significantly boosted state-of-the-art performance on natural language processing ( NLP ) tasks ( Liu , 2019 ; Yang et al. , 2019a ; Wadden et al. , 2019 ; Zhu et al. , 2020 ; Guu et al. , 2020 ) . One of the most effective models for this process is BERT ( Devlin et al. , 2019 ) . However , despite significant success , fine-tuning remains unstable , especially when using the large variant of BERT ( BERTLarge ) on small datasets , where pre-training stands to provide the most significant benefit . Identical learning processes with different random seeds often result in significantly different and sometimes degenerate models following fine-tuning , even though only a few , seemingly insignificant aspects of the learning process are impacted by the random seed ( Phang et al. , 2018 ; Lee et al. , 2020 ; Dodge et al. , 2020 ) .1 As a result , practitioners resort to multiple random trials for model selection . This increases model deployment costs and time , and makes scientific comparison challenging ( Dodge et al. , 2020 ) . This paper is a study of different aspects of the few-sample fine-tuning optimization process . Our goal is to better understand the impact of common choices with regard to the optimization algorithm , model initialization , and the number of fine-tuning training iterations . We identify suboptimalities in common community practices : the use of a non-standard optimizer introduces bias in the gradient estimation ; the top layers of the pre-trained BERT model provide a bad initialization point for finetuning ; and the use of a pre-determined , but commonly adopted number of training iterations hurts convergence . We study these issues and their remedies through experiments on multiple common benchmarks , focusing on few-sample fine-tuning scenarios . Once these suboptimal practices are addressed , we observe that degenerate runs are eliminated and performance becomes much more stable . This makes it unnecessary to execute numerous random restarts as proposed in Dodge et al . ( 2020 ) . Our experiments show the remedies we experiment with for each issue have overlapping effect . For example , allocating more training iterations can eventually compensate for using the non-standard biased optimizer , even though the combination of a bias-corrected optimizer and re-initializing some of the pre-trained model parameters can reduce fine-tuning computational costs . This empirically highlights how different aspects of fine-tuning influence the stability of the process , at times in a similar manner . In the light of our observations , we re-evaluate several techniques ( Phang et al. , 2018 ; Lee et al. , 2020 ; Howard & Ruder , 2018 ) that ∗Equal contribution , 4 Work done at ASAPP . 1Fine-tuning instability is also receiving significant practitioner attention . For example : https : //github.com/zihangdai/xlnet/issues/96 and https : //github.com/huggingface/transformers/issues/265 . were recently proposed to increase few-sample fine-tuning stability and show a significant decrease in their impact . Our work furthers the empirical understanding of the fine-tuning process , and the optimization practices we outline identify impactful avenues for the development of future methods . 2 BACKGROUND AND RELATED WORK . BERT The Bidirectional Encoder Representations from Transformers ( BERT ; Devlin et al. , 2019 ) model is a Transformer encoder ( Vaswani et al. , 2017 ) trained on raw text using masked language modeling and next-sentence prediction objectives . It generates an embedding vector contextualized through a stack of Transformer blocks for each input token . BERT prepends a special [ CLS ] token to the input sentence or sentence pairs . The embedding of this token is used as a summary token for the input for classification tasks . This embedding is computed with an additional fully-connected layer with a tanh non-linearity , commonly referred to as the pooler , to aggregate the information for the [ CLS ] embedding . Fine-tuning The common approach for using the pre-trained BERT model is to replace the original output layer with a new task-specific layer and fine-tune the complete model . This includes learning the new output layer parameters and modifying all the original weights , including the weights of word embeddings , Transformer blocks , and the pooler . For example , for sentence-level classification , an added linear classifier projects the [ CLS ] embedding to an unnormalized probability vector over the output classes . This process introduces two sources of randomness : the weight initialization of the new output layer and the data order in the stochastic fine-tuning optimization . Existing work ( Phang et al. , 2018 ; Lee et al. , 2020 ; Dodge et al. , 2020 ) shows that these seemingly benign factors can influence the results significantly , especially on small datasets ( i.e. , < 10K examples ) . Consequently , practitioners often conduct many random trials of fine-tuning and pick the best model based on validation performance ( Devlin et al. , 2019 ) . Fine-tuning Instability The instability of the BERT fine-tuning process has been known since its introduction ( Devlin et al. , 2019 ) , and various methods have been proposed to address it . Phang et al . ( 2018 ) show that fine-tuning the pre-trained model on a large intermediate task stabilizes later fine-tuning on small datasets . Lee et al . ( 2020 ) introduce a new regularization method to constrain the fine-tuned model to stay close to the pre-trained weights and show that it stabilizes fine-tuning . Dodge et al . ( 2020 ) propose an early stopping method to efficiently filter out random seeds likely to lead to bad performance . Concurrently to our work , Mosbach et al . ( 2020 ) also show that BERTADAM leads to instability during fine-tuning . Our experiments studying the effect of training longer are related to previous work studying this question in the context of training models from scratch ( Popel & Bojar , 2018 ; Nakkiran et al. , 2019 ) . BERT Representation Transferability BERT pre-trained representations have been widely studied using probing methods showing that the pre-trained features from intermediate layers are more transferable ( Tenney et al. , 2019b ; a ; Liu et al. , 2019a ; Hewitt & Manning , 2019 ; Hewitt & Liang , 2019 ) or applicable ( Zhang et al. , 2020 ) to new tasks than features from later layers , which change more after fine-tuning ( Peters et al. , 2019 ; Merchant et al. , 2020 ) . Our work is inspired by these findings , but focuses on studying how the pre-trained weights influence the fine-tuning process . Li et al . ( 2020 ) propose to re-initialize the final fully-connected layer of a ConvNet and show performance gain for image classification.2 Concurrent to our work , Tamkin et al . ( 2020 ) adopt a similar methodology of weight re-initialization ( Section 5 ) to study the transferability of BERT . In contrast to our study , their work emphasizes pinpointing the layers that contribute the most in transfer learning , and the relation between probing performance and transferability . 3 EXPERIMENTAL METHODOLOGY . Data We follow the data setup of previous studies ( Lee et al. , 2020 ; Phang et al. , 2018 ; Dodge et al. , 2020 ) to study few-sample fine-tuning using eight datasets from the GLUE benchmark ( Wang et al. , 2019b ) . The datasets cover four tasks : natural language inference ( RTE , QNLI , MNLI ) , paraphrase detection ( MRPC , QQP ) , sentiment classification ( SST-2 ) , and linguistic acceptability ( CoLA ) . Appendix A provides dataset statistics and a description of each dataset . We primarily 2This concurrent work was published shortly after our study was posted . Algorithm 1 : the ADAM pseudocode adapted from Kingma & Ba ( 2014 ) , and provided for reference . g2t denotes the elementwise square gt gt . β1 and β2 to the power t are denoted as βt1 β t 2 . All operations on vectors are element-wise . The suggested hyperparameter values according to Kingma & Ba ( 2014 ) are : α = 0.001 , β1 = 0.9 , β2 = 0.999 , and = 10−8 . BERTADAM ( Devlin et al. , 2019 ) omits the bias correction ( lines 9–10 ) , and treats mt and vt as m̂t and v̂t in line 11 . Require : α : learning rate ; β1 , β2 ∈ [ 0 , 1 ) : exponential decay rates for the moment estimates ; f ( θ ) : stochastic objective function with parameters θ ; θ0 : initial parameter vector ; λ ∈ [ 0 , 1 ) : decoupled weight decay . 1 : m0 ← 0 ( Initialize first moment vector ) 2 : v0 ← 0 ( Initialize second moment vector ) 3 : t← 0 ( Initialize timestep ) 4 : while θt not converged do 5 : t← t+ 1 6 : gt ← ∇θft ( θt−1 ) ( Get gradients w.r.t . stochastic objective at timestep t ) 7 : mt ← β1 ·mt−1 + ( 1− β1 ) · gt ( Update biased first moment estimate ) 8 : vt ← β2 · vt−1 + ( 1− β2 ) · g2t ( Update biased second raw moment estimate ) 9 : m̂t ← mt/ ( 1− βt1 ) ( Compute bias-corrected first moment estimate ) 10 : v̂t ← vt/ ( 1− βt2 ) ( Compute bias-corrected second raw moment estimate ) 11 : θt ← θt−1 − α · m̂t/ ( √ v̂t + ) ( Update parameters ) 12 : end while 13 : return θt ( Resulting parameters ) focus on four datasets ( RTE , MRPC , STS-B , CoLA ) that have fewer than 10k training samples , because BERT fine-tuning on these datasets is known to be unstable ( Devlin et al. , 2019 ) . We also complement our study by downsampling all eight datasets to 1k training examples following Phang et al . ( 2018 ) . While previous studies ( Lee et al. , 2020 ; Phang et al. , 2018 ; Dodge et al. , 2020 ) focus on the validation performance , we split held-out test sets for our study.3 For RTE , MRPC , STS-B , and CoLA , we divide the original validation set in half , using one half for validation and the other for test . For the other four larger datasets , we only study the downsampled versions , and split additional 1k samples from the training set as our validation data and test on the original validation set . Experimental Setup Unless noted otherwise , we follow the hyperparameter setup of Lee et al . ( 2020 ) . We fine-tune the uncased , 24-layer BERTLarge model with batch size 32 , dropout 0.1 , and peak learning rate 2 × 10−5 for three epochs . We clip the gradients to have a maximum norm of 1 . We apply linear learning rate warm-up during the first 10 % of the updates followed by a linear decay . We use mixed precision training using Apex4 to speed up experiments . We show that mixed precision training does not affect fine-tuning performance in Appendix C. We evaluate ten times on the validation set during training and perform early stopping . We fine-tune with 20 random seeds to compare different settings . 4 OPTIMIZATION ALGORITHM : DEBIASING OMISSION IN BERTADAM . The most commonly used optimizer for fine-tuning BERT is BERTADAM , a modified version of the ADAM first-order stochastic optimization method . It differs from the original ADAM algorithm ( Kingma & Ba , 2014 ) in omitting a bias correction step . This change was introduced by Devlin et al . ( 2019 ) , and subsequently made its way into common open source libraries , including the official implementation,5 huggingface ’ s Transformers ( Wolf et al. , 2019 ) ,6 AllenNLP ( Gardner et al. , 2018 ) , GluonNLP ( Guo et al. , 2019 ) , jiant ( Wang et al. , 2019c ) , MT-DNN ( Liu et al. , 2020 ) , and FARM.7 As a result , this non-standard implementation is widely used in both industry and research ( Wang et al. , 2019a ; Phang et al. , 2018 ; Lee et al. , 2020 ; Dodge et al. , 2020 ; Sun et al. , 2019 ; Clark et al. , 2020 ; Lan et al. , 2020 ; Houlsby et al. , 2019 ; Stickland & Murray , 2019 ; Liu et al. , 2019b ) . We observe that the bias correction omission influences the learning rate , especially early in the fine-tuning process , and is one of the primary reasons for instability in fine-tuning BERT ( Devlin et al. , 2019 ; Phang et al. , 2018 ; Lee et al. , 2020 ; Dodge et al. , 2020 ) . Algorithm 1 shows the ADAM algorithm , and highlights the omitted line in the non-standard BERTADAM implementation . At each optimization step ( lines 4–11 ) , ADAM computes the exponen- 3The original test sets are not publicly available . 4https : //github.com/NVIDIA/apex 5https : //github.com/google-research/bert/blob/f39e881/optimization.py # L108-L157 6The default was changed from BERTADAM to debiased ADAM in commit ec07cf5a on July 11 , 2019 . 7https : //github.com/deepset-ai/FARM 3 tial moving average of the gradients ( mt ) and the squared gradients ( vt ) , where β1 , β2 parameterize the averaging ( lines 7–8 ) . Because ADAM initializes mt and vt to 0 and sets exponential decay rates β1 and β2 close to 1 , the estimates of mt and vt are heavily biased towards 0 early during learning when t is small . Kingma & Ba ( 2014 ) computes the ratio between the biased and the unbiased estimates of mt and vt as ( 1− βt1 ) and ( 1− βt2 ) . This ratio is independent of the training data . The model parameters θ are updated in the direction of the averaged gradient mt divided by the square root of the second moment √ vt ( line 11 ) . BERTADAM omits the debiasing ( lines 9–10 ) , and directly uses the biased estimates in the parameters update . Figure 1 shows the ratio m̂t√ v̂t between the update using the biased and the unbiased estimation as a function of training iterations . The bias is relatively high early during learning , indicating overestimation . It eventually converges to one , suggesting that when training for sufficient iterations , the estimation bias will have negligible effect.8 Therefore , the bias ratio term is most important early during learning to counteract the overestimation of mt and vt during early iterations . In practice , ADAM adaptively re-scales the learning rate by √ 1−βt2 1−βt1 . This correction is crucial for BERT finetuning on small datasets with fewer than 10k training samples because they are typically fine-tuned with less than 1k iterations ( Devlin et al. , 2019 ) . The figure shows the number of training iterations for RTE , MRPC , STS-B , CoLA , and MNLI . MNLI is the only one of this set with a large number of supervised training examples . For small datasets , the bias ratio is significantly higher than one for the entire fine-tuning process , implying that these datasets suffer heavily from overestimation in the update magnitude . In comparison , for MNLI , the majority of fine-tuning occurs in the region where the bias ratio has converged to one . This explains why fine-tuning on MNLI is known to be relatively stable ( Devlin et al. , 2019 ) . We evaluate the importance of the debiasing step empirically by fine-tuning BERT with both BERTADAM and the debiased ADAM9 for 50 random seeds on RTE , MRPC , STS-B , and CoLA . Figure 2 summarizes the performance distribution . The bias correction significantly reduces the performance variance across different random trials and the four datasets . Without the bias correction we observe many degenerate runs , where fine-tuned models fail to outperform the random baseline . For example , on RTE , 48 % of fine-tuning runs have an accuracy less than 55 % , which is close to random guessing . Figure 3 further illustrates this difference by plotting the mean and the range of training loss during fine-tuning across different random trials on RTE . Figure 11 in Appendix F shows similar plots for MRPC , STS-B , and CoLA . The biased BERTADAM consistently leads to worse averaged training loss , and on all datasets to higher maximum training loss . This indicates models trained with BERTAdam are underfitting and the root of instability lies in optimization . 8Our experiments on the completely MNLI dataset confirm using the unbiased estimation does not improve nor degrade performance for large datasets ( Appendix D ) . 9We use the PyTorch ADAM implementation https : //pytorch.org/docs/1.4.0/_modules/torch/optim/adamw.html . We simulate a realistic setting of multiple random trials following Dodge et al . ( 2020 ) . We use bootstrapping for the simulation : given the 50 fine-tuned models we trained , we sample models with replacement , perform model selection on the validation set , and record the test results ; we repeat this process 1k times to estimate mean and variance . Figure 4 shows the simulated test results as a function of the number of random trials . Appendix E provides the same plots for validation performance . Using the debiased ADAM we can reliably achieve good results using fewer random trials ; the difference in expected performance is especially pronounced when we perform less than 10 trials . Whereas the expected validation performance monotonically improves with more random trials ( Dodge et al. , 2020 ) , the expected test performance deteriorates when we perform too many random trials because the model selection process potentially overfits the validation set . Based on these observations , we recommend performing a moderate number of random trials ( i.e. , 5 or 10 ) .

Summary: This paper investigates fine-tuning BERT for few-sample datasets. Notably, the authors find debiasing omission in BERT-adam. They find original debiased adam is better than BERT-adam. Besides, they also find re-initializing top layers can speed up learning and achieve better performance. These two findings are interesting. Another finding fine-tuning BERT for Longer is incremental to some extend.