Revisiting Few-sample BERT Fine-tuning

1 INTRODUCTION . Fine-tuning self-supervised pre-trained models has significantly boosted state-of-the-art performance on natural language processing ( NLP ) tasks ( Liu , 2019 ; Yang et al. , 2019a ; Wadden et al. , 2019 ; Zhu et al. , 2020 ; Guu et al. , 2020 ) . One of the most effective models for this process is BERT ( Devlin et al. , 2019 ) . However , despite significant success , fine-tuning remains unstable , especially when using the large variant of BERT ( BERTLarge ) on small datasets , where pre-training stands to provide the most significant benefit . Identical learning processes with different random seeds often result in significantly different and sometimes degenerate models following fine-tuning , even though only a few , seemingly insignificant aspects of the learning process are impacted by the random seed ( Phang et al. , 2018 ; Lee et al. , 2020 ; Dodge et al. , 2020 ) .1 As a result , practitioners resort to multiple random trials for model selection . This increases model deployment costs and time , and makes scientific comparison challenging ( Dodge et al. , 2020 ) . This paper is a study of different aspects of the few-sample fine-tuning optimization process . Our goal is to better understand the impact of common choices with regard to the optimization algorithm , model initialization , and the number of fine-tuning training iterations . We identify suboptimalities in common community practices : the use of a non-standard optimizer introduces bias in the gradient estimation ; the top layers of the pre-trained BERT model provide a bad initialization point for finetuning ; and the use of a pre-determined , but commonly adopted number of training iterations hurts convergence . We study these issues and their remedies through experiments on multiple common benchmarks , focusing on few-sample fine-tuning scenarios . Once these suboptimal practices are addressed , we observe that degenerate runs are eliminated and performance becomes much more stable . This makes it unnecessary to execute numerous random restarts as proposed in Dodge et al . ( 2020 ) . Our experiments show the remedies we experiment with for each issue have overlapping effect . For example , allocating more training iterations can eventually compensate for using the non-standard biased optimizer , even though the combination of a bias-corrected optimizer and re-initializing some of the pre-trained model parameters can reduce fine-tuning computational costs . This empirically highlights how different aspects of fine-tuning influence the stability of the process , at times in a similar manner . In the light of our observations , we re-evaluate several techniques ( Phang et al. , 2018 ; Lee et al. , 2020 ; Howard & Ruder , 2018 ) that ∗Equal contribution , 4 Work done at ASAPP . 1Fine-tuning instability is also receiving significant practitioner attention . For example : https : //github.com/zihangdai/xlnet/issues/96 and https : //github.com/huggingface/transformers/issues/265 . were recently proposed to increase few-sample fine-tuning stability and show a significant decrease in their impact . Our work furthers the empirical understanding of the fine-tuning process , and the optimization practices we outline identify impactful avenues for the development of future methods . 2 BACKGROUND AND RELATED WORK . BERT The Bidirectional Encoder Representations from Transformers ( BERT ; Devlin et al. , 2019 ) model is a Transformer encoder ( Vaswani et al. , 2017 ) trained on raw text using masked language modeling and next-sentence prediction objectives . It generates an embedding vector contextualized through a stack of Transformer blocks for each input token . BERT prepends a special [ CLS ] token to the input sentence or sentence pairs . The embedding of this token is used as a summary token for the input for classification tasks . This embedding is computed with an additional fully-connected layer with a tanh non-linearity , commonly referred to as the pooler , to aggregate the information for the [ CLS ] embedding . Fine-tuning The common approach for using the pre-trained BERT model is to replace the original output layer with a new task-specific layer and fine-tune the complete model . This includes learning the new output layer parameters and modifying all the original weights , including the weights of word embeddings , Transformer blocks , and the pooler . For example , for sentence-level classification , an added linear classifier projects the [ CLS ] embedding to an unnormalized probability vector over the output classes . This process introduces two sources of randomness : the weight initialization of the new output layer and the data order in the stochastic fine-tuning optimization . Existing work ( Phang et al. , 2018 ; Lee et al. , 2020 ; Dodge et al. , 2020 ) shows that these seemingly benign factors can influence the results significantly , especially on small datasets ( i.e. , < 10K examples ) . Consequently , practitioners often conduct many random trials of fine-tuning and pick the best model based on validation performance ( Devlin et al. , 2019 ) . Fine-tuning Instability The instability of the BERT fine-tuning process has been known since its introduction ( Devlin et al. , 2019 ) , and various methods have been proposed to address it . Phang et al . ( 2018 ) show that fine-tuning the pre-trained model on a large intermediate task stabilizes later fine-tuning on small datasets . Lee et al . ( 2020 ) introduce a new regularization method to constrain the fine-tuned model to stay close to the pre-trained weights and show that it stabilizes fine-tuning . Dodge et al . ( 2020 ) propose an early stopping method to efficiently filter out random seeds likely to lead to bad performance . Concurrently to our work , Mosbach et al . ( 2020 ) also show that BERTADAM leads to instability during fine-tuning . Our experiments studying the effect of training longer are related to previous work studying this question in the context of training models from scratch ( Popel & Bojar , 2018 ; Nakkiran et al. , 2019 ) . BERT Representation Transferability BERT pre-trained representations have been widely studied using probing methods showing that the pre-trained features from intermediate layers are more transferable ( Tenney et al. , 2019b ; a ; Liu et al. , 2019a ; Hewitt & Manning , 2019 ; Hewitt & Liang , 2019 ) or applicable ( Zhang et al. , 2020 ) to new tasks than features from later layers , which change more after fine-tuning ( Peters et al. , 2019 ; Merchant et al. , 2020 ) . Our work is inspired by these findings , but focuses on studying how the pre-trained weights influence the fine-tuning process . Li et al . ( 2020 ) propose to re-initialize the final fully-connected layer of a ConvNet and show performance gain for image classification.2 Concurrent to our work , Tamkin et al . ( 2020 ) adopt a similar methodology of weight re-initialization ( Section 5 ) to study the transferability of BERT . In contrast to our study , their work emphasizes pinpointing the layers that contribute the most in transfer learning , and the relation between probing performance and transferability . 3 EXPERIMENTAL METHODOLOGY . Data We follow the data setup of previous studies ( Lee et al. , 2020 ; Phang et al. , 2018 ; Dodge et al. , 2020 ) to study few-sample fine-tuning using eight datasets from the GLUE benchmark ( Wang et al. , 2019b ) . The datasets cover four tasks : natural language inference ( RTE , QNLI , MNLI ) , paraphrase detection ( MRPC , QQP ) , sentiment classification ( SST-2 ) , and linguistic acceptability ( CoLA ) . Appendix A provides dataset statistics and a description of each dataset . We primarily 2This concurrent work was published shortly after our study was posted . Algorithm 1 : the ADAM pseudocode adapted from Kingma & Ba ( 2014 ) , and provided for reference . g2t denotes the elementwise square gt gt . β1 and β2 to the power t are denoted as βt1 β t 2 . All operations on vectors are element-wise . The suggested hyperparameter values according to Kingma & Ba ( 2014 ) are : α = 0.001 , β1 = 0.9 , β2 = 0.999 , and = 10−8 . BERTADAM ( Devlin et al. , 2019 ) omits the bias correction ( lines 9–10 ) , and treats mt and vt as m̂t and v̂t in line 11 . Require : α : learning rate ; β1 , β2 ∈ [ 0 , 1 ) : exponential decay rates for the moment estimates ; f ( θ ) : stochastic objective function with parameters θ ; θ0 : initial parameter vector ; λ ∈ [ 0 , 1 ) : decoupled weight decay . 1 : m0 ← 0 ( Initialize first moment vector ) 2 : v0 ← 0 ( Initialize second moment vector ) 3 : t← 0 ( Initialize timestep ) 4 : while θt not converged do 5 : t← t+ 1 6 : gt ← ∇θft ( θt−1 ) ( Get gradients w.r.t . stochastic objective at timestep t ) 7 : mt ← β1 ·mt−1 + ( 1− β1 ) · gt ( Update biased first moment estimate ) 8 : vt ← β2 · vt−1 + ( 1− β2 ) · g2t ( Update biased second raw moment estimate ) 9 : m̂t ← mt/ ( 1− βt1 ) ( Compute bias-corrected first moment estimate ) 10 : v̂t ← vt/ ( 1− βt2 ) ( Compute bias-corrected second raw moment estimate ) 11 : θt ← θt−1 − α · m̂t/ ( √ v̂t + ) ( Update parameters ) 12 : end while 13 : return θt ( Resulting parameters ) focus on four datasets ( RTE , MRPC , STS-B , CoLA ) that have fewer than 10k training samples , because BERT fine-tuning on these datasets is known to be unstable ( Devlin et al. , 2019 ) . We also complement our study by downsampling all eight datasets to 1k training examples following Phang et al . ( 2018 ) . While previous studies ( Lee et al. , 2020 ; Phang et al. , 2018 ; Dodge et al. , 2020 ) focus on the validation performance , we split held-out test sets for our study.3 For RTE , MRPC , STS-B , and CoLA , we divide the original validation set in half , using one half for validation and the other for test . For the other four larger datasets , we only study the downsampled versions , and split additional 1k samples from the training set as our validation data and test on the original validation set . Experimental Setup Unless noted otherwise , we follow the hyperparameter setup of Lee et al . ( 2020 ) . We fine-tune the uncased , 24-layer BERTLarge model with batch size 32 , dropout 0.1 , and peak learning rate 2 × 10−5 for three epochs . We clip the gradients to have a maximum norm of 1 . We apply linear learning rate warm-up during the first 10 % of the updates followed by a linear decay . We use mixed precision training using Apex4 to speed up experiments . We show that mixed precision training does not affect fine-tuning performance in Appendix C. We evaluate ten times on the validation set during training and perform early stopping . We fine-tune with 20 random seeds to compare different settings . 4 OPTIMIZATION ALGORITHM : DEBIASING OMISSION IN BERTADAM . The most commonly used optimizer for fine-tuning BERT is BERTADAM , a modified version of the ADAM first-order stochastic optimization method . It differs from the original ADAM algorithm ( Kingma & Ba , 2014 ) in omitting a bias correction step . This change was introduced by Devlin et al . ( 2019 ) , and subsequently made its way into common open source libraries , including the official implementation,5 huggingface ’ s Transformers ( Wolf et al. , 2019 ) ,6 AllenNLP ( Gardner et al. , 2018 ) , GluonNLP ( Guo et al. , 2019 ) , jiant ( Wang et al. , 2019c ) , MT-DNN ( Liu et al. , 2020 ) , and FARM.7 As a result , this non-standard implementation is widely used in both industry and research ( Wang et al. , 2019a ; Phang et al. , 2018 ; Lee et al. , 2020 ; Dodge et al. , 2020 ; Sun et al. , 2019 ; Clark et al. , 2020 ; Lan et al. , 2020 ; Houlsby et al. , 2019 ; Stickland & Murray , 2019 ; Liu et al. , 2019b ) . We observe that the bias correction omission influences the learning rate , especially early in the fine-tuning process , and is one of the primary reasons for instability in fine-tuning BERT ( Devlin et al. , 2019 ; Phang et al. , 2018 ; Lee et al. , 2020 ; Dodge et al. , 2020 ) . Algorithm 1 shows the ADAM algorithm , and highlights the omitted line in the non-standard BERTADAM implementation . At each optimization step ( lines 4–11 ) , ADAM computes the exponen- 3The original test sets are not publicly available . 4https : //github.com/NVIDIA/apex 5https : //github.com/google-research/bert/blob/f39e881/optimization.py # L108-L157 6The default was changed from BERTADAM to debiased ADAM in commit ec07cf5a on July 11 , 2019 . 7https : //github.com/deepset-ai/FARM 3 tial moving average of the gradients ( mt ) and the squared gradients ( vt ) , where β1 , β2 parameterize the averaging ( lines 7–8 ) . Because ADAM initializes mt and vt to 0 and sets exponential decay rates β1 and β2 close to 1 , the estimates of mt and vt are heavily biased towards 0 early during learning when t is small . Kingma & Ba ( 2014 ) computes the ratio between the biased and the unbiased estimates of mt and vt as ( 1− βt1 ) and ( 1− βt2 ) . This ratio is independent of the training data . The model parameters θ are updated in the direction of the averaged gradient mt divided by the square root of the second moment √ vt ( line 11 ) . BERTADAM omits the debiasing ( lines 9–10 ) , and directly uses the biased estimates in the parameters update . Figure 1 shows the ratio m̂t√ v̂t between the update using the biased and the unbiased estimation as a function of training iterations . The bias is relatively high early during learning , indicating overestimation . It eventually converges to one , suggesting that when training for sufficient iterations , the estimation bias will have negligible effect.8 Therefore , the bias ratio term is most important early during learning to counteract the overestimation of mt and vt during early iterations . In practice , ADAM adaptively re-scales the learning rate by √ 1−βt2 1−βt1 . This correction is crucial for BERT finetuning on small datasets with fewer than 10k training samples because they are typically fine-tuned with less than 1k iterations ( Devlin et al. , 2019 ) . The figure shows the number of training iterations for RTE , MRPC , STS-B , CoLA , and MNLI . MNLI is the only one of this set with a large number of supervised training examples . For small datasets , the bias ratio is significantly higher than one for the entire fine-tuning process , implying that these datasets suffer heavily from overestimation in the update magnitude . In comparison , for MNLI , the majority of fine-tuning occurs in the region where the bias ratio has converged to one . This explains why fine-tuning on MNLI is known to be relatively stable ( Devlin et al. , 2019 ) . We evaluate the importance of the debiasing step empirically by fine-tuning BERT with both BERTADAM and the debiased ADAM9 for 50 random seeds on RTE , MRPC , STS-B , and CoLA . Figure 2 summarizes the performance distribution . The bias correction significantly reduces the performance variance across different random trials and the four datasets . Without the bias correction we observe many degenerate runs , where fine-tuned models fail to outperform the random baseline . For example , on RTE , 48 % of fine-tuning runs have an accuracy less than 55 % , which is close to random guessing . Figure 3 further illustrates this difference by plotting the mean and the range of training loss during fine-tuning across different random trials on RTE . Figure 11 in Appendix F shows similar plots for MRPC , STS-B , and CoLA . The biased BERTADAM consistently leads to worse averaged training loss , and on all datasets to higher maximum training loss . This indicates models trained with BERTAdam are underfitting and the root of instability lies in optimization . 8Our experiments on the completely MNLI dataset confirm using the unbiased estimation does not improve nor degrade performance for large datasets ( Appendix D ) . 9We use the PyTorch ADAM implementation https : //pytorch.org/docs/1.4.0/_modules/torch/optim/adamw.html . We simulate a realistic setting of multiple random trials following Dodge et al . ( 2020 ) . We use bootstrapping for the simulation : given the 50 fine-tuned models we trained , we sample models with replacement , perform model selection on the validation set , and record the test results ; we repeat this process 1k times to estimate mean and variance . Figure 4 shows the simulated test results as a function of the number of random trials . Appendix E provides the same plots for validation performance . Using the debiased ADAM we can reliably achieve good results using fewer random trials ; the difference in expected performance is especially pronounced when we perform less than 10 trials . Whereas the expected validation performance monotonically improves with more random trials ( Dodge et al. , 2020 ) , the expected test performance deteriorates when we perform too many random trials because the model selection process potentially overfits the validation set . Based on these observations , we recommend performing a moderate number of random trials ( i.e. , 5 or 10 ) .

Summary: Large language models (LM) architectures, such as BERT, XLNet, etc., are not generally trained from scratch, but rather used as pretrained models. Among all, BERT is one of the most widely used ones, and its use on downstream tasks mainly consists on a stage of fine-tuning, where the new layers added are trained, and the rest of parameters of the network are left unfrozen, and hence are adjusted slightly to better fit the new task. However, this step of fine-tuning BERT is known to be quite unstable, and depends on a large set of factors, especially the initialization. Since the final performance on these downstream tasks can vary notably, different approaches has been proposed to circumvent this, but still the most common solution consists simply on choosing the best performing model, from a few random initialisation, using the validation set.