On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines

1 INTRODUCTION . Pre-trained transformer-based masked language models such as BERT ( Devlin et al. , 2019 ) , RoBERTa ( Liu et al. , 2019 ) , and ALBERT ( Lan et al. , 2020 ) have had a dramatic impact on the NLP landscape in the recent year . The standard recipe for using such models typically involves training a pretrained model for a few epochs on a supervised downstream dataset , which is known as fine-tuning . While fine-tuning has led to impressive empirical results , dominating a large variety of English NLP benchmarks such as GLUE ( Wang et al. , 2019b ) and SuperGLUE ( Wang et al. , 2019a ) , it is still poorly understood . Not only have fine-tuned models been shown to pick up spurious patterns and biases present in the training data ( Niven and Kao , 2019 ; McCoy et al. , 2019 ) , but also to exhibit a large training instability : fine-tuning a model multiple times on the same dataset , varying only the random seed , leads to a large standard deviation of the fine-tuning accuracy ( Devlin et al. , 2019 ; Dodge et al. , 2020 ) . Few methods have been proposed to solve the observed instability ( Phang et al. , 2018 ; Lee et al. , 2020 ) , however without providing a sufficient understanding of why fine-tuning is prone to such failure . The goal of this work is to address this shortcoming . More specifically , we investigate the following question : Why is fine-tuning prone to failures and how can we improve its stability ? We start by investigating two common hypotheses for fine-tuning instability : catastrophic forgetting and small size of the fine-tuning datasets and demonstrate that both hypotheses fail to explain fine-tuning instability . We then investigate fine-tuning failures on datasets from the popular GLUE benchmark and show that the observed fine-tuning instability can be decomposed into two separate aspects : ( 1 ) optimization difficulties early in training , characterized by vanishing gradients , and ( 2 ) differences in generalization late in training , characterized by a large variance of development set accuracy for runs with almost equivalent training loss . Based on our analysis , we present a simple but strong baseline for fine-tuning pre-trained language models that significantly improves the fine-tuning stability compared to previous works ( Fig . 1 ) . Moreover , we show that our findings apply not only to the widely used BERT model but also to more recent pre-trained models such as RoBERTa and ALBERT . 2 RELATED WORK . The fine-tuning instability of BERT has been pointed out in various studies . Devlin et al . ( 2019 ) report instabilities when fine-tuning BERTLARGE on small datasets and resort to performing multiple restarts of fine-tuning and selecting the model that performs best on the development set . Recently , Dodge et al . ( 2020 ) performed a large-scale empirical investigation of the fine-tuning instability of BERT . They found dramatic variations in fine-tuning accuracy across multiple restarts and argue how it might be related to the choice of random seed and the dataset size . Few approaches have been proposed to directly address the observed fine-tuning instability . Phang et al . ( 2018 ) study intermediate task training ( STILTS ) before fine-tuning with the goal of improving performance on the GLUE benchmark . They also find that their proposed method leads to improved fine-tuning stability . However , due to the intermediate task training , their work is not directly comparable to ours . Lee et al . ( 2020 ) propose a new regularization technique termed Mixout . The authors show that Mixout improves stability during fine-tuning which they attribute to the prevention of catastrophic forgetting . Another line of work investigates optimization difficulties of pre-training transformer-based language models ( Xiong et al. , 2020 ; Liu et al. , 2020 ) . Similar to our work , they highlight the importance of the learning rate warmup for optimization . Both works focus on pre-training and we hence view them as orthogonal to our work . 3 BACKGROUND . 3.1 DATASETS . We study four datasets from the GLUE benchmark ( Wang et al. , 2019b ) following previous work studying instability during fine-tuning : CoLA , MRPC , RTE , and QNLI . Detailed statistics for each of the datasets can be found in Section 7.2 in the Appendix . CoLA . The Corpus of Linguistic Acceptability ( Warstadt et al. , 2018 ) is a sentence-level classification task containing sentences labeled as either grammatical or ungrammatical . Fine-tuning on CoLA was observed to be particularly stable in previous work ( Phang et al. , 2018 ; Dodge et al. , 2020 ; Lee et al. , 2020 ) . Performance on CoLA is reported in Matthew ’ s correlation coefficient ( MCC ) . MRPC . The Microsoft Research Paraphrase Corpus ( Dolan and Brockett , 2005 ) is a sentence-pair classification task . Given two sentences , a model has to judge whether the sentences paraphrases of each other . Performance on MRPC is measured using the F1 score . RTE . The Recognizing Textual Entailment dataset is a collection of sentence-pairs collected from a series of textual entailment challenges ( Dagan et al. , 2005 ; Bar-Haim et al. , 2006 ; Giampiccolo et al. , 2007 ; Bentivogli et al. , 2009 ) . RTE is the second smallest dataset in the GLUE benchmark and fine-tuning on RTE was observed to be particularly unstable ( Phang et al. , 2018 ; Dodge et al. , 2020 ; Lee et al. , 2020 ) . Accuracy is used to measure performance on RTE . QNLI . The Question-answering Natural Language Inference dataset contains sentence pairs obtained from SQuAD ( Rajpurkar et al. , 2016 ) . Wang et al . ( 2019b ) converted SQuAD into a sentence pair classification task by forming a pair between each question and each sentence in the corresponding paragraph . The task is to determine whether the context sentence contains the answer to the question , i.e . entails the answer . Accuracy is used to measure performance on QNLI . 3.2 FINE-TUNING . Unless mentioned otherwise , we follow the default fine-tuning strategy recommended by Devlin et al . ( 2019 ) : we fine-tune uncased BERTLARGE ( henceforth BERT ) using a batch size of 16 and a learning rate of 2e−5 . The learning rate is linearly increased from 0 to 2e−5 for the first 10 % of iterations—which is known as a warmup—and linearly decreased to 0 afterward . We apply dropout with probability p = 0.1 and weight decay with λ = 0.01 . We train for 3 epochs on all datasets and use global gradient clipping . Following Devlin et al . ( 2019 ) , we use the AdamW optimizer ( Loshchilov and Hutter , 2019 ) without bias correction . We decided to not show results for BERTBASE since previous works observed no instability when fine-tuning BERTBASE which we also confirmed in our experiments . Instead , we show additional results on RoBERTaLARGE ( Liu et al. , 2019 ) and ALBERTLARGE-V2 ( Lan et al. , 2020 ) using the same fine-tuning strategy . We note that compared to BERT , both RoBERTa and ALBERT have slightly different hyperparameters . In particular , RoBERTa uses weight decay with λ = 0.1 and no gradient clipping , and ALBERT does not use dropout . A detailed list of all default hyperparameters for all models can be found in Section 7.3 of the Appendix . Our implementation is based on HuggingFace ’ s transformers library ( Wolf et al. , 2019 ) and is available online : https : //github . com/uds-lsv/bert-stable-fine-tuning . Fine-tuning stability . By fine-tuning stability we mean the standard deviation of the fine-tuning performance ( measured , e.g. , in terms of accuracy , MCC or F1 score ) over the randomness of an algorithm . We follow previous works ( Phang et al. , 2018 ; Dodge et al. , 2020 ; Lee et al. , 2020 ) and measure fine-tuning stability using the development sets from the GLUE benchmark . We discuss alternative notions of stability in Section 7.1 in the Appendix . Failed runs . Following Dodge et al . ( 2020 ) , we refer to a fine-tuning run as a failed run if its accuracy at the end of training is less or equal to that of a majority classifier on the respective dataset . The majority baselines for all tasks are found in Section 7.2 in the Appendix . 4 INVESTIGATING PREVIOUS HYPOTHESES FOR FINE-TUNING INSTABILITY . Previous works on fine-tuning predominantly state two hypotheses for what can be related to finetuning instability : catastrophic forgetting and small training data size of the downstream tasks . Despite the ubiquity of these hypotheses ( Devlin et al. , 2019 ; Phang et al. , 2018 ; Dodge et al. , 2020 ; Lee et al. , 2020 ) , we argue that none of them has a causal relationship with fine-tuning instability . 4.1 DOES CATASTROPHIC FORGETTING CAUSE FINE-TUNING INSTABILITY ? . Catastrophic forgetting ( McCloskey and Cohen , 1989 ; Kirkpatrick et al. , 2017 ) refers to the phenomenon when a neural network is sequentially trained to perform two different tasks , and it loses its ability to perform the first task after being trained on the second . More specifically , in our setup , it means that after fine-tuning a pre-trained model , it can no longer perform the original masked language modeling task used for pre-training . This can be measured in terms of the perplexity on the original training data . Although the language modeling performance of a pre-trained model correlates with its fine-tuning accuracy ( Liu et al. , 2019 ; Lan et al. , 2020 ) , there is no clear motivation for why preserving the original masked language modeling performance after fine-tuning is important.1 In the context of fine-tuning BERT , Lee et al . ( 2020 ) suggest that their regularization method has an effect of alleviating catastrophic forgetting . Thus , it is important to understand how exactly catastrophic forgetting occurs during fine-tuning and how it relates to the observed fine-tuning instability . To better understand this , we perform the following experiment : we fine-tune BERT on RTE , following the default strategy by Devlin et al . ( 2019 ) . We select three successful and three failed fine-tuning runs and evaluate their masked language modeling perplexity on the test set of the WikiText-2 language modeling benchmark ( Merity et al. , 2016 ) .2 We sequentially substitute the top-k layers of the network varying k from 0 ( i.e . all layers are from the fine-tuned model ) to 24 ( i.e . all layers are from the pre-trained model ) . We show the results in Fig . 2 ( a ) and ( b ) . We can observe that although catastrophic forgetting occurs for the failed models ( Fig . 2a ) — perplexity on WikiText-2 is indeed degraded for k = 0 — the phenomenon is much more nuanced . Namely , catastrophic forgetting affects only the top layers of the network — in our experiments often around 1An exception could by the case where supervised fine-tuning is performed as an intermediate training step , e.g . with the goal of domain adaptation . We leave an investigation of this setting for future work . 2BERT was trained on English Wikipedia , hence WikiText-2 can be seen as a subset of its training data . 10 out of 24 layers , and the same is however also true for the successfully fine-tuned models , except for a much smaller increase in perplexity . Another important aspect of our experiment is that catastrophic forgetting typically requires that the model at least successfully learns how to perform the new task . However , this is not the case for the failed fine-tuning runs . Not only is the development accuracy equal to that of the majority classifier , but also the training loss on the fine-tuning task ( here RTE ) is trivial , i.e . close to − ln ( 1/2 ) ( see Fig . 2 ( c ) ) . This suggests that the observed fine-tuning failure is rather an optimization problem causing catastrophic forgetting in the top layers of the pre-trained model . We will show later that the optimization aspect is actually sufficient to explain most of the fine-tuning variance .

Summary: This paper considers the stability of fine-tuning BERT-LARGE models, with considerations for RoBERTa and ALBERT. In particular, it aims to demonstrate that previously identified reasons, catastrophic forgetting and small fine-tuning datasets, fail to explain the observed instability. Instead, it posits that the instability is caused by optimization difficulties that lead to vanishing gradients.