---
abstract: 'Fine-tuning pre-trained transformer-based language models such as BERT has become a common practice dominating leaderboards across various NLP benchmarks.  Despite the strong empirical performance of fine-tuned models, fine-tuning is an unstable process: training the same model with multiple random seeds can result in a large variance of the task performance.  Previous literature [@devlin-etal-2019-bert; @Lee2020Mixout:; @dodge2020fine] identified two potential reasons for the observed instability: catastrophic forgetting and a small size of the fine-tuning datasets.  In this paper, we show that both hypotheses fail to explain the fine-tuning instability.  We analyze BERT, RoBERTa, and ALBERT, fine-tuned on three commonly used datasets from the GLUE benchmark and show that the observed instability is caused by optimization difficulties that lead to vanishing gradients.  Additionally, we show that the remaining variance of the downstream task performance can be attributed to differences in generalization where fine-tuned models with the same training loss exhibit noticeably different test performance.  Based on our analysis, we present a simple but strong baseline that makes fine-tuning BERT-based models significantly more stable than previously proposed approaches.  Code to reproduce our results is available online: <https://github.com/uds-lsv/bert-stable-fine-tuning>.' 
author:
- |
    Marius Mosbach\
    Saarland Informatics Campus, Saarland University\
    `mmosbach@lsv.uni-saarland.de`\
    Maksym Andriushchenko\
    École polytechnique fédérale de Lausanne\
    `maksym.andriushchenko@epfl.ch`\
    Dietrich Klakow\
    Saarland Informatics Campus, Saarland University\
    `dietrich.klakow@lsv.uni-saarland.de`\
bibliography:
- 'references.bib'
title: 'On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines'
---

Introduction
============

Pre-trained transformer-based masked language models such as BERT [@devlin-etal-2019-bert], RoBERTa [@liu2019roberta], and ALBERT [@Lan2020ALBERT:] have had a dramatic impact on the NLP landscape in the recent year.  The standard recipe of using such models typically involves training a pre-trained model for few epochs on a supervised downstream dataset.  A process referred to as fine-tuning. 

While fine-tuning has led to impressive empirical results, dominating a large variety of English NLP benchmarks such as GLUE [@wang2018glue] and SuperGLUE [@NIPS2019_8589] it is still poorly understood.  Not only have fine-tuned models been shown to pick up spurious patterns and biases present in the training data [@niven-kao-2019-probing; @mccoy-etal-2019-right], but also to exhibit a large training instability: fine-tuning a model multiple times on the same dataset, varying only the random seed, leads to a large standard deviation of the fine-tuning accuracy [@devlin-etal-2019-bert; @dodge2020fine]. 

Few methods have been proposed to solve the observed instability [@phang2018sentence; @Lee2020Mixout:], however without providing a sufficient understanding of why fine-tuning is prone to such failure.  The goal of this work is to address this shortcoming.  More specifically, we investigate the following question:

*Why is fine-tuning prone to failures and how can we improve its stability?* 

We start by investigating two common hypotheses for fine-tuning instability: catastrophic forgetting and a small size of the fine-tuning datasets and demonstrate that both hypotheses fail to explain fine-tuning instability.  We then investigate fine-tuning failures on three datasets from the popular GLUE benchmark and show that the observed fine-tuning instability can be decomposed into two unique aspects: (1) optimization difficulties early in training, characterized by vanishing gradients, and (2) differences in generalization late in training, characterized by a large variation of development set accuracy for runs with almost equivalent training performance. 

Based on our analysis, we present a simple but strong baseline for fine-tuning pre-trained language models that significantly improves the fine-tuning stability compared to previous works (Fig.  \[fig:teaser-box-plots\]).  Moreover, we show that our findings apply not only to the widely used BERT model but also to more recent models such as RoBERTa and ALBERT. 

[.32]{} ![ Our proposed fine-tuning strategy leads to very stable results with very concentrated development set performance over 25 different random seeds across all three datasets on .  In particular, we significantly outperform the recently proposed approach of @Lee2020Mixout: in terms of fine-tuning stability.[]{data-label="fig:teaser-box-plots"}](boxplot-bert-large-uncased-rte-teaser "fig:"){width="100.00000%"}

 

[.32]{} ![ Our proposed fine-tuning strategy leads to very stable results with very concentrated development set performance over 25 different random seeds across all three datasets on .  In particular, we significantly outperform the recently proposed approach of @Lee2020Mixout: in terms of fine-tuning stability.[]{data-label="fig:teaser-box-plots"}](boxplot-bert-large-uncased-mrpc-teaser "fig:"){width="100.00000%"}

 

[.32]{} ![ Our proposed fine-tuning strategy leads to very stable results with very concentrated development set performance over 25 different random seeds across all three datasets on .  In particular, we significantly outperform the recently proposed approach of @Lee2020Mixout: in terms of fine-tuning stability.[]{data-label="fig:teaser-box-plots"}](boxplot-bert-large-uncased-cola-teaser "fig:"){width="100.00000%"}

Related work
============

The fine-tuning instability of BERT has been pointed out in various studies.  @devlin-etal-2019-bert report instabilities when fine-tuning on small datasets and resort to performing multiple restarts of fine-tuning and selecting the model that performs best on the development set.  Recently, @dodge2020fine performed a large-scale empirical investigation of the fine-tuning instability of BERT.  They found dramatic variations in fine-tuning accuracy across multiple restarts and argue how it might be related to the choice of random seed and the dataset size. 

Few approaches have been proposed to directly address the observed fine-tuning instability.  @phang2018sentence study intermediate task training (STILTS) before fine-tuning with the goal of improving performance on the GLUE benchmark.  They also find that their proposed method leads to improved fine-tuning stability.  However, due to the intermediate task training, their work is not directly comparable to ours.  @Lee2020Mixout: propose a new regularization technique termed Mixout.  The authors show that Mixout improves stability during fine-tuning which they attribute to the prevention of catastrophic forgetting. 

Another line of work investigates optimization difficulties when pre-training transformer-based language models [@xiong2020layer; @Liu2020On].  Similar to our work, they highlight the importance of warmup for optimization.  Both works focus on pre-training and we hence view them as orthogonal to our work. 

Background
==========

We first present the datasets used for fine-tuning and briefly recall the overall experimental protocol of fine-tuning pre-trained masked language models such as BERT. 

Datasets
--------

We study three datasets from the GLUE benchmark [@wang2018glue]: CoLA, MRPC, and RTE.  Statistics for each of the three datasets can be found in Section \[sec:appendix-stats\] in the appendix. 

[CoLA.]{}  The Corpus of Linguistic Acceptability [@warstadt2018neural] is a sentence-level classification task containing sentences labeled as either grammatical or ungrammatical.  Fine-tuning on CoLA was observed to be particularly stable in previous work [@phang2018sentence; @dodge2020fine; @Lee2020Mixout:].  Performance on CoLA is reported in Matthew’s correlation coefficient (MCC). 

[MRPC.]{}  The Microsoft Research Paraphrase Corpus [@dolan-brockett-2005-automatically] is a sentence-pair classification tasks.  Given two sentences, a model has to judge whether the sentences paraphrases of each other.  Performance on MRPC is measured using the average of accuracy and $F_1$ score. 

[RTE.]{}  The Recognizing Textual Entailment dataset is a collection of sentence-pairs collected from a series of textual entailment challenges [@10.1007/11736790_9; @barhaim-rte; @10.5555/1654536.1654538; @Bentivogli09thefifth].  RTE is the second smallest dataset in the GLUE benchmark and fine-tuning on RTE was observed to be particularly unstable [@phang2018sentence; @dodge2020fine; @Lee2020Mixout:].  Accuracy is used to measure performance on RTE. 

Fine-tuning
-----------

Unless mentioned otherwise, we follow the default fine-tuning strategy recommended by [@devlin-etal-2019-bert]: we fine-tune uncased (henceforth BERT) using a batch size of $16$ and a learning rate of $\num{2e-05}$. The learning rate is linearly increased from $0$ to $\num{2e-05}$ for the first $10\%$ of iterations—which is known as *warmup*—and linearly decreased to $0$ afterwards.  We apply dropout with probability $p=0.1$ and weight decay with $\lambda=0.01$. We train for 3 epochs on all datasets and use global gradient clipping.  Following [@devlin-etal-2019-bert], we use the AdamW optimizer [@loshchilov2018decoupled] *without* bias correction. 

We fine-tune [@liu2019roberta] and [@Lan2020ALBERT:] using the same strategy.  We note that compared to BERT, both RoBERTa and ALBERT have slightly different hyperparamters which we changed accordingly.  RoBERTa uses weight decay with $\lambda=0.1$ and no gradient clipping.  ALBERT does not use dropout.  A detailed list of all default hyper-parameters for all models can be found in Section \[sec:appendix-hyper\] of the appendix. 

[Failed runs.]{}  Following @dodge2020fine, we refer to a fine-tuning run as a *failed run* if its accuracy (either at the end of training or after early stopping) is less or equal to that of a majority class classifier on the respective dataset.  Majority baselines for all tasks are found in the appendix. 

[Implementation.]{}  For our experiments, all models were trained on single NVIDIA V100 GPUs.  Our implementation is based on HuggingFace’s transformers library [@wolf2019transformers] and the code to reproduce our results is available at <https://github.com/uds-lsv/bert-stable-fine-tuning>. 

Investigating previous explanations of fine-tuning instability {#sec:previous-explanations}
==============================================================

[.33]{} ![ Language modeling perplexity for three failed () and successful () fine-tuning runs of BERT on RTE where we replace the weights of the top-$k$ layers with their pre-trained values.  We can observe that it is often sufficient to reset around 10 layers out of 24 to recover back the language modeling abilities of the pre-trained model. ( c) shows the average training loss and development accuracy ($\pm1 $std) for 5 failed fine-tuning runs on RTE.  Failed fine-tuning runs lead to a trivial training loss suggesting an optimization problem. []{ data-label="fig:cat-forgetting"}](mlm-failed-rte "fig:"){width="100.00000%"}

 

[.33]{} ![ Language modeling perplexity for three failed () and successful () fine-tuning runs of BERT on RTE where we replace the weights of the top-$k$ layers with their pre-trained values.  We can observe that it is often sufficient to reset around 10 layers out of 24 to recover back the language modeling abilities of the pre-trained model. ( c) shows the average training loss and development accuracy ($\pm1 $std) for 5 failed fine-tuning runs on RTE.  Failed fine-tuning runs lead to a trivial training loss suggesting an optimization problem. []{ data-label="fig:cat-forgetting"}](mlm-successful-rte "fig:"){width="100.00000%"}

 

[.33]{} ![ Language modeling perplexity for three failed () and successful () fine-tuning runs of BERT on RTE where we replace the weights of the top-$k$ layers with their pre-trained values.  We can observe that it is often sufficient to reset around 10 layers out of 24 to recover back the language modeling abilities of the pre-trained model. ( c) shows the average training loss and development accuracy ($\pm1 $std) for 5 failed fine-tuning runs on RTE.  Failed fine-tuning runs lead to a trivial training loss suggesting an optimization problem. []{ data-label="fig:cat-forgetting"}](task-performance-rte-failed "fig:"){width="100.00000%"}

Previous works on fine-tuning predominantly state two hypotheses for what causes fine-tuning instability: *catastrophic forgetting* and *small training data size* of the downstream tasks.  Despite the ubiquity of these hypotheses [@devlin-etal-2019-bert; @phang2018sentence; @dodge2020fine; @Lee2020Mixout:; @Zhu2020FreeLB:], we argue that that none of them can fully explain the fine-tuning instability. 

Does catastrophic forgetting cause fine-tuning instability? 
-----------------------------------------------------------

*Catastrophic forgetting* [@mccloskey1989catastrophic; @Kirkpatrick_2017] refers to the phenomenon when a neural network is sequentially trained to perform two different tasks, and it loses its ability to perform the first task after being trained on the second.  More specifically, in our setup it means that after fine-tuning a pre-trained model, it can no longer perform the original masked language modeling task used for pre-training.  This can be measured in terms of the perplexity on the original training data.  Although the language modeling performance of a pre-trained model correlates with its fine-tuning accuracy [@liu2019roberta; @Lan2020ALBERT:], there is no clear motivation for why preserving the original masked language modeling performance after fine-tuning is important. 

[@Lee2020Mixout:] directly motivate their approach as to prevent catastrophic forgetting, thus it is important to understand how exactly it occurs in fine-tuning and how it relates to the observed fine-tuning instability.  To understand this, we perform the following experiment.  We fine-tune BERT on RTE, following the default strategy by [@devlin-etal-2019-bert].  We select three successful and three failed fine-tuning runs and evaluate their masked language modeling perplexity on the test set of the WikiText-2 language modeling benchmark [@merity2016pointer].[^1] We sequentially substitute the top-$k$ layers of the network varying $k$ from $0$ (i.e. all layers are from the fine-tuned model) to $24$ (i.e. all layers are from the pre-trained model). 

We show the results in Fig.  \[fig:cat-forgetting\] (a) and (b).  We can observe that although catastrophic forgetting occurs for the failed models (Fig.  \[fig:mlm-failed\]) — perplexity on WikiText-2 is indeed degraded for $k=0$ — the phenomenon is much more nuanced.  Namely, catastrophic forgetting affects only the top layers of the network — in our experiments often around $10$ out of $24$ layers, and the same is however also true for the successfully fine-tuned models, except for a much smaller increase in perplexity. 

Another important aspect of our experiment is that catastrophic forgetting typically requires that the model at least successfully learns how to perform the new task.  However, this is not the case for the failed fine-tuning runs.  Not only is the development accuracy equal to that of the majority classifier, but also the training loss on the fine-tuning task (here RTE) is trivial, i.e. close to $-\ln(\nicefrac{1}{2})$ (see Fig.  \[fig:cat-forgetting\] (c)).  This suggests that the observed fine-tuning failure is rather an optimization problem *causing* catastrophic forgetting in the top layers of the pre-trained model.  We will show later that the optimization aspect is actually sufficient to explain most of the variance in the fine-tuning performance. 

Do small training datasets cause fine-tuning instability? 
---------------------------------------------------------

[.40]{} ![  Results on down-sampled MRPC and CoLA using the default fine-tuning scheme of BERT [@devlin-etal-2019-bert].  The leftmost boxplot in each sub-figure shows the development accuracy when training on the full training set.  Training on less samples hurts generalization but not stability. []{ data-label="fig:cola-sampled"}](boxplot-bert-large-uncased-mrpc-sampled "fig:"){width="100.00000%"}

 

[.40]{} ![  Results on down-sampled MRPC and CoLA using the default fine-tuning scheme of BERT [@devlin-etal-2019-bert].  The leftmost boxplot in each sub-figure shows the development accuracy when training on the full training set.  Training on less samples hurts generalization but not stability. []{ data-label="fig:cola-sampled"}](boxplot-bert-large-uncased-cola-sampled "fig:"){width="100.00000%"}

Having a small training dataset is by far the most commonly stated hypothesis for fine-tuning instability.  Multiple recent works [@devlin-etal-2019-bert; @phang2018sentence; @Lee2020Mixout:; @Zhu2020FreeLB:; @dodge2020fine; @pruksachatkun2020intermediate] that have observed BERT fine-tuning to be unstable explicitly attribute this finding to the small number of training examples. 

To test if having a small training dataset actually causes instability we perform the following experiment:[^2] we randomly sample $1{,}000$ training samples from the CoLA and MRPC training datasets and fine-tune BERT for 25 random seeds on each dataset.  We compare two different setting: first, training for 3 epochs on the reduced training dataset and second, training for the same number of *iterations* as on the full training dataset.  We show the results in Fig.  \[fig:cola-sampled\].  Note that training on less data does indeed affect the fraction of failed runs.  However, when we train for as many iterations as on the full training dataset, we obtain back the original level of fine-tuning stability.  Further, as expected, we observe that training on less samples affects the generalization of the model, leading to a worse validation performance on both tasks. 

We conclude from this experiment, that the role of training dataset size per se is *orthogonal* to fine-tuning stability.  What is crucial is rather for how many iterations we train.  As our experiment shows, the observed increase in instability when training with smaller datasets can rather be attributed to the reduction of the number of iterations (changing the effective learning rate schedule) which, as we will show in the next section, has a crucial influence on the fine-tuning stability. 

Disentangling optimization and generalization in fine-tuning instability {#sec:disentangling}
========================================================================

Our findings in Section \[sec:previous-explanations\] detail that while both catastrophic forgetting and small size of the datasets indeed *correlate* with fine-tuning instability, none of them are causing it.  In this section, we argue that the fine-tuning instability is an optimization problem, and it admits a simple solution.  Additionally, we show that even though a large fraction of the fine-tuning instability can be explained by optimization, the remaining instability can be attributed to generalization issues where fine-tuning runs with the same training loss exhibit noticeable differences in validation performance. 

[.40]{} ![  Gradient norms (plotted on a *logarithmic scale*) of different layers on RTE for a failed and sucessful run of BERT fine-tuning.  We observe that the failed run is characterized by *vanishing gradients* in the bottom layers of the network.  Additional plots for other weight matrices can be found in the appendix. []{ data-label="fig:layer-gradient-norms-bert"}](gradient-norm-layers-rte-bert-large-attention-output-dense-weight-failed "fig:"){width="100.00000%"}

 

[.40]{} ![  Gradient norms (plotted on a *logarithmic scale*) of different layers on RTE for a failed and sucessful run of BERT fine-tuning.  We observe that the failed run is characterized by *vanishing gradients* in the bottom layers of the network.  Additional plots for other weight matrices can be found in the appendix. []{ data-label="fig:layer-gradient-norms-bert"}](gradient-norm-layers-rte-bert-large-attention-output-dense-weight-successful "fig:"){width="100.00000%"}

The role of optimization
------------------------

[Failed fine-tuning runs suffer from vanishing gradients.]{}  We observed in Fig.  \[fig:training-loss-failed\] that the failed runs have practically constant training loss throughout training.  In order to better understand the nature of this phenomenon, in Fig.  \[fig:layer-gradient-norms-bert\] we plot the $\ell_2$ gradient norms of the loss function with respect to different layers of BERT, for one failed and successful fine-tuning run.  For the failed run we see large gradients only for the top layers and *vanishing gradients* for the bottom layers.  This is in large contrast to the successful run.  While we also observe small gradients in the beginning of training (until iteration $70$), gradients start to grow as training continues.  Moreover, at the end of fine-tuning we observe the gradient norms nearly $2\times$ orders of magnitude larger than that of the failed run.  Similar visualizations for additional layers and weights can be found in the appendix.  We observe the same behaviour also for RoBERTa and ALBERT models, and the corresponding figures can be found in Appendix (Fig.  \[fig:layer-gradient-norms-roberta\] and \[fig:layer-gradient-norms-albert\]). 

Importantly, we note that the vanishing gradients we observe during fine-tuning are harder to resolve than the standard *vanishing gradient problem* [@hochreiter1991untersuchungen; @bengio1994learning].  In particular, common weight initialization schemes [@glorot2010understanding; @he2015delving] ensure that the pre-activations of each layer of the network have zero mean and unit variance in expectation.  However, we cannot simply modify the weights of a pre-trained model to ensure this property since this would conflict the idea of using the pre-trained weights. 

[Importance of bias correction in ADAM.]{}  Following [@devlin-etal-2019-bert], subsequent works on fine-tuning BERT-based models use the ADAM optimizer [@kingma2015adam].  A subtle detail of the fine-tuning scheme of [@devlin-etal-2019-bert] is that it *does not* include the bias correction in ADAM. 

[.33]{} ![  Box plots showing the fine-tuning performance of () BERT, () RoBERTa, () ALBERT for different learning rates $\alpha$ with and without bias correction (BC) on RTE.  For BERT and ALBERT, having bias correction leads to more stable results and allows to train using larger learning rates.  For RoBERTa the effect is less pronounced but still visible. []{ data-label="fig:bert_roberta_albert"}](boxplot-bert-large-uncased-rte-comparison "fig:"){width="100.00000%"}

 

[.33]{} ![  Box plots showing the fine-tuning performance of () BERT, () RoBERTa, () ALBERT for different learning rates $\alpha$ with and without bias correction (BC) on RTE.  For BERT and ALBERT, having bias correction leads to more stable results and allows to train using larger learning rates.  For RoBERTa the effect is less pronounced but still visible. []{ data-label="fig:bert_roberta_albert"}](boxplot-roberta-large-rte "fig:"){width="100.00000%"}

 

[.33]{} ![  Box plots showing the fine-tuning performance of () BERT, () RoBERTa, () ALBERT for different learning rates $\alpha$ with and without bias correction (BC) on RTE.  For BERT and ALBERT, having bias correction leads to more stable results and allows to train using larger learning rates.  For RoBERTa the effect is less pronounced but still visible. []{ data-label="fig:bert_roberta_albert"}](boxplot-albert-large-v2-rte "fig:"){width="100.00000%"}

@kingma2015adam already describe the effect of the bias correction as to reduce the learning rate at the beginning of training.  By rewriting the update equations of ADAM as follows, we can clearly see this effect of bias correction: $$\begin{aligned}
    \alpha_t &\leftarrow \alpha \cdot \sqrt{1 - \beta^t_2} / (1 - \beta^t_1), \label{eq:adam-step-size} \\
    \theta_t &\leftarrow \theta_{t-1} - \alpha_t \cdot m_t / (\sqrt{v_t} + \epsilon),\end{aligned}$$

[rb]{}[0.33]{}

![ image](bias-correction){width="32.00000%"}

where $m_t$ and $v_t$ are biased first and second moment estimates respectively.  Equation  shows that bias correction simply boils down to reducing the original step size $\alpha$ by a multiplicative factor $\sqrt{1 - \beta^t_2} / (1 - \beta^t_1)$ which is significantly below $1$ for the first iterations of training and approaches $1$ as the number of training iterations $t$ increases (see Fig.  \[fig:bias-correction\]).  Along the same lines, [@You2020Large] explicitly remark that bias correction in ADAM has similar effect to warmup which is widely used in deep learning to prevent divergence early in training [@he2016deep; @goyal2017accurate; @devlin-etal-2019-bert; @Wong2020Fast]. 

The implicit warmup of ADAM is likely to be an important factor that contributed to its success.  We argue that fine-tuning BERT-based language models is not an exception.  In Fig.  \[fig:bert\_roberta\_albert\] we show the results of fine-tuning on RTE with and without bias correction for BERT, RoBERTa, and ALBERT models.  We observe that there is a significant benefit in combining warmup with bias correction, particularly for BERT and ALBERT.  Even though for RoBERTa fine-tuning is already more stable even without bias correction, adding bias correction gives an additional improvement.[^3]

Our results show that bias correction is useful if we want to get best performance within 3 epochs, the default recommendation by @devlin-etal-2019-bert.  An alternative solution is to simply train longer with a smaller learning rate, which also leads to much more stable fine-tuning.  We provide a more detailed ablation study in Appendix (Fig.  \[fig:full-boxplots-rte\]) with analogous box plots for BERT using various learning rates, numbers of training epochs, with and without bias correction. 

[Loss surfaces.]{}  To get further intuition about the fine-tunung failure, we provide loss surface visualizations [@NIPS2018_7875; @hao-etal-2019-visualizing] of failed and successful runs when fine-tuning BERT.  Denote by $\theta_p$, $\theta_f$, $\theta_s$ the parameters of the pre-trained model, failed model, and successfully trained model, respectively.  We plot a two-dimensional loss surface $f(\alpha, \beta) = \mathcal{L}(\theta_p + \alpha \delta_1 + \beta \delta_2)$ in the subspace spanned by $\delta_1 = \theta_f - \theta_p$ and $\delta_2 = \theta_s - \theta_p$ centered at the weights of the pre-trained model $\theta_p$. Additional details are specified in the appendix. 

Contour plots of the loss surfaces for RTE, MRPC, and CoLA are shown in Fig.  \[fig:loss-surfaces\].  They provide further evidence for our findings on vanishing gradients: for failed fine-tuning runs gradient descent converges to a “bad” valley with a sub-optimal training loss.  Moreover, this bad valley is separated from the local minimum (to which the successfully trained run converged) by a barrier.  Interestingly, we observe a highly similar geometry for all three datasets providing further support for our interpretation of fine-tuning instability as a primarily optimization issue. 

[.33]{} ![ 2D loss surfaces in the subspace spanned by $\delta_1 = \theta_f - \theta_p$ and $\delta_2 = \theta_s - \theta_p$ on RTE, MRPC and CoLA.  $\theta_p$, $\theta_f$, $\theta_s$ denote the parameters of the pre-trained, failed, and successfully trained model, respectively.[]{data-label="fig:loss-surfaces"}](loss-surface-rte-3-epochs "fig:"){width="100.00000%"}

 

[.33]{} ![ 2D loss surfaces in the subspace spanned by $\delta_1 = \theta_f - \theta_p$ and $\delta_2 = \theta_s - \theta_p$ on RTE, MRPC and CoLA.  $\theta_p$, $\theta_f$, $\theta_s$ denote the parameters of the pre-trained, failed, and successfully trained model, respectively.[]{data-label="fig:loss-surfaces"}](loss-surface-mrpc-3-epochs "fig:"){width="100.00000%"}

 

[.33]{} ![ 2D loss surfaces in the subspace spanned by $\delta_1 = \theta_f - \theta_p$ and $\delta_2 = \theta_s - \theta_p$ on RTE, MRPC and CoLA.  $\theta_p$, $\theta_f$, $\theta_s$ denote the parameters of the pre-trained, failed, and successfully trained model, respectively.[]{data-label="fig:loss-surfaces"}](loss-surface-cola-3-epochs "fig:"){width="100.00000%"}

The role of generalization
--------------------------

[r]{}[0.35]{}

![ image](task-performance-lines-rte){width="33.00000%"}\
![ image](train-loss-dev-performance-scatter-rte){width="33.00000%"}

Having established a better understanding of the fine-tuning optimization process and how it relates to the observed instability, we now turn to the generalizations aspects of fine-tuning instability.  In order to show that the remaining fine-tuning variance can be attributed to generalization, we perform the following experiment.  We fine-tune BERT on RTE following the default scheme of @devlin-etal-2019-bert but train for 10 epochs. 

Fig.  \[fig:generalization\]a shows the development set accuracy during fine-tuning for 10 successful runs.  Fig.  \[fig:generalization\]b shows the development set accuracy vs. training loss of *all* BERT models fine-tuned on RTE for this paper, in total 500 models (see also Fig.  \[fig:full-boxplots-rte\]). 

We find that despite achieving close to zero training loss overfitting is not an issue during fine-tuning.  This is consistent with previous work by @hao-etal-2019-visualizing, which arrived at a similar conclusion.  Based on our results, we argue that it is even desirable to perform a large number of training iterations since the development accuracy varies considerably during fine-tuning and it does not degrade even when the training loss is as low as $10^{-5}$.

Combining these findings with results from the previous section we arrive at the conclusion that the observed fine-tuning instability can be decomposed into two aspects: optimization and generalization.  In the next section we propose a simple solution addressing both issues. 

A simple but hard-to-beat baseline for fine-tuning BERT
=======================================================

As our findings in Section \[sec:disentangling\] show, the empirically observed instability of fine-tuning BERT can be attributed to vanishing gradients early in training as well as differences in generalization late in training.  Given the new understanding of fine-tuning instability we propose the following guidelines for fine-tuning transformer-based masked language models:

-   Use small learning rates combined with bias correction to avoid vanishing gradients early in training. 

-   Increase the number of iterations considerably and train to (almost) zero training loss while making use of early stopping. 

Following our guidelines we propose a new baseline strategy for fine-tuning BERT: We use a learning rate of $\num{2e-05}$ and train for 20 epochs.  The learning rate is linearly increased for the first $10\%$ of steps and linearly decayed to zero afterwards.  We use early stopping based on validation accuracy.  All other hyperparamters are kept unchanged.  A full ablation study on all three dataset testing various combinations of the changed hyperparamters is presented in Section \[sec:appendix-ablations\] in the appendix. 

[Results.]{}  Despite the simplicity of our proposed fine-tuning strategy we obtain strong empirical performance.  Table \[tab:results\] and Fig.\[fig:teaser-box-plots\] show the results of fine-tuning BERT on RTE, MRPC, and CoLA.  We compare to the default strategy by @devlin-etal-2019-bert and the recently proposed Mixout method by @Lee2020Mixout:.  On RTE and CoLA, we do not only significantly improve fine-tuning stability ($3\times$ and $20\times$ smaller standard deviation, respectively) but also consistently improve overall performance (larger mean, median and maximum).  On MRPC, we significantly improve fine-tuning stability ($4\times$ smaller standard deviation) while reaching overall similar performance in terms of mean, median and maximum score. 

Finally, we note that the increased computational cost of our proposed fine-tuning scheme is not a major problem in practice.  Comparing runtimes – on RTE, MRPC, and CoLA – to training for only 3 epochs, our fine-tuning scheme results in an average increase of training time by a factor of $6$. Nevertheless we believe that overall our findings will lead to *more efficient* fine-tuning because of the significantly improved stability, effectively reducing the number of necessary fine-tuning runs. 

Conclusions
===========

In this work, we have resolved existing misconceptions about the reasons behind fine-tuning instability and proposed a new baseline strategy for fine-tuning that leads to significantly improved fine-tuning stability and overall improved results on three datasets from the GLUE benchmark. 

By analyzing failed fine-tuning runs, we find that neither catastrophic forgetting nor small dataset sizes sufficiently explain fine-tuning instability.  Rather, our analysis reveals that fine-tuning instability can be characterized by two distinct problems: (1) optimization difficulties early in training, characterized by vanishing gradients, and (2) differences in generalization, characterized by a large variation of development set accuracy for runs with almost equivalent training performance. 

Based upon our analysis, we propose a simple but strong baseline strategy for fine-tuning BERT, even outperforming previous work in terms of fine-tuning stability and overall performance. 

Broader Impact {#broader-impact .unnumbered}
==============

Given the wide adoption and ubiquity of the pre-train then fine-tune scheme in today’s NLP landscape, we believe that a better understanding of the fine-tuning process of pre-trained transformer-based language models can help to reduce the large computational cost of NLP models [@schwartz2019green].  Particularly, a better understanding of the fine-tuning instability combined with solutions for addressing it helps to avoid the need to repeat fine-tuning experiments a large number of times in order to obtain the best score on a leaderboard. 

Acknowledgments {#acknowledgments .unnumbered}
===============

We thank Anna Khokhlova for her help with the language modeling experiments, Cheolhyoung Lee and Jesse Dodge for providing us with details of their works, and Badr Abdullah and Aditya Mogadala for their helpful comments on a draft of this paper. 

This work was funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) – Project-ID 232722074 – SFB 1102. 

Appendix {#sec:appendix .unnumbered}
========

Task statistics {#sec:appendix-stats}
---------------

Statistics for each of the datasets studied in this paper.  All datasets are publicly available and can be downloaded here: <https://github.com/nyu-mll/jiant>. 

  ****                **RTE**   **MRPC**                 **CoLA**
  ------------------- --------- ------------------------ ----------
  Training            $2491$    $3669$                   $8551$
  Development         $278$     $409$                    $1043$
  Majority baseline   $0.53$    $0.75$                   $0.0$
  Metric              Acc.       $\frac{F_1 + Acc.}{2}$   MCC

  : Dataset statistics and majority baselines.[]{data-label="tab:glue-tasks-stats"}

Hyperparamters {#sec:appendix-hyper}
--------------

Hyperparameters for BERT, RoBERTa, and ALBERT used for all our experiments. 

  **Hyperparam**           **BERT**                      **RoBERTa**                   **ALBERT**
  ------------------------ ----------------------------- ----------------------------- -----------------------------
  Epochs                   $3, 10, 20$                   $3$                           $3$
  Learning rate            $\num{1e-05} - \num{5e-05}$   $\num{1e-05} - \num{3e-05}$   $\num{1e-05} - \num{3e-05}$
  Learning rate schedule   warmup-linear                 warmup-linear                 warmup-linear
  Warmup ratio             $0.1$                         $0.1$                         $0.1$
  Batch size               $16$                          $16$                          $16$
  Adam $\epsilon$          $\num{1e-06}$                 $\num{1e-06}$                 $\num{1e-06}$
  Adam $\beta_1$           $0.9$                         $0.9$                         $0.9$
  Adam $\beta_2$           $0.999$                       $0.98$                        $0.999$
  Adam bias correction     $\{$True, False$\}$           $\{$True, False$\}$           $\{$True, False$\}$
  Dropout                  $0.1$                         $0.1$                         –
  Weight decay             $0.01$                        $0.1$                         –
  Clipping gradient norm   $1.0$                         –                             $1.0$
  Number of random seeds   $25$                          $25$                          $25$

  : Hyperparameters used for fine-tuning.[]{data-label="tab:hyperparamters"}

Ablation studies {#sec:appendix-ablations}
----------------

Figures \[fig:full-boxplots-rte\], \[fig:full-boxplots-mrpc\], and \[fig:full-boxplots-cola\] show the results of fine-tuning on RTE, MRPC, and CoLA with different combinations of learning rate, number of training epochs, and bias correction.  We make the following observations:

-   When training for only 3 epochs, disabling bias correction clearly hurts performance. 

-   With bias correction, training with larger learning rates is possible. 

-   Combining the usage of bias correction with training for more epochs leads to the best performance. 

[.99]{} ![ Full ablation of fine-tuning BERT on **RTE**.  For each setting we vary only the number of training steps, learning rate, and usage of bias correction (BC).  All other hyperparameters are unchanged.  We fine-tune 25 models for each setting.  $\star$ shows the setting which we recommend as a new baseline fine-tuning strategy.[]{data-label="fig:full-boxplots-rte"}](boxplot-full-bert-large-uncased-rte "fig:"){width="100.00000%"}

[.99]{} ![ Full ablation of fine-tuning BERT on **MRPC**.  For each setting we vary only the number of training steps, learning rate, and usage of bias correction (BC).  All other hyperparameters are unchanged.  We fine-tune 25 models for each setting.  $\star$ shows the setting which we recommend as a new baseline fine-tuning strategy.[]{data-label="fig:full-boxplots-mrpc"}](boxplot-full-bert-large-uncased-mrpc "fig:"){width="100.00000%"}

[.99]{} ![ Full ablation of fine-tuning BERT on **CoLA**.  For each setting we vary only the number of training steps, learning rate, and usage of bias correction (BC).  All other hyperparameters are unchanged.  We fine-tune 25 models for each setting.  $\star$ shows the setting which we recommend as a new baseline fine-tuning strategy.[]{data-label="fig:full-boxplots-cola"}](boxplot-full-bert-large-uncased-cola "fig:"){width="100.00000%"}

Additional gradient norm visualizations {#sec:appendix-viz}
---------------------------------------

We provide additional visualizations for the vanishing gradients observed when fine-tuning RoBERTa, ALBERT, and BERT in Figures \[fig:layer-gradient-norms-roberta\], \[fig:layer-gradient-norms-albert\], \[fig:layer-gradient-norms-bert-all\].  Note that for ALBERT besides the pooler and classification layers, we plot only the gradient norms of a single hidden layer (referred to as `layer0`) because of weight sharing. 

[Gradient norms and MLM perplexity.]{}  We can see from the gradient norm visualizations for BERT in Figures \[fig:layer-gradient-norms-bert\] and \[fig:layer-gradient-norms-bert-all\] that the gradient norm of the pooler and classification layer remains large.  Hence, even though the gradients on most layers of the model vanish, we still update the weights on the top layers.  In fact, this explains the large increase in MLM perplexity for the failed models which is shown in Fig.  \[fig:mlm-failed\].  While most of the layers do not change as we continue training, the top layers of network change dramatically. 

[.40]{} ![  Gradient norms (plotted on a *logarithmic scale*) of additional weight matrices of **RoBERTa** fine-tuned on RTE.  Corresponding layer names are in the captions.  We show gradient norms corresponding to a single failed and single successful, respectively. []{ data-label="fig:layer-gradient-norms-roberta"}](gradient-norm-layers-rte-roberta-large-attention-self-key-weight-failed "fig:"){width="100.00000%"}

 

[.40]{} ![  Gradient norms (plotted on a *logarithmic scale*) of additional weight matrices of **RoBERTa** fine-tuned on RTE.  Corresponding layer names are in the captions.  We show gradient norms corresponding to a single failed and single successful, respectively. []{ data-label="fig:layer-gradient-norms-roberta"}](gradient-norm-layers-rte-roberta-large-attention-self-key-weight-successful "fig:"){width="100.00000%"}

 

[.40]{} ![  Gradient norms (plotted on a *logarithmic scale*) of additional weight matrices of **RoBERTa** fine-tuned on RTE.  Corresponding layer names are in the captions.  We show gradient norms corresponding to a single failed and single successful, respectively. []{ data-label="fig:layer-gradient-norms-roberta"}](gradient-norm-layers-rte-roberta-large-attention-self-query-weight-failed "fig:"){width="100.00000%"}

 

[.40]{} ![  Gradient norms (plotted on a *logarithmic scale*) of additional weight matrices of **RoBERTa** fine-tuned on RTE.  Corresponding layer names are in the captions.  We show gradient norms corresponding to a single failed and single successful, respectively. []{ data-label="fig:layer-gradient-norms-roberta"}](gradient-norm-layers-rte-roberta-large-attention-self-query-weight-successful "fig:"){width="100.00000%"}

 

[.40]{} ![  Gradient norms (plotted on a *logarithmic scale*) of additional weight matrices of **RoBERTa** fine-tuned on RTE.  Corresponding layer names are in the captions.  We show gradient norms corresponding to a single failed and single successful, respectively. []{ data-label="fig:layer-gradient-norms-roberta"}](gradient-norm-layers-rte-roberta-large-attention-self-value-weight-failed "fig:"){width="100.00000%"}

 

[.40]{} ![  Gradient norms (plotted on a *logarithmic scale*) of additional weight matrices of **RoBERTa** fine-tuned on RTE.  Corresponding layer names are in the captions.  We show gradient norms corresponding to a single failed and single successful, respectively. []{ data-label="fig:layer-gradient-norms-roberta"}](gradient-norm-layers-rte-roberta-large-attention-self-value-weight-successful "fig:"){width="100.00000%"}

 

[.40]{} ![  Gradient norms (plotted on a *logarithmic scale*) of additional weight matrices of **RoBERTa** fine-tuned on RTE.  Corresponding layer names are in the captions.  We show gradient norms corresponding to a single failed and single successful, respectively. []{ data-label="fig:layer-gradient-norms-roberta"}](gradient-norm-layers-rte-roberta-large-attention-output-dense-weight-failed "fig:"){width="100.00000%"}

 

[.40]{} ![  Gradient norms (plotted on a *logarithmic scale*) of additional weight matrices of **RoBERTa** fine-tuned on RTE.  Corresponding layer names are in the captions.  We show gradient norms corresponding to a single failed and single successful, respectively. []{ data-label="fig:layer-gradient-norms-roberta"}](gradient-norm-layers-rte-roberta-large-attention-output-dense-weight-successful "fig:"){width="100.00000%"}

[.40]{} ![  Gradient norms (plotted on a *logarithmic scale*) of additional weight matrices of **ALBERT** fine-tuned on RTE.  Corresponding layer names are in the captions.  We show gradient norms corresponding to a single failed and single successful, respectively. []{ data-label="fig:layer-gradient-norms-albert"}](gradient-norm-layers-rte-albert-large-v2-attention-key-weight-failed "fig:"){width="100.00000%"}

 

[.40]{} ![  Gradient norms (plotted on a *logarithmic scale*) of additional weight matrices of **ALBERT** fine-tuned on RTE.  Corresponding layer names are in the captions.  We show gradient norms corresponding to a single failed and single successful, respectively. []{ data-label="fig:layer-gradient-norms-albert"}](gradient-norm-layers-rte-albert-large-v2-attention-key-weight-successful "fig:"){width="100.00000%"}

 

[.40]{} ![  Gradient norms (plotted on a *logarithmic scale*) of additional weight matrices of **ALBERT** fine-tuned on RTE.  Corresponding layer names are in the captions.  We show gradient norms corresponding to a single failed and single successful, respectively. []{ data-label="fig:layer-gradient-norms-albert"}](gradient-norm-layers-rte-albert-large-v2-attention-query-weight-failed "fig:"){width="100.00000%"}

 

[.40]{} ![  Gradient norms (plotted on a *logarithmic scale*) of additional weight matrices of **ALBERT** fine-tuned on RTE.  Corresponding layer names are in the captions.  We show gradient norms corresponding to a single failed and single successful, respectively. []{ data-label="fig:layer-gradient-norms-albert"}](gradient-norm-layers-rte-albert-large-v2-attention-query-weight-successful "fig:"){width="100.00000%"}

 

[.40]{} ![  Gradient norms (plotted on a *logarithmic scale*) of additional weight matrices of **ALBERT** fine-tuned on RTE.  Corresponding layer names are in the captions.  We show gradient norms corresponding to a single failed and single successful, respectively. []{ data-label="fig:layer-gradient-norms-albert"}](gradient-norm-layers-rte-albert-large-v2-attention-value-weight-failed "fig:"){width="100.00000%"}

 

[.40]{} ![  Gradient norms (plotted on a *logarithmic scale*) of additional weight matrices of **ALBERT** fine-tuned on RTE.  Corresponding layer names are in the captions.  We show gradient norms corresponding to a single failed and single successful, respectively. []{ data-label="fig:layer-gradient-norms-albert"}](gradient-norm-layers-rte-albert-large-v2-attention-value-weight-successful "fig:"){width="100.00000%"}

 

[.40]{} ![  Gradient norms (plotted on a *logarithmic scale*) of additional weight matrices of **ALBERT** fine-tuned on RTE.  Corresponding layer names are in the captions.  We show gradient norms corresponding to a single failed and single successful, respectively. []{ data-label="fig:layer-gradient-norms-albert"}](gradient-norm-layers-rte-albert-large-v2-attention-dense-weight-failed "fig:"){width="100.00000%"}

 

[.40]{} ![  Gradient norms (plotted on a *logarithmic scale*) of additional weight matrices of **ALBERT** fine-tuned on RTE.  Corresponding layer names are in the captions.  We show gradient norms corresponding to a single failed and single successful, respectively. []{ data-label="fig:layer-gradient-norms-albert"}](gradient-norm-layers-rte-albert-large-v2-attention-dense-weight-successful "fig:"){width="100.00000%"}

[.45]{} ![  Gradient norms (plotted on a *logarithmic scale*) of additional weight matrices of **BERT** fine-tuned on RTE.  Corresponding layer names are in the captions.  We show gradient norms corresponding to a single failed and single successful, respectively. []{ data-label="fig:layer-gradient-norms-bert-all"}](gradient-norm-layers-rte-bert-large-attention-self-key-weight-failed "fig:"){width="100.00000%"}

 

[.45]{} ![  Gradient norms (plotted on a *logarithmic scale*) of additional weight matrices of **BERT** fine-tuned on RTE.  Corresponding layer names are in the captions.  We show gradient norms corresponding to a single failed and single successful, respectively. []{ data-label="fig:layer-gradient-norms-bert-all"}](gradient-norm-layers-rte-bert-large-attention-self-key-weight-successful "fig:"){width="100.00000%"}

[.45]{} ![  Gradient norms (plotted on a *logarithmic scale*) of additional weight matrices of **BERT** fine-tuned on RTE.  Corresponding layer names are in the captions.  We show gradient norms corresponding to a single failed and single successful, respectively. []{ data-label="fig:layer-gradient-norms-bert-all"}](gradient-norm-layers-rte-bert-large-attention-self-query-weight-failed "fig:"){width="100.00000%"}

 

[.45]{} ![  Gradient norms (plotted on a *logarithmic scale*) of additional weight matrices of **BERT** fine-tuned on RTE.  Corresponding layer names are in the captions.  We show gradient norms corresponding to a single failed and single successful, respectively. []{ data-label="fig:layer-gradient-norms-bert-all"}](gradient-norm-layers-rte-bert-large-attention-self-query-weight-successful "fig:"){width="100.00000%"}

 

[.45]{} ![  Gradient norms (plotted on a *logarithmic scale*) of additional weight matrices of **BERT** fine-tuned on RTE.  Corresponding layer names are in the captions.  We show gradient norms corresponding to a single failed and single successful, respectively. []{ data-label="fig:layer-gradient-norms-bert-all"}](gradient-norm-layers-rte-bert-large-attention-self-value-weight-failed "fig:"){width="100.00000%"}

 

[.45]{} ![  Gradient norms (plotted on a *logarithmic scale*) of additional weight matrices of **BERT** fine-tuned on RTE.  Corresponding layer names are in the captions.  We show gradient norms corresponding to a single failed and single successful, respectively. []{ data-label="fig:layer-gradient-norms-bert-all"}](gradient-norm-layers-rte-bert-large-attention-self-value-weight-successful "fig:"){width="100.00000%"}

Loss surfaces {#sec:appendix-loss-surfaces}
-------------

For Fig.  \[fig:loss-surfaces\], we define the range for both $\alpha$ and $\beta$ as $[-1.5, 1.5]$ and sample 40 points for each axis.  We evaluate the loss on 128 samples from the training dataset of each task using *all* model parameters, including the classification layer.  We disabled dropout for generating the surface plots. 

Fig.  \[fig:gradient-norm-surfaces\] shows contour plots of the total gradient norm.  We can again see that the point to which the failed model converges to ($\theta_f$) is separated from the point the successful model converges to ($\theta_s$) by a barrier.  Moreover, on all the three datasets we can clearly see the valley around $\theta_f$ with a small gradient norm. 

[.33]{} ![ 2D gradient norm surfaces in the subspace spanned by $\delta_1 = \theta_f - \theta_p$ and $\delta_2 = \theta_s - \theta_p$ for BERT fine-tuned on RTE, MRPC and CoLA.  $\theta_p$, $\theta_f$, $\theta_s$ denote the parameters of the pre-trained, failed, and successfully trained model, respectively.[]{data-label="fig:gradient-norm-surfaces"}](total-gradient-norm-rte-3-epochs "fig:"){width="100.00000%"}

 

[.33]{} ![ 2D gradient norm surfaces in the subspace spanned by $\delta_1 = \theta_f - \theta_p$ and $\delta_2 = \theta_s - \theta_p$ for BERT fine-tuned on RTE, MRPC and CoLA.  $\theta_p$, $\theta_f$, $\theta_s$ denote the parameters of the pre-trained, failed, and successfully trained model, respectively.[]{data-label="fig:gradient-norm-surfaces"}](total-gradient-norm-mrpc-3-epochs "fig:"){width="100.00000%"}

 

[.33]{} ![ 2D gradient norm surfaces in the subspace spanned by $\delta_1 = \theta_f - \theta_p$ and $\delta_2 = \theta_s - \theta_p$ for BERT fine-tuned on RTE, MRPC and CoLA.  $\theta_p$, $\theta_f$, $\theta_s$ denote the parameters of the pre-trained, failed, and successfully trained model, respectively.[]{data-label="fig:gradient-norm-surfaces"}](total-gradient-norm-cola-3-epochs "fig:"){width="100.00000%"}

Generalization {#sec:appendix-generalization}
--------------

We provide additional plots of training loss versus validation performance for MRPC and CoLA in Fig.  \[fig:appendix-generalization-all\] (we also show the plot for RTE again for a side-by-side comparison).  We use the 500 models from the ablation study (see Fig.  \[fig:full-boxplots-rte\], \[fig:full-boxplots-mrpc\], \[fig:full-boxplots-cola\]).  We observe a significant variance in validation performance even though the training loss is the same.  This occurs not only on RTE, but on MRPC and CoLA as well which have larger validation sets (see Table \[tab:glue-tasks-stats\]).  Moreover, we can observe that there is no overfitting when training to close to zero training loss.  This justifies our fine-tuning scheme which involves training for a larger number of epochs. 

[.33]{} ![ Development performance versus training loss at the end of training for 500 models fine-tuned on RTE, MRPC, and CoLA, respectively. []{ data-label="fig:appendix-generalization-all"}](train-loss-dev-performance-scatter-rte "fig:"){width="100.00000%"}

 

[.33]{} ![ Development performance versus training loss at the end of training for 500 models fine-tuned on RTE, MRPC, and CoLA, respectively. []{ data-label="fig:appendix-generalization-all"}](train-loss-dev-performance-scatter-mrpc "fig:"){width="100.00000%"}

 

[.33]{} ![ Development performance versus training loss at the end of training for 500 models fine-tuned on RTE, MRPC, and CoLA, respectively. []{ data-label="fig:appendix-generalization-all"}](train-loss-dev-performance-scatter-cola "fig:"){width="100.00000%"}

[^1]: BERT was trained on English Wikipedia, hence WikiText-2 can be seen as a subset of its training data. 

[^2]: We remark that a similar experiment was done in [@phang2018sentence], but with a different goal of showing that their extended pre-training procedure is able to improve the fine-tuning stability. 

[^3]: We note that bias correction is enabled by default in original fairseq [@ott2019fairseq] implementation of RoBERTa.