On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines
Authors: Marius Mosbach
Maksym Andriushchenko
Dietrich Klakow
Venue: ICLR 2021 Poster (2021)
Keywords: fine-tuning stability
transfer learning
pretrained language model
BERT

Abstract: Fine-tuning pre-trained transformer-based language models such as BERT has become a common practice dominating leaderboards across various NLP benchmarks. Despite the strong empirical performance of fine-tuned models, fine-tuning is an unstable process: training the same model with multiple random seeds can result in a large variance of the task performance. Previous literature (Devlin et al., 2019; Lee et al., 2020; Dodge et al., 2020) identified two potential reasons for the observed instability: catastrophic forgetting and small size of the fine-tuning datasets. In this paper, we show that both hypotheses fail to explain the fine-tuning instability. We analyze BERT, RoBERTa, and ALBERT, fine-tuned on commonly used datasets from the GLUE benchmark, and show that the observed instability is caused by optimization difficulties that lead to vanishing gradients. Additionally, we show that the remaining variance of the downstream task performance can be attributed to differences in generalization where fine-tuned models with the same training loss exhibit noticeably different test performance. Based on our analysis, we present a simple but strong baseline that makes fine-tuning BERT-based models significantly more stable than the previously proposed approaches. Code to reproduce our results is available online: https://github.com/uds-lsv/bert-stable-fine-tuning.

Published as a conference paper at ICLR 2021
ON THE STABILITY OF FINE-TUNING BERT: MISCON- CEPTIONS, EXPLANATIONS, AND STRONG BASELINES
Marius Mosbach
Maksym Andriushchenko
Spoken Language Systems (LSV)
Theory of Machine Learning Lab École polytechnique fédérale de Lausanne ma mak maksym. andri www.driiusscchhoo@epppp... ch
Saarland Informatics Campus, Saarland University mmosbache Lsv. www..uni--i uni-saarlande
Dietrich Klakow
Spoken Language Systems (LSV)
Saarland Informatics Campus, Saarland University dietrich, klakkwa lsv. www.uuiissaaaaaann.. de
ABSTRACT
Fine-tuning pre-trained transformer- based language models such as BERT has become a common practice dominating leaderboards across various NLP bench- marks. Despite the strong empirical performance of fine-tuned models, fine-tuning is an unstable process: training the same model with multiple random seeds can result in a large variance of the task performance. Previous literature (De- vlin et al.) 2019: Lee al., 2020) Dodge et al.) (020) identified two potential reasons for the observed instability: catastrophic forgetting and small size of the fine-tuning datasets. In this paper, show that both hypotheses fail to ex- plain the fine-tuning instability. We analyze BERT, RoBERTa, and ALBERT, fine-tuned on commonly used datasets from the GLUE benchmark, and show that the observed instability is caused by optimization difficulties that lead to vanishing gradients. Additionally, we show that the remaining variance of the downstream task performance can be attributed to differences in generalization where fine-tuned models with the same training loss exhibit noticeably different test performance. Based on our analysis, we present simple but strong baseline that makes fine-tuning BERT-based models significantly more stable than the pre- viously proposed approaches. Code to reproduce our results is available online: Ittpss: /attps://github withub.thhuuugggggg com/ uds 1sv/bert stable- -ei-e-inne--n-nnne tuning
INTRODUCTION
Pre-trained transformer-based masked language models such as BERT (Devlin et al. 2019). RoBERTa (Liu et al. 2019), and ALBERT (Lan et al. 2020) have had dramatic impact on the NLP landscape in the recent year. The standard recipe for using such models typically involves training pre- trained model for few epochs on a supervised downstream dataset, which is known as fine-tuning. While fine-tuning has led to impressive empirical results, dominating a large variety of English NLP benchmarks such as GLUE (Wang et al. 2019b) and SuperGLUE (Wang al.] 2019a). poorly understood. Not only have fine-tuned models been shown to pick up spurious patterns and biases present in the training data (Niven and Kao 2019 McCoy et al., 2019), but also to exhibit a large training instability: fine-tuning model multiple times the same dataset, varying only the random seed, leads to a large standard deviation of the fine-tuning accuracy (Devlin et al. 2019 Dodge et al. 2020).
still
Few methods have been proposed to solve the observed instability (Phang et al., 2018 Lee et al.
2020). however without providing sufficient understanding of why fine-tuning is prone to such failure. The goal of this work is to address this shortcoming. More specifically, we investigate the following question:
Why is fine-tuning prone to failures and how improve its stability?

---

Published as a conference paper at ICLR 2021
0.80
0.96
0.70
0.75
0.60
0.50
0.85
0.60
0.55
T
0.80
0.20
0.10
0.75
0.00
Devlin
Ours
Devin
Ours
(a) RTE
(b) MRPC
(c) CoLA
Figure I: Our proposed fine- fine-tuning strategy leads to very stable results with very concentrated development set performance over 25 different random seeds across all three datasets on BERT. In particular, we significantly outperform the recently proposed approach of Lee al. (2200) in terms of fine-tuning stability.
We start by investigating two common hypotheses for fine-tuning instability: catastrophic forgetting and small size the fine-tuning datasets and demonstrate that both hypotheses fail to explain fine-tuning instability. We then investigate fine-tuning failures datasets from the popular GLUE benchmark and show that the observed fine-tuning instability can be decomposed into two separate aspects: (1) optimization difficulties early in training, characterized by vanishing gradients, and (2) differences in generalization late training, characterized by a large variance of development set accuracy for runs with almost equivalent training loss.
Based on our analysis, we present a simple but strong baseline for fine-tuning finettuning pre-trained language models that significantly improves the fine-tuning stability compared previous works (Fig. [ Moreover, we show that our findings apply not only to the widely used BERT model but also to more recent pre-trained models such RoBERTa and ALBERT.
2 RELATED WORK
The fine-tuning instability of BERT has been pointed out in various studies. Devlin et al. (2019) report instabilities when fine-tuning BERT on small datasets and resort performing multiple restarts of fine-tuning and selecting the model that performs best on the development set. Recently, Dodge et al. (2020) performed large-scale empirical investigation of the fine-tuning instability of BERT. They found dramatic variations in fine-tuning accuracy across multiple restarts and argue how it might be related to the choice random seed and the dataset size.
Few approaches have been proposed to directly address the observed fine-tuning instability, Phang ct al. (2018) study intermediate task training (STILTS) before fine-tuning with the goal of improving performance on the GLUE benchmark. They also find that their proposed method leads to improved fine-tuning stability. However, due to the intermediate task training, their work is not directly regularization technique termed Mixout. The
comparable to ours. Lee et al. (2020) propose authors show that Mixout improves stability during fine-tuning which they attribute to the prevention of catastrophic forgetting.
Another line of work investigates optimization difficulties of pre-training transformer-basee language models (ioog et al. 22200) Liu al. 2020). Similar to our work, they highlight the importance of the learning rate warmup for optimization. Both works focus on pre-training and we hence view them as orthogonal to our work.
3 BACKGROUND
3.1 DATASETS
We study four datasets from the GLUE benchmark (Wang et al.) 2019b) following previous work studying instability during fine-tuning: CoLA, MRPC, RTE, and QNLI. Detailed statistics for each of the datasets can be found in Section 7.2 in the Appendix.
2

---

Published as a conference paper at ICLR 2021
CoLA. The Corpus of Linguistic Acceptability (Warstadt et al.
2018 sentence-level classification
task containing sentences labeled as either grammatical or ungrammatical. tine-tuning on CoLA was observed to be particularly stable in previous work (Phang al. 2018 Dodge al. 2020: Lee et al.
2020). Performance on CoLAA is reported in Matthew's correlation coefficient (MCC).
MRPC. The Microsoft Research Paraphrase Corpus (Dolan and Brockett 2005 is sentence-pair
classification task. Given two sentences, a model has to judge whether the sentences paraphrases of each other. Performance on MRPC measured using the Fi score.
RTE. The Recognizing Textual Entailment dataset a collection of sentence-pairs collected from series of textual entailment challenges (Dagan al. 2005 Bar-Haim et al.
2006 Giampiccolo
ct al. 2007: Bentivogli et al., 2009). RTE is the second smallest dataset in the GLUE benchmark and
fine-tuning on RTE was observed be particularly unstable (Phang al. 2018 Dodge al. 2020
Lee et al. 2020 Accuracy is used to measure performance on RTE.
QNLI. The Question- answering Natural Language Inference dataset contains sentence pairs obtained from SQuAD (Rajpurkar et al., 2016). WWang et al. (2019b) converted SQuAD into a sentence pair classification task by forming a pair between each question and each sentence in the corresponding paragraph. The task is to determine whether the context sentence contains the answer to the question, iee. entails the answer. Accuracy is used to measure performance on QNLI.
3.2 FINE-TUNING
Unless mentioned otherwise, we follow the default fine-tuning strategy recommended by Devlin al. (019):: we fine-tune uncased LARG (henceforth BERT) using batch size of 16 and a learning rate of 2e-5. The learning rate is linearly increased from 0 to 2e-5 for the first 10% of known as a warmup-and linearly decreased to 0 afterward. We apply dropout with probability p 0.1 and weight decay with À 0.01. We train for 3 epochs on all datasets and use global gradient clipping. Following Devlin et al. (2019), we use the Adam W optimizer (Loshchilov and Hutter, 2019) without bias correction.
crrationss-hicee
We decided to not show results for BERT BASE since previous works observed no instability when fine-tuning BERTBASE which also confirmed in experiments. Instead, we show additional results on RoBERTaLARGE (Liu et al., 2019) and ALBERTLARGE-V2 (Lan et al.) 2020) using the same fine-tuning strategy. We note that compared BERT, both ROBERTa and ALBERT have slightly different hyperparameters. In particular, RoBERTa uses weight decay with = 0.1 and no gradient clipping, and ALBERT does not use dropout. A detailed list of all default hyperparameters for all models can be found in Section 7.3 of the Appendix. Our implementation is based on HuggingFace's transformers library (Wolf et al. 2019) and is available online: hattps://gite tuning
com
Fine-tuning stability. By fine-tuning stability we mean the standard deviation of the fine-tuning performance (measured, e.g., in terms of accuracy, MCC or F, score) over the randomness of an algorithm. We follow previous works (Phang et al.) 2018: Dodge et al., 2020 Lee al. (020) and measure fine-tuning stability using the development sets from the GLUE benchmark. We discuss alternative notions of stability in Section 7.1 in the Appendix.
Failed runs. Following Dodge al. (2020), we refer to fine-tuning run as a failed run if its accuracy at the end of training is less or equal to that of a majority classifier on the respective dataset. The majority baselines for all tasks are found in Section 7.2 in the Appendix.
4 INVESTIGATING PREVIOUS HYPOTHESES FOR FINE-TUNING INSTABILITY
Previous works on fine-tuning predominantly state two hypotheses for what can be related to fine- tuning instability: catastrophic forgetring and small training data size of the downstream tasks. Despite the ubiquity of these hypotheses (Devlin et al.) 2019 Phang et al.) 2018) Dodge et al.) 2020
Lee et al. 2020,, we argue that none of them has causal relationship with fine-tuning instability.
3

---

Published as a conference paper at ICLR 2021
0.75
0.8
MLM perplxitty
109
MLM perplesity
0.70
0.65
0.6
19-
0.60
0.4
10%-
0.55
13
0.50
10P
9.37
0.45-
2
22
10 14
2010
400
Layers replaced
Layers replaced
Iteations
(a) Perplexity of failed models (b) Perplexity of successful models
(c) Training of failed models
Figure Language modeling perplexity for three failed (a) and successful (b) fine-tuning runs of BERT on RTE where we replace the weights of the top-k layers with their pre-trained values. We can observe that it is often sufficient to reset around 10 layers out of 24 to recover back the language modeling abilities of the pre-trained model. (c) shows the average training loss and development accuracy (# 1std) for 10 failed fine-tuning runs RTE, Failed mne-tunin runs lead to trivial training loss suggesting an optimization problem.
0.90
0.70
0.90
0.60
0.70
8 0.60
0.50
0.50
0.40
0.70
0.30
0.30
0.20
0.20
0.60
0.10
majerty demalerr
0.10
by cleet
0.00
Full train set
1k points
1k points
Full train set
1k points
1k points
Full train set
1k points
1k points
spachs
epochs
11 epochs
epochs
3 epochs
26 epochs
3 eeochs
epochs
312 epochs
(a) MRPC
(b) CoLA
(c) QNLI
results down-sampled MRPC, CoLA, and QNLI using the default fine-tuning scheme of BERT (Devlin et al. 2019). The leftmost boxplot in each sub-figure shows the development accuracy when training on the full training set.
Figure 3: Development
4.1 DOES CATASTROPHIC FORGETTING CAUSE FINE-TUNING INSTABILITY?
2017) refers to the phe- nomenon when a neural network is sequentially trained to perform two different tasks, and it loses its ability to perform the first task after being trained on the second. More specifically, in our setup, it means that after fine-tuning pre-trained model, it can no longer perform the original masked language modeling task used for pre-training. This can measured in terms of the perplexity on original training data. Although the language modeling performance of pre- praitraind model correlates with its fine-tuning accuracy (Liu et al. 2019: Lan et al., (200)), there is no clear motivation for why preserving the original masked language modeling performance after fine-tuning is important [
Catastrophic forgetting McCloskey and Cohen 1989
Kirkpatrick
In the context of fine-tuning BERT, Lee et al. (220)) suggest that their regularization method has an effect of alleviating catastrophic forgetting. Thus, it is important understand how exactly catastrophic forgetting occurs during fine-tuning and how it relates to the observed fine-tuning instability. To better understand this, we perform the following experiment: we fine-tune BERT on RTE, following the default strategy by Devlin et al.] (2019). We select three successful and three failed fine-tuning runs and evaluate their masked language modeling perplexity on the test set of the WikiText-2 language modeling benchmark (Merity et al. 20166) We sequentially substitute the top-k layers of the network varying from 0 (i.e. all layers are from the fine-tuned model) to 24 (i.e.
all layers are from the pre-trained model). We show the results in Fig. 2 (a) and (b).
We can observe that although catastrophic forgetting occurs for the failed models (Fig. perplex- ity on WikiText-2 is indeed degraded for k = 0 the phenomenon is much more nuanced. Namely, catastrophic forgetting affects only the top layers of the network - our experiments often around
1 exception could by the case where supervised fine-tuning performed as an intermediate training step, c.g. with the goal of domain adaptation. We leave an investigation of this setting for future work. BERT was trained on English Wikipedia, hence WikiText-2 he subset of its training data.
4

---

Published as a conference paper at ICLR 2021
10 out of 24 layers, and the same is however also true for the successfully fine-tuned models, except for a much smaller increase in perplexity.
Another important aspect of our experiment is that catastrophic forgetting typically requires that the model at Ieast successfully learns how to perform the new task. However, this is not the case for the failed fine-tuning runs. Not only is the development accuracy equal to that of the majority classifier, but also the training loss on the fine-tuning task (here RTE) is trivial, i.e. close to In(2/2) (see Fig. (c)). This suggests that the observed fine-tuning failure is rather an optimization problem causing catastrophic forgetting in the top layers of the pre-trained model. We will show later that the optimization aspect actually sufficient to explain most of the fine-tuming ane-tuning variance.
4.2 Do SMALL TRAINING DATASETS CAUSE FINE-TUNING INSTABILITY?
Having a small training dataset is by far the most commonly stated hypothesis for tine-tuning instability. Multiple recent works (Devlin et al. 2019 Phang et al. 2018 Lee et al.) 2020 Zhu et al. 2020; Dodge et al. 2020 Pruksachatkun et al. (020)) that have observed BERT fine-tuning to be unstable relate this finding the small number of training examples.
To test if having a small training dataset inherently leads to fine-tuning instability we perform the following experiment we randomly sample 1,000 training samples from the CoLA, MRPC, and QNLI training datasets and fine-tune BERT using 25 different random seeds on each dataset. We compare two different settings: first, training for 3 epochs the reduced training dataset, and second, training for the same number of iterations as on the full training dataset. We show the results in Fig. and observe that training on less data does indeed affect the fine-tuning variance, in particular, there are many more failed runs. However, when we simply train for as many iterations as on the full training dataset, we almost completely recover the original variance of the fine-tuning performance. We also observe no failed runs on MRPC and QNLI and only single failed run on CoLA which is similar to the results obtained by training on the full training set. Further, as expected, we observe that training on fewer samples affects the generalization of the model, leading to worse development set performance on all three tasks
We conclude from this experiment, that the role of training dataset size per is orthogonal to fine-tuning stability. What is crucial is rather the number of training iterations. As our experiment shows, the observed increase in instability when training with smaller datasets rather be attributed to the reduction of the number of iterations (that changes the effective learning rate schedule) which, as we will show in the next section, has crucial influence on the fine-tuning stability.
5 DISENTANGLING OPTIMIZATION AND GENERALIZATION IN FINE-TUNING INSTABILITY
Our findings in Section detail that while both catastrophic forgetting and small size of the datasets indeed correlate with fine-tuning instability, none of them are causing it. In this section, we argue that the fine-tuning instability is an optimization problem, and it admits a simple solution. Additionally, we show that even though large fraction of the fine-tuning instability can be explained by optimization, the remaining instability can be attributed to generalization issues where fine-tuning runs with the same training loss exhibit noticeable differences in the development set performance.
5.1 THE ROLE OF OPTIMIZATION
Failed fine-tuning runs suffer from vanishing gradients. We observed in Fig. 2c that the failed runs have practically constant training loss throughout the training (see Fig. 14 in the Appendix for comparison with successful fine-tuning tuninng. In order better understand this phenomenon, we plot the &, gradient norms of the loss function with respect to different layers of BERT, for one failed and successful fine-tuning run. For the failed run we see large enough gradients only for the
Fig.
P We remark that similar experiment was done Phang 20188) but with different goal of showing that their extended pre-training procedure can improve fine-tuning stability.
"Section 7.8 in the Appendix shows that the same holds true for datasets from different domains than the pre-training data.
5

---

Published as a conference paper at ICLR 2021
Gradient norm
Gradient norm
10-2
10-2
10
10-9
200
400
200 Iterations
400
Iterations
Failed run
(b) Successful run
Figure 4: Gradient norms (plotted on logarithmic scale) of different layers on RTE for a failed and successful run of BERT fine-tuning. We observe that the failed run is characterized by vanishing gradients in the bottom layers of the network. Additional plots for other weight matrices be found in the Appendix.
0.85
8.85
0.85
820
5.00
8.00
0.70
0.65
0.0
635
01
051
BERT
(b) RoBERTA
ALBERT
Figure 5: Box plots showing the fine-tuning performance of BERT, ( RoBERTa, (c) ALBERT for different learning rates with and without bias correction (BC) on RTE. For BERT and ALBERT, having bias correction leads stable results and allows train using larger learning rates. For RoBERTa, the effect is less pronounced visible.
top layers and vanishing gradients for the bottom layers. This is in large contrast to the successful run. While we also observe small gradients at the beginning of training (until iteration 70), gradients start to grow training continues. Moreover, at the end of fine-tuning, nearly 2x orders of magnitude larger than that of the failed run. Similar visualizations for additional layers and weights also for RoBERTa and ALBERT models, and the corresponding figures can be found the Appendix well (Fig. and 12).
observe gradient norms
be found in Fig. 10 in the Appendix. Moreover, observe the same behavior
Importantly, we note that the vanishing gradients we observe during fine-tuning are harder to resolve than the standard vanishing gradient problem (Hochreiter 199T Bengio et al. 1994). In particular, common weight initialization schemes (Glorot and Bengio 2010 He et al. (2015) ensure that the pre-activations of each layer of the network have zero mean and unit variance expectation. However, we cannot simply modify the weights of pre-trained model on each layer to ensure this property since this would conflict with the idea of using the pre-trained weights.
Importance of bias correction in ADAM. Following Devlin et al. (2019), subsequent works on fine-tuning BERT-based models use the ADAM optimizer (Kingma and Ba 2015). subtle detail of the fine-tuning scheme of Devlin et al. (2019) is that it does not include the bias correction in ADAM. Kingma and Ba (2015) already describe the effect of the bias correction as reduce the learning rate at the beginning of training. By rewriting the update equations of ADAM as follows, can clearly see this effect of bias correction:
0 a V 3/((1 89)
(1)
0, + 01-1 m cc,
(2)

---

Published as a conference paper at ICLR 2021
15
10
10
9.5
05
121
05
W
073
022
08
0.98
-05
0.35
-20
-10 100
-13
-10
1.0
(a) RTE
(b) MRPC
(c) CoLA
Figure 7: 2D loss surfaces in the subspace spanned by 61 0, 02 and 62 0, 0n RTE, MRPC, and
CoLA. 0p. oy. 0, denote the parameters the pre-trained. failed. and successfully trained model, respectively.
Here and 14 are biased first and second moment estimates respectively. Equation [T] shows that bias correction simply boils down reducing the original step size by multiplica- tive factor /1 / (1- which is significantly below 1 for the first iterations training and approaches as the number of training iterations t increases (see Fig. (6. Along the same lines. You et al.] (2020) explicitly remark that bias correction in ADAM has similar effect to the warmup which is widely used in deep learning to prevent divergence early in training (He al. 2016 Goyal et al. 2017 Devlin al. 2019 Wong
0.8
0.6
0.4
0.2
Figure 6: The bias correction term of
et al. 2020).
ADAM (81 = 0.9 and 832 0.999).
The implicit warmup of ADAM is likely to be an important factor that contributed to its success. We argue that fine-tuning BERT- based language models is not an exception. In Fig. 5 we show the results of fine-tuning on RTE with and without bias correction for BERT, RoBERTa, and ALBERT models We observe that there is significant benefit in combining warmup with bias correction, particularly for BERT and ALBERT. Even though for RoBERTa fine-tuning is already more stable even without bias correction, adding bias correction gives additional improvement.
Our results show that bias correction useful if we want to get the best performance within epochs, the default recommendation by Devlin et al. (20T9). An alternative solution is to simply train longer with a smaller learning rate, which also leads to much more stable fine-tuning. We provide a more detailed ablation study in Appendix (Fig. ( with analogous box plots for BERT using various learning rates, numbers of training epochs, with and without bias correction. Finally, concurrently to work, Zhang al. 1) also make similar observation about the importance of bias correction and longer training which gives further evidence to our findings.
Loss surfaces. To get further intuition about the fine-tuning failure, provide loss surface visu- alizations (Li et al. 2018 Hao et al., 2019) of failed and successful runs when fine-tuning BERT. Denote by pp. y Og the parameters of the pre-trained model, failed model, and successfully trained model, respectively. We plot two-dimensional loss surface f(a, 3) L(Op + aσ; + 382) in the subspace spanned by 61 0, Op and 82 0₂ Op centered at the weights of the pre- pre-trained model fp. Additional details are specified in Section 7.6 in the Appendix.
They provide additional evidence to our findings on vanishing gradients: for failed fine-tuning runs gradient descent converges to a "bad" valley with a sub-optimal training loss. Moreover, this bad valley is separated from the local minimum (to which the successfully trained run converged) by barrier (see also Fig. 13 in the Appendix). Interestingly, we observe highly similar geometry for all three datasets providing further support for interpretation of fine- fine-tuning instability as primarily optimization issue.
Contour plots of the loss surfaces for RTE, MRPC, and CoLA are shown in Fig. 7
'Some of the hyperparameter settings lead to small fine-tuning variance where all runs lead performance helow the majority haseline. Obviously, such fine-tuning stability is of limited use.
7

---

Published as a conference paper at ICLR 2021
75
70
70
Accuracy
65
Accuracy
60
60-
55
50
50
45
0.5k 1.5k 2.0k 2.5k 3.0k Iterations
10
10
10
3
10
10
109
Training loss
(a) Development set accuracy over training
(b) Generalization performance training loss
Figure 8: Development set accuracy for multiple fine-tuning runs RTE. The models for (a) trained with 10 different seeds, and models for (b) are taken the training, and trained with different seeds and hyperparameters.
5.2 THE ROLE OF GENERALIZATION
We now turn to the generalization aspects of fine-tuning instability. order to show that the remaining fine-tuning variance the development set can be attributed to generalization, we perform the following experiment: we fine-tune BERT on RTE for 20 epochs and show the development set accuracy for 10 successful runs in Fig. Sh. Further, show in Fig. (8b the development set accuracy vs. training loss of all BERT models fine-tuned on RTE for the full ablation study (shown in Fig. 9 in the Appendíx), in total 450 models.
We find that despite achieving close to zero training loss overfitting is not an issue during fine-tuning. This is consistent with previous work (Hao al.. 2019), which arrived similar conclusion. Based on our results, we argue that it is even desirable to train for a larger number of iterations since the development accuracy varies considerably during fine-tuning and it does not degrade even when the training loss is as low as 10
Combining these findings with results from the previous section, we conclude that the fine-tuning instability can be decomposed into two aspects: optimization and generalization. In the next section, we propose simple solution addressing both issues.
A SIMPLE BUT HARD-TO-BEAT BASELINE FOR FINE-TUNING BERT
As our findings in Section 5 show, the empirically observed instability of fine-tuning BERT can be attributed to vanishing gradients early in training as well as differences in generalization late in training. Given the new understanding of fine-tuning instability, we propose the following guidelines for fine-tuning transformer based masked language models on small datasets:
Use small learning rates with bias correction avoid vanishing gradients early in training. Increase the number of iterations considerably and train to (almost) zero training loss.
This leads to the following simple baseline scheme: fine-tune BERT using ADAM with bias correction and a learning rate of 2c-5. The training is performed for 20 epochs, and the learning rate is linearly increased for the first 10% of steps and linearly decayed to zero afterward. All other hyperparameters are kept unchanged. A full ablation study on RTE testing various combinations of the changed hyperparameters is presented in Section 7.4 in the Appendix.
Results. Despite the simplicity of our proposed fine-tuning strategy, we obtain strong empirical performance. Table[]and Fig. [ssoww the results of fine-tuning BERT on RTE, MRPC, and CoLA. We compare to the default strategy of Devlin et al. (2019) and the recent Mixout method from Lee et al. (2020). First, we observe that our method leads to a much more stable fine-tunigg performance on all three datasets evidenced by the significantly smaller standard deviation of the final performance. To further validate our claim about the fine-tume tuning stability, we run Levene's test (Levene) 1960) to check the equality of variances for the distributions of the final performances each dataset. For all
8

---

Published as a conference paper at ICLR 2021
Table I: Standard deviation, mean, and maximum performance the development set of RTE, MRPC, and CoLA when fine-tuning BERT over 25 random seeds. Standard deviation: lower is better, i.e. fine-tuning is more stable. denotes significant difference (p < 0.001) when compared to the second smallest standard deviation.
RTE
MRPC
CoLA
Approach
std
mean
max
std
mean
max
std
mean
Devlin et al.
2019
4.5
50.9
67.5
3.9
84.0
91.2
25.6
45.6
64.6
Lee et al. 2020)
7.9
65.3
74.4
3.8
87.8
91.8
20.9
51.9
64.0
Ours
2.7*
67.3
71.1
0.8"
90.3
91.7
1.8"
62.1
65.3
three datasets, the test results in a p-value less than 0.001 when we compare the variances between our method and the method achieving the second smallest variance. Second, we also observe that our method improves the overall fine-tuning performance: Table [ we achieve a higher mean value on all datasets and also comparable or better maximum performance MRPC and CoLA respectively.
Finally, we note that we suggest to increase the number of fine-tuning iterations only for small datasets, and thus the increased computational cost of our proposed scheme is not a problem in practice. Moreover, we think that overall our findings lead to more efficient fin-tuning because the significantly improved stability which effectively reduces the number of necessary fine-tuning runs.
7 CONCLUSIONS
In this work, we have discussed the existing hypotheses regarding the reasons behind fine-tuning instability and proposed a new baseline strategy for fine-tuning that leads to significantly improved fine-tuning stability and overall improved results on commonly used datasets from the GLUE benchmark.
By analyzing failed fine-tuning runs, we find that neither catastrophic forgetting nor small dataset sizes sufficiently explain fine-tuning instability. Instead, our analysis reveals that fine-tuning instability can be characterized by two distinct problems: (I) optimization difficulties early in training, characterized by vanishing gradients, and (2) differences in generalization, characterized by large variance of development set accuracy for runs with almost equivalent training performance.
Based on our analysis, we propose a simple but strong baseline strategy for fine-tuning BERT which outperforms the previous works in terms of fine-tuning stability and overall performance.
ACKNOWLEDGMENTS
We thank Anna Khokhlova for her help with the language modeling experiments, Cheolhyoung Lee and Jesse Dodge for providing us with details of their works, and Badr Abdullah and Aditya Mogadala for their helpful comments on a draft of this paper.
This work was funded by the Deutsche Forschungsgemeinschatt (DFG, German Research Foundation) Project-ID 232722074 - SFB 1102.
REFERENCES
Roy Bar-Haim, Ido Dagan, Bill Dolan, Lisa Ferro, and Danilo Giampiccolo. 2006. The second pascal recognising textual entailment challenge. Proceedings of the Second PASCAL Challenges Workshop Recognising Textual Entailment.
Yoshua Bengio, Patrice Simard, and Paolo Frasconi. 1994. Learning feng-term dependencies with gradient descent difficult. IEEE transactions on neural networks, 5(2):177-66.
Luisa Bentivogli, Ido Dagan, Hoa Trang Dang, Danilo Giampiccolo, and Bernardo Magnini. 2009. The fifth pascal recognizing textual entailment challenge. In Proceedings of the Second Text Analysis Conference (TAC 2009).

---

Published as a conference paper at ICLR 2021
Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The pascal recognising textual entailment challenge] In Proceedings of the First International Conference on Machine Learning Chal- lenges: Evaluating Predictive Uncertainty Visual Object Classification, and Recognizing Textual Entailment, MI.CW'05, page 177-190, Berlin, Heidelberg. Springer-Verlas Verlag.
Jacob Devlin, Ming-Wer Wei Chang, Kenton Lee, and Kristina Toutanova. 2019, BERT: Pre-training of deep bidirectional transformers for language understanding In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Minneapolis, Minnesota.
Technologies, Volume / (Long and Short Papers), pages Association for Computational Linguistics.
Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah Smith. 2020. Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping. arXiv preprint ar arXiv:2002.06305.
William B. Dolan and Chris Brockett. 2005. Automatically constructing a corpus of sentential paraphrases, In Proceedings of the Third International Workshop on Paraphrasing (IWP2005).
Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. 2007. The third pascal recog- nizing textual entailment challenge. In Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing. RTE '07, page 1-9, USA. Association for Computational Linguis-
tics.
Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 249-256.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. 2017. Accurate large minibatch sgd: Training imagenet in hour
Yaru Hao, Li Dong, Furu Wei, and Ke Xu. 2019. Visualizing and understanding the effectiveness of BERT In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCNLP), pages 4143-4152, Hong Kong, China. Association for Computational Linguistics.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEFE international conference on computer vision, pages 1026-1034.
Kaiming Hc, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778.
Sepp Hochreiter. 1991. Untersuchungen zu dynamischen neuronalen netzen. Diploma, Technische Universität München, 91(1).
Tushar Khot, Ashish Sabharwal, and Peter Clark. 2018 Scitail: A textual entailment dataset from science question answering
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization In International Conference on Learning Representations.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barre Barwinska, and et al. 2017. Overcoming catastrophic forgetting in neural networks) Proceedings of the National Academy of Sciences, 114(13)352--3352
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. Albert: A lite bert for self-supervised learning of language representations In International Conference Learning Representations.
10

---

Published as a conference paper at ICLR 2021
Cheolhyoung Lee, Kyunghyun Cho, and Wanmo Kang. 2020. Mixout: Effective regularization
to finetune large-scale scale pretrained language models In International Conference on Learning Representations.
Howard Levene. 1960. Robust tests for equality of variances. Contributions probability and statistics: Essays in honor of Harold Hotelling, pages 278-292.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. 2018. Visualizing the loss landscape of neural nets, In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems 31, pages 6389-6399,
Curran Associates, Inc.
Liyua Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. 2020. On the variance of the adaptive learning rate and beyond, In International Conference on Learning Reprexentations.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: robustly optimized bert pretraining approach. arXiv preprint 197.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization In International Conference on Learning Representations.
Michael McCloskey and Neal J Cohen. 1989. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pages 109-165. Elsevier.
R. Thomas McCoy, Junghyun Min, and Tal Linzen. 2020. BERTs of a feather do not generalize together: Large variability in generalization across models with similar test set performance In Proceedings of the Third BlackboxNLP Workshop Analyzing and Interpreting Neural Networks for NLP, pages 217-227, Online, Association for Computational Linguistics.
Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference) In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3428-3448, Florence, Italy, Association for Computational Linguistics.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture models. arXiv preprint arXiv: / 1609 07844.7.3
Timothy Niven and Hung- Yu Kao. 2019. Probing neural network comprehension of natural language arrgmments] In Proceedings of the 57th Anmual Meeting of the Association for Computational Linguistics, pages 4658 4664, Florence, Italy. Association for Computational Linguistics.
Jason Phang, Thibault Févry, and Samuel R Bowman. 2018. Sentence encoders on stilts: Supplemen- tary training on intermediate labeled-date tasks. arXiv preprint arXiv: 1810.08
Yada Pruksachatkun, Jason Phang. Haokun Liu, Phu Mon Htut, Xiaoyi Zhang, Richard Yuanzhe Pang, Clara Vania, Katharina Kann, and Samuel R. Bowman. 2020. Intermediate-lask transfer learning with pretrained language models: When and why does it work? In Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics, pages Association for Computational Linguistics.
Online.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQUAD: 100,000+
questions for machine comprehension of text In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-2392, Austin, Texas. Association for Computational Linguistics.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Oner Levy, and Samuel Bowman. 2019a. Superglue: A stickier benchmark for general-purpose language understanding systems| In H. Wallach, H. Larochelle, A. Beygelzimer. F. d' Alché Buc, E. Fox. and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 2266-3280 Curran Associates, Inc.
11

---

Published as a conference paper at ICLR 2021
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019b. GLUE: A multi- benchmark and analysis platform for natural language understanding In International Conference on Learning Representations.
Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. 2018. Neural network acceptability judgments. arXiv preprint arXiv: 124711.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, al. 2019. Transformers: State-of-the- art natural language processing, arXiv preprint arXiv:1910.037)
Eric Wong, Leslie Rice, and J. Zico Kolter. 2020. Fast is better than free: Revisiting adversarial training In International Conference on Learning Representations.
Ruibin Xiong, Yunchang Yang. Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tie- Yan Liu. 2020. On layer normalization in the transformer architecture. arXiv preprint vrXiv:2002.04745
Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. 2020. [arge batch optimization for deep learning: Training bert in 76 minutes In International Conference Learning Representations.
Tianyi Zhang, Felix Wu, Arzoo Katiyar, Kilian Q Weinberger, and Yoav Artzi 2021. Revisiting few-sample BERT fine-tuning In International Conference on Learning Representations.
Chen Zhu, Yu Cheng, Zhe Gan, Siqi Sun, Tom Goldstein, and Jingjing Liu. 2020. Freelb: Enhanced adversarial training for natural language understanding In International Conference on Learning Representations.
12

---

Published as a conference paper at ICLR 2021
APPENDIX
7.1
ALTERNATIVE NOTIONS OF STABILITY
Here, elaborate other possible definitions of fine-tuning stability. The definition that throughout the paper follows the previous work (Phang et al. 2218) Dodge et al.] 2020 Lee et al. 2020). For example, while Dodge et al. (2020) do not directly define fine-tuning stability, they report and analyze the standard deviation of the validation performance (e.g., see Section 4.1 of their paper), Along the
lines, an earlier work of Phang al. (0118), which studies the influence of intermediate fine-tuning, discusses the variance of the validation performance (see Section 4: Results, paragraph Fine- Tining Stability therein) and shows the standard deviation over multiple random seeds in Figure 1.
For simplicity, let us assume that the performance metric is accuracy and we have two classes. Let A be a randomized fine-tuning algorithm that produces classifier fA, and let us denote data points as (xr, y) DD where D is the data- gentratnnn distribution. Our definition of fine-tuning stability can be formalized as follows:
Souns = Var (Ezyy [[]]]]]]]] = Var [Accuracy(fa)]
This definition directly measures the variance of the performance metric and aims to answer the question: If we perform fine-tuning multiple times, how large will the difference in performance be?
An alternative definition of fine-tuning stability that could be considered is per-point stability where the expectation and variance are interchanged:
Sper-point (A) = Ez.9 [Var [+fr(=)=]]]
This definition captures a different notion of stability. Namely, it captures stability per data point by measuring how much the classifiers fA differ on the same point given label y. Studying the per-point fine-tuning stability can be useful to better understand the properties of fine-tuned models and we refer to McCoy et al. (2020) for study in this direction.
7.2 TASK STATISTICS
Statistics for each of the datasets studied in this paper are shown in Table 2 All datasets are publicly available. The GLUE datasets can be downloaded here: /ttpps://qithtii com/ nyu-m11/ jiant, SciTail is available at https::/// /ttps://giii http:///ggttu.uuue com a www.e/aiiicta.
Table 2: Dataset statistics and majority baselines.
RTE
MRPC
CoLA
QNLI
SciTail
Training
2491
3669
8551
104744
23596
Development
278
409
1043
5464
1304
Majority baseline
0.53
0.75
0.0
0.50
0.63
Metric
Acc.
F1 score
MCC
Acc.
Acc.
7.3 HYPERPARAMETERS
Hyperparameters for BERT, RoBERTa, and ALBERT used for all our experiments are shown in Table 3
7.4 ABLATION STUDIES
Figure 9 shows the results of fine-tuning on RTE with different combinations of learning rate, number of training epochs, and bias correction. We make the following observations:
When training for only 3 epochs, disabling bias correction clearly hurts performance.
With bias correction, training with larger learning rates is possible.
Combining the usage of bias correction with training for more epochs leads to the best performance.
13

---

Published as a conference paper at ICLR 2021
maximum
majority classifier
mean
1e-5 + epochs
2e + epochs
x
3e + epochs
epochs
epochs
3e-5 10 epochs
1e-5 20 epochs
26-5 20 epochs
*
3 20 epochs
le-5 epochs BC
2e-5 epochs BC
0
3e-5 5 epochs BC
0
00
0
*
1e-3 5 + 10 epochs BC
2e-5 + 10 epochs BC
3e-5 + 10 epochs BC
X
1e-5 + 20 epochs BC
2e-5 + 20 epochs BC
X
3e-5 + 20 epochs BC
0.45
0.50
0.55
0.60
0.65
0.70
0.75
Accuracy
Figure Full ablation of fine-tuning BERT on For each setting, we vary only the number of training steps. learning rate, and usage of hias correction (BC). All other hyperparameters are unchanged. We fine-tune 25 models for each setting. shows the setting which we recommend as a new baseline fine-tuning strategy.
14

---

Published as a conference paper at ICLR 2021
Table 3: Hyperparameters used for fine- tie-ttuing.
Hyperparam
BERT
RoBERTa
ALBERT
Epochs
3, 20,00, 20
3
3
Learning rate
le-5 5e-5
le-5 3e-5
le-5 - 3e-5
Learning rate schedule
warmup-linear
warmup-linear
warmup- lineer
Warmup ratio
0.1
0.1
0.1
Batch size
16
16
16
Adam
le-6
le-6
le-6
Adam 81
0.9
0.9
0.9
Adam B2
0.999
0.98
0.999
Adam bias correction
(True, False)
(True, False}
(True, False]
Dropout
0.1
0.1
Weight decay
0.01
0.1
Clipping gradient norm
1.0
1.0
Number of random seeds
25
25
7.5 ADDITIONAL GRADIENT NORM VISUALIZZTIONNS
We provide additional visualizations for the vanishing gradients observed when fine-tuning BERT,
RoBERTa, and ALBERT in Figures I0 II 12
Note that for ALBERT besides the pooler and classification layers, plot only the gradient norms of single hidden layer (referred as layer0) because of weight sharing.
Gradient norms and MLM perplexity. We can see from the gradient norm visualizations for BERT in Figures 4 and 10] that the gradient norm of the pooler and classification layer remains large. Hence, even though the gradients on most layers of the model vanish, we still update the weights on the top layers. In fact, this explains the large increase in MLM perplexity for the failed models which is shown in Fig. 2a While most of the layers do not change as we continue training, the top layers of the network change dramatically.
7.6
LOSS SURFACES
define the range for both and 3 [-1.5, 1.5) and sample 40 points for each axis. We evaluate the loss on 128 samples from the training dataset of each task using all model parameters, including the classification layer. We disabled dropout for generating the surface plots.
For Fig. 7
Fig. 13 shows contour plots of the total gradient norm. We can again see that the point to which the failed model converges to (0y) is separated from the point the successful model converges (0,) by a barrier. Moreover, on all the three datasets we can clearly see the valley around B with small gradient norm.
7.7 TRAINING CURVES
Fig. 14 shows training curves for 10 successful and 10 failed fine-tuning runs on RTE, We can clearly observe that all 10 failed runs have common pattern: throughout the training, their training loss stays close to that at initialization. This implies an optimization problem and suggests to reconsider the optimization scheme.
7.8 ADDITIONAL FINE-TUNING RESULTS
We report additional fine-tuning results on the SciTail dataset (Khot et al., 2018 in Table to demonstrate that our findings generalize to datasets from other domains.
Due to its comparatively large size (28k training samples) fine-tuning on SciTail with the Devlin et al. (2019) scheme is already very stable even when trained for 3 epochs. This is comparable to what find for QNLI Section 4.1 When applying our fine-tuning scheme to SciTail, the results are very close to that of Devlin et al. (201)). On the other hand, when training smaller subset of SciTail
15

---

Published as a conference paper at ICLR 2021
lose
low
Chall cadde Naxt
Cianlication leel
Gradient norm
10
Gradient norm
10
10-2
10-2
10
10
200
400
200 Iterations
400
Iterations
(a) Failed run: attention.key
(b) Successful run: attention.key
101
Gradient norm
10a
Gradient
10 2.
10 2.
10-5
10-5
200
400
200 Iterations
400
Iterations
(c) Failed run: attention query
(d) Successful run: attention.quee
Layer
Gradient norm
10a
Gradient norm
10a_
10-2.
10-2
10-5
10-5
200
400
200
400
Iterations
Iterations
(e) Failed run: attention.valle value
(f) Successful run: attention. value
Figure Gradient norms (plotted a logarithmic scale) of additional weight matrices of BERT fine-tuned on RTE, Corresponding layer names are in the captions. We show gradient norms corresponding to a single failed and single successful, respectively.
16

---

Published as a conference paper at ICLR 2021
Gradient norm
Gradient norm
10a
w
10
10-1
10-3
10-3
200
400
200 Iterations
400
Iterations
(a) Failed run: attention.ker key
(b) Successful run: attention key
Gradient norm
10
Gradient norm
10a
10-
10-1
10-3
10-3
200
400
200 Iterations
4000
Iterations
(c) Failed run: attention.guue.. query
(d) Successful run: attention quenrtoo.queryy query
10-
Gradient norm
10
Gradient
10-1-
10-1
10
10-3
200 Iterations
400
200 Iterations
(e) Failed run: attention. value
(1) Successful run: attention value
Gradient norm
Gradient norm
10-1-
10-1
10
200
400
200
Iterations
Iterations
(g) Failed run:
attention.outpuu..ou
(h) Successful run: cetauiionnoutut..deess
Figure 11: Gradient norms (plotted on a logarithmicc scale) of additional weight matrices of RoBERTa fine-tuned
on RTE. Corresponding layer names in the captions. We show gradient norms corresponding single failed and single successful, respectively.
17

---

Published as a conference paper at ICLR 2021
Gradient norm
10:
Gradient norm
10a
10-2
10-2
10-3
10-3
200
400
0
200 Iterations
400
Iterations
(a) Failed run: attention.key
(b) Successful run: attention. key
Gradient norm
10-
Gradient morm
10a
10-2
10-2
10-5
10-5
200 Iterations
400
0
200 Iterations
400
(c) Failed run: attention.queey
(d) Successful run: attention query
Gradient norm
10
Gradient norm
10
10-2.
10-2
10--
10-5
200 Iterations
400
200 Iterations
(c) Failed run: attention, value
(1) Successful rum: attention value
Gradient norm
10
Gradient norm
10at
10-2
10-2
10-3
10-5
200
400
200
Iterations
Iterations
(g) Failed run: attention.dens.
(h) Successful run: attention. dense
Figure 12: Gradient norms (plotted on a logarithmic scale) of additional weight matrices of ALBERT fine-tuned on RTE. Corresponding layer names in the captions. We show gradient norms corresponding to single failed and single successful, respectively.
18

---

Published as a conference paper at ICLR 2021
13
00
01
-05
26
-20
16
-12
1.0
35
15
(a) RTE
(b) MRPC
(c) CoLA
Figure 13: 2D gradient surfaces in the subspace spanned by di 0gp and d2 0, 0p for BERT fine-tuned on RTE, MRPC and CoLA. 0,. 0y. 0. denote the parameters the pre-trained, failed, and successfully trained model, respectively.
0.75
0.8
0.75
0.8
0.70
0.70
0.6
0.6
0.65
0.65
Training lo0z
loss
0.60
0.4
0.60
0.4
0.55
0.55
0.50
0.2
0.50
0.2
0.45
0.45
0.0
0.0
0.40
0.40
0
2000
0
200
400
Iterations
Iterations
(a) Successful runs
(b) Failed runs
Figure 14: The accuracy and training loss (a) 10 successful runs with our fine-tuning scheme and (b) 10 failed runs with fine-tuning scheme Devlin on RTE. Solid line shows mean, error bars show 11d.
Table 4: Standard deviation, mean, and maximum performance on the development set of SciTail when fine- tuning BERT over 25 random seeds. Standard deviation: lower is better, i.c. fine-tuning is more stable.
SciTail
Approach
std
Devlin et al.
2019) full train set, 3 epochs
0.4
95.1
96.0
Devlin 2019). Ik samples, epochs
17.9
69.8
89.4
Devlin
2019), samples, 71 epochs
0.9
87.5
89.1
Ours, full train
0.6
95.1
96.8
(Ik training samples) we can clearly see the same results as also observed in Fig. for MRPC, CoLA, and QNLI, i.e. using more training iterations improves the fine-tuning stability.
We conclude from this experiment that our findings and guidelines generalize to datasets from other domains well. This gives further evidence that the observed instability is not property of any particular dataset but rather result of too few training iterations based on the common fine-tuning practice of using fixed number of epochs (and iterations) independently of the training data size.
19


Reviews:
[{"review_id": "foyAUHbCnmu", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "################################ \n\nSummary:\n\nThis paper considers the stability of fine-tuning BERT-LARGE models, with considerations for RoBERTa and ALBERT. In particular, it aims to demonstrate that previously identified reasons, catastrophic forgetting and small fine-tuning datasets, fail to explain the observed instability. Instead, it posits that the instability is caused by optimization difficulties that lead to vanishing gradients.\n\n################################ \n\nReasons for score:\n\nOverall, I lean toward accept. The underlying argument is simple and slightly advances a conversation about the underlying workings of BERT-based models while leaving substantial room for future exploration. This work would be strengthened by clarifying prescriptive guidance on the baseline recommended and experimentation on additional datasets.\n\n################################ \n\nStrengths:\n\n- Broad interest. BERT-based models have become prevalent in NLP due to their strong empirical performance. However, variation in fine-tuning hinders the consistency of the demonstrated gains. While existing hypotheses have been made, this work provides another point of evidence to advance the conversation around fine-tuning.\n\n- Presentation. The linearity of the argument is easy to follow and stylistic choices, e.g. the motivating question in the introduction, highlight the key aspects of this work. As a reader this is appreciated insofar as claims are stated unambiguously and then can be validated by the work that follows rather than intuited.\n\n################################ \n\nWeaknesses:\n\n- Prescriptive baseline. The two guidelines in Section 6 are excellent, but as a reader I immediately wonder how these generalize to the datasets where I might apply BERT-based models. The guidelines of using a \"small learning rate with bias correction\" and \"increase the number of iterations\" are good, but can they be characterized in more detail? In other words, how can I make these (hopefully) generalizable insights concrete for future applications?\n\n- Additional datasets. While the existing datasets are sufficient for illustrating the arguments made, additional datasets would further support generalizable insights by providing additional points of evidence with respect to dataset size, and variance. This may support the point above regarding a characterization of large vs. small.\n\n################################ \n\nQuestions:\n\n- Is there any additional insight you can provide characterizing the baseline? If I were to apply these insights to a NLI dataset in another domain, e.g. MedNLI, could I expect these to hold?\n\n- While the aspects of pretraining and continual pretraining are explicitly deferred in the related works section, are there any insights you can provide about their impact on these findings for fine-tuning?\n\n\n################################ \n\nAfter response:\n\nThank you for the clarifications. The response and changes address several of my concerns. While I will keep the score currently, I would consider it slightly higher given the additional information (i.e. ~6.5).", "extra_scores": {}, "extra_text": {}}, {"review_id": "QZZiAFdLpQ", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper explores why fine-tuning is an unstable process, and proposes a strategy to stabilize it.\n\nThe experiments use BERT, RoBERTa, and ALBERT, fine-tuned on three popular datasets from the GLUE benchmark. \n\nBased on the analysis of failed fine-tuning runs, the authors conclude that while catastrophic forgetting and small size of the datasets used for finetuning (both of which are common explanations for fine-tuning instability)   indeed correlate with fine-tuning instability, they do not directly cause it.\n\nIn a set of experiments it is shown that instead, fine-tuning instability is caused by: (1) optimization difficulties early in training, \ncharacterized by vanishing gradients, and (2) differences in generalization, characterized by a large variance of development set accuracy for runs with similar training performance.\n\nTo address these issues, the authors propose the following guidelines for fine-tuning transformer-based masked language models on \\emph{small} datasets:\nUse small learning rates with bias correction to avoid vanishing gradients early in training; and, increase the number of iterations considerably and train to (almost) zero training loss.\n\nPros:\n- The experimental approach is insightful and seems solid\n- The paper is exceptionally well-written and illustrated\n- A contribution of practical importance for a large community\n\nCons:\n- The focus is on a particular type of problem: transformer-based masked language models that are fine-tuned on small datasets\n- I noted that the proposed fine-tuning guidelines are only evaluated using BERT, whereas the failure analyses apply also to RoBERTa and AlBERT. (Does that mean that AlBERT and RoBERTa benefit less from the proposed guidelines?)\n\nMore comments:\n\nThe authors make careful word choices throughout. I'm not sure that I like this sentence though - \"there is no clear motivation for why preserving the  original masked language modeling performance after fine-tuning is important\". This obviously depends on the task. The end task could involve  slot-filling for example, with fine-tuning used for domain adaptation.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "JaY7QfNW79", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "What is the goal of the paper?\nInvestigating stability of fine-tuning BERT. \n\nWhat has been done before?\nFinetuning BERT exhibit a large training instability (Devlin et al., 2019; Dodge et al., 2020) i.e. training the same model with multiple random seeds can result in a large variance of the task performance.\n\nFew methods have been proposed to solve the observed instability (Phang et al., 2018; Lee et al., 2020), however without providing a sufficient understanding of why fine-tuning is prone to such failure.\n\nThis work tries to answer the question : Why is fine-tuning prone to failures and how can we improve its stability?\n\nAnother line of work investigates optimization difficulties of pre-training transformer-based language models (Xiong et al., 2020; Liu et al., 2020). Both works focus on pre-training and thus orthogonal to this work.\n\nWhat are the contributions of the paper?\nInvestigating two common hypotheses for fine-tuning instability: catastrophic forgetting and small size of the fine-tuning datasets and demonstrating that both hypotheses fail to explain fine-tuning instability.\n\nInvestigating fine-tuning failures on datasets from the popular GLUE benchmark and show that the observed fine-tuning instability can be decomposed into two separate aspects: (1) optimization difficulties, characterized by vanishing gradients, and (2) differences in generalization, characterized by a large variance of development set accuracy for runs with almost equivalent training loss.\n\nPresented a simple but strong baseline that makes fine-tuning BERT-based models significantly more stable than the previously proposed approaches.\n\n\t\t\t\nWhat are the key techniques/experiments used to investigate this task?\nTo investigate catastrophic forgetting - comparing language modeling perplexity for failed and successful fine-tuning runs of BERT.\nTo investigate the effect of small size of the fine-tuning datasets - compare fine-tuning using  downsampled data sets for 3 epochs (standard) vs. more epochs.\nVisualizing gradient norms of different layers for a failed and successful run of BERT fine-tuning. \nLoss surface visualizations of failed and successful runs when fine-tuning BERT\nVisualizing development accuracy vs. training loss at the end of the training for all BERT models fine-tuned for the paper\nFine-tuning performance of (a) BERT, (b) RoBERTa, (c) ALBERT for different learning rates α with and without bias correction (BC)\n\n\t\nWhat are the main results?\nDoes catastrophic forgetting cause fine-tuning instability? No.\nCatastrophic forgetting occurs for both failed and successful models in the top layers of the network,  except for a much smaller increase in perplexity in case of successful models. Catastrophic forgetting typically requires that the model at least successfully learns how to perform the new task. However, this is not the case for the failed fine-tuning runs.\n\nDo small training datasets cause fine-tuning instability? No.\nTraining on less data does indeed affect the fine-tuning variance. However, when one simply trains for as many iterations as on the full training dataset, one almost completely recovers the original variance of the fine-tuning performance.\n\nObserved instability is caused by \noptimization difficulties that lead to vanishing gradient : Development accuracy of failed fine-tuning runs is less or equal to that of the majority classifier, but also the training loss on the fine-tuning task is trivial. This suggests that the observed fine-tuning failure is rather an optimization problem causing catastrophic forgetting in the top layers of the pre-trained model.\ndifferences in generalization : Training on fewer samples affects the generalization of the model, leading to a worse development set performance on all three tasks. The role of training dataset size per se is orthogonal to fine-tuning stability. What is crucial is rather the number of training iterations.\n\nA simple but hard to beat baseline for fine-tuning bert (better standard deviation, better mean, and competitive maximum performance)\n• Use small learning rates with bias correction to avoid vanishing gradients early in training. \n• Increase the number of iterations considerably and train to (almost) zero training loss.\n\nStrengths\nThe approach is well motivated and well-placed in the literature. \nPaper claims look correct technically and are experimentally rigorous.\nPaper is easy and clear to read.\nAuthors have made an attempt to make their findings reproducible.\nFindings apply not only to the widely used BERT model but also to more recent pre-trained models such as RoBERTa and ALBERT. These findings should benefit others as fine tuning BERT based models is a very common practice now.\n\n\nWeaknesses\nThe role of generalization has not been discussed rigorously.\nDifferent sets of GLUE datasets were used for different experiments without any explanation for the selection.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "IYAT-AwIR4Q", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Overview\n\nThe paper focuses on the instability phenomenon happening in the fine-tuning of BERT-like models in downstream tasks. The reasons of such instability were assumed to be catastrophic forgetting and the small size of datasets on which being fine-tuned in previous literature. The authors conduct experiments on several sub tasks of GLUE in an attempt to show the aforementioned two assumptions cannot explain the instability of fine-tuning. Instead they claim that the real reasons are gradient vanishing and the lack of generalization and subsequently propose a set of training hyperparameters to improve the stability.\n\n### Pros\n\nThe analysis of the impact of dataset size is designed clear and concise, which shows it is the number of iterations that really matters rather than the size of datasets. This conclusion is valuable for the audience.\n\nThe suggestions of training hyperparameters provided at the end of the paper is somewhat useful for future researchers.\n\n### Cons\n\nThe authors define term \"stability\" by \"the std of fine-tuning performance (acc, F1, MCC), the lower the better\". This metric is too curtness to precisely represent the real stability. A very simple counter example: assume a fine-tuned model A has 50% accuracy on a binary classification task while model B's predictions are all opposite to model A's predictions, which also makes it a 50% accuracy model. These two models obviously show the most instable situation that they disagree on every single sample. While the std of performance is 0, which by the definition of the authors it is the most stable case. The correct practice is to measure how much it changes on the prediction of the same input between different models.\n\nSince the definition is very important and used through the whole analysis, the fact that it is defined too rough makes the following conclusions in the paper not convincible enough.\n\nBesides, the proposed improvement method is only using slightly different training hyperparameters, which is possible to be covered by a simple grid-search of hyperparameters and therefore cannot be seen as a big contribution.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper identifies the causal factors behind a major known issue in deep learning for NLP: Fine-tuning models on small datasets after self-supervised pretraining can be extremely unstable, with models needing dozens of restarts to achieve acceptable performance in some cases. The paper then introduces a simple suggested fix.

Pros:
- The motivating problem is important: A large fraction of all computing time used on language-understanding tasks involves fine-tuning runs under the protocol studied here, and the problem of fine-tuning self-supervised models should be of broader interest at ICLR.
- The proposed fix is simple and well-demonstrated. It consists of only an adjustment to the range of values considered in hyperparameter tuning (which is significant, since BERT and related papers *explicitly advise* users to use inappropriate values) and an adjustment to the implementation of the optimizer.

Cons:
- The method is demonstrated on a relatively small set of difficult text-classification datasets, so the behavior studied here may be different in very different dataset size, task difficulty, or label entropy regimes.

This paper was divisive, so I gave it a fairly close look myself, and I'm persuaded by R1 and the other two positive reviewers: This is a classic example of a 'strong baselines paper', in that demonstrates that a more careful use of established methods can obviate the need for additional tricks.

R3 raised two major concerns that they presented as potentially fatal, but that I find unpersuasive.
- This paper studies stability in model performance, not stability in predictions on individual data points. R3 argues that the latter sense of stability is the more important problem. Stability is an ambiguous term in this context, and both versions of the problem are interesting. However, as the authors pointed out, the definition of stability that is used here is consistent with previous work, and is widely accepted to be a major practical problem in NLP. I don't think this is a weakness of this paper, rather, it's an opportunity for someone else to write another, different paper on a different problem.
- R3 claims that the results are described as being more positive than they actually are, and the figure is potentially misleading. Looking at the quantitative results with R3's points in mind, I still see clear support for both of the paper's main suggestions. R3 opened up some potentially important questions about the handling of outliers in particular, but these questions were raised too late for the authors to be allowed to respond, and I don't see any evidence in the paper that anything improper was done. The marked outliers are clearly much farther from the mean/median in terms of standard deviations than the unmarked outliers. So, I don't see any evidence that these concerns reflect real methodological problems.
Decision: Accept (Poster)