Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: ### Overview

The paper focuses on the instability phenomenon happening in the fine-tuning of BERT-like models in downstream tasks. The reasons of such instability were assumed to be catastrophic forgetting and the small size of datasets on which being fine-tuned in previous literature. The authors conduct experiments on several sub tasks of GLUE in an attempt to show the aforementioned two assumptions cannot explain the instability of fine-tuning. Instead they claim that the real reasons are gradient vanishing and the lack of generalization and subsequently propose a set of training hyperparameters to improve the stability.

### Pros

The analysis of the impact of dataset size is designed clear and concise, which shows it is the number of iterations that really matters rather than the size of datasets. This conclusion is valuable for the audience.

The suggestions of training hyperparameters provided at the end of the paper is somewhat useful for future researchers.

### Cons

The authors define term "stability" by "the std of fine-tuning performance (acc, F1, MCC), the lower the better". This metric is too curtness to precisely represent the real stability. A very simple counter example: assume a fine-tuned model A has 50% accuracy on a binary classification task while model B's predictions are all opposite to model A's predictions, which also makes it a 50% accuracy model. These two models obviously show the most instable situation that they disagree on every single sample. While the std of performance is 0, which by the definition of the authors it is the most stable case. The correct practice is to measure how much it changes on the prediction of the same input between different models.

Since the definition is very important and used through the whole analysis, the fact that it is defined too rough makes the following conclusions in the paper not convincible enough.

Besides, the proposed improvement method is only using slightly different training hyperparameters, which is possible to be covered by a simple grid-search of hyperparameters and therefore cannot be seen as a big contribution.

review_title: Reviews and Comments

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: What is the goal of the paper?
Investigating stability of fine-tuning BERT. 

What has been done before?
Finetuning BERT exhibit a large training instability (Devlin et al., 2019; Dodge et al., 2020) i.e. training the same model with multiple random seeds can result in a large variance of the task performance.

Few methods have been proposed to solve the observed instability (Phang et al., 2018; Lee et al., 2020), however without providing a sufficient understanding of why fine-tuning is prone to such failure.

This work tries to answer the question : Why is fine-tuning prone to failures and how can we improve its stability?

Another line of work investigates optimization difficulties of pre-training transformer-based language models (Xiong et al., 2020; Liu et al., 2020). Both works focus on pre-training and thus orthogonal to this work.

What are the contributions of the paper?
Investigating two common hypotheses for fine-tuning instability: catastrophic forgetting and small size of the fine-tuning datasets and demonstrating that both hypotheses fail to explain fine-tuning instability.

Investigating fine-tuning failures on datasets from the popular GLUE benchmark and show that the observed fine-tuning instability can be decomposed into two separate aspects: (1) optimization difficulties, characterized by vanishing gradients, and (2) differences in generalization, characterized by a large variance of development set accuracy for runs with almost equivalent training loss.

Presented a simple but strong baseline that makes fine-tuning BERT-based models significantly more stable than the previously proposed approaches.

			
What are the key techniques/experiments used to investigate this task?
To investigate catastrophic forgetting - comparing language modeling perplexity for failed and successful fine-tuning runs of BERT.
To investigate the effect of small size of the fine-tuning datasets - compare fine-tuning using  downsampled data sets for 3 epochs (standard) vs. more epochs.
Visualizing gradient norms of different layers for a failed and successful run of BERT fine-tuning. 
Loss surface visualizations of failed and successful runs when fine-tuning BERT
Visualizing development accuracy vs. training loss at the end of the training for all BERT models fine-tuned for the paper
Fine-tuning performance of (a) BERT, (b) RoBERTa, (c) ALBERT for different learning rates α with and without bias correction (BC)

	
What are the main results?
Does catastrophic forgetting cause fine-tuning instability? No.
Catastrophic forgetting occurs for both failed and successful models in the top layers of the network,  except for a much smaller increase in perplexity in case of successful models. Catastrophic forgetting typically requires that the model at least successfully learns how to perform the new task. However, this is not the case for the failed fine-tuning runs.

Do small training datasets cause fine-tuning instability? No.
Training on less data does indeed affect the fine-tuning variance. However, when one simply trains for as many iterations as on the full training dataset, one almost completely recovers the original variance of the fine-tuning performance.

Observed instability is caused by 
optimization difficulties that lead to vanishing gradient : Development accuracy of failed fine-tuning runs is less or equal to that of the majority classifier, but also the training loss on the fine-tuning task is trivial. This suggests that the observed fine-tuning failure is rather an optimization problem causing catastrophic forgetting in the top layers of the pre-trained model.
differences in generalization : Training on fewer samples affects the generalization of the model, leading to a worse development set performance on all three tasks. The role of training dataset size per se is orthogonal to fine-tuning stability. What is crucial is rather the number of training iterations.

A simple but hard to beat baseline for fine-tuning bert (better standard deviation, better mean, and competitive maximum performance)
• Use small learning rates with bias correction to avoid vanishing gradients early in training. 
• Increase the number of iterations considerably and train to (almost) zero training loss.

Strengths
The approach is well motivated and well-placed in the literature. 
Paper claims look correct technically and are experimentally rigorous.
Paper is easy and clear to read.
Authors have made an attempt to make their findings reproducible.
Findings apply not only to the widely used BERT model but also to more recent pre-trained models such as RoBERTa and ALBERT. These findings should benefit others as fine tuning BERT based models is a very common practice now.


Weaknesses
The role of generalization has not been discussed rigorously.
Different sets of GLUE datasets were used for different experiments without any explanation for the selection.

review_title: Clear, technically correct, experimentally rigorous, reproducible investigation on instability of fine-tuning BERT. 

final_score: rating: 6: Marginally above acceptance threshold
review_content: ################################ 

Summary:

This paper considers the stability of fine-tuning BERT-LARGE models, with considerations for RoBERTa and ALBERT. In particular, it aims to demonstrate that previously identified reasons, catastrophic forgetting and small fine-tuning datasets, fail to explain the observed instability. Instead, it posits that the instability is caused by optimization difficulties that lead to vanishing gradients.

################################ 

Reasons for score:

Overall, I lean toward accept. The underlying argument is simple and slightly advances a conversation about the underlying workings of BERT-based models while leaving substantial room for future exploration. This work would be strengthened by clarifying prescriptive guidance on the baseline recommended and experimentation on additional datasets.

################################ 

Strengths:

- Broad interest. BERT-based models have become prevalent in NLP due to their strong empirical performance. However, variation in fine-tuning hinders the consistency of the demonstrated gains. While existing hypotheses have been made, this work provides another point of evidence to advance the conversation around fine-tuning.

- Presentation. The linearity of the argument is easy to follow and stylistic choices, e.g. the motivating question in the introduction, highlight the key aspects of this work. As a reader this is appreciated insofar as claims are stated unambiguously and then can be validated by the work that follows rather than intuited.

################################ 

Weaknesses:

- Prescriptive baseline. The two guidelines in Section 6 are excellent, but as a reader I immediately wonder how these generalize to the datasets where I might apply BERT-based models. The guidelines of using a "small learning rate with bias correction" and "increase the number of iterations" are good, but can they be characterized in more detail? In other words, how can I make these (hopefully) generalizable insights concrete for future applications?

- Additional datasets. While the existing datasets are sufficient for illustrating the arguments made, additional datasets would further support generalizable insights by providing additional points of evidence with respect to dataset size, and variance. This may support the point above regarding a characterization of large vs. small.

################################ 

Questions:

- Is there any additional insight you can provide characterizing the baseline? If I were to apply these insights to a NLI dataset in another domain, e.g. MedNLI, could I expect these to hold?

- While the aspects of pretraining and continual pretraining are explicitly deferred in the related works section, are there any insights you can provide about their impact on these findings for fine-tuning?


################################ 

After response:

Thank you for the clarifications. The response and changes address several of my concerns. While I will keep the score currently, I would consider it slightly higher given the additional information (i.e. ~6.5).
review_title: Encouraging Discussion through Simplicity

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper explores why fine-tuning is an unstable process, and proposes a strategy to stabilize it.

The experiments use BERT, RoBERTa, and ALBERT, fine-tuned on three popular datasets from the GLUE benchmark. 

Based on the analysis of failed fine-tuning runs, the authors conclude that while catastrophic forgetting and small size of the datasets used for finetuning (both of which are common explanations for fine-tuning instability)   indeed correlate with fine-tuning instability, they do not directly cause it.

In a set of experiments it is shown that instead, fine-tuning instability is caused by: (1) optimization difficulties early in training, 
characterized by vanishing gradients, and (2) differences in generalization, characterized by a large variance of development set accuracy for runs with similar training performance.

To address these issues, the authors propose the following guidelines for fine-tuning transformer-based masked language models on \emph{small} datasets:
Use small learning rates with bias correction to avoid vanishing gradients early in training; and, increase the number of iterations considerably and train to (almost) zero training loss.

Pros:
- The experimental approach is insightful and seems solid
- The paper is exceptionally well-written and illustrated
- A contribution of practical importance for a large community

Cons:
- The focus is on a particular type of problem: transformer-based masked language models that are fine-tuned on small datasets
- I noted that the proposed fine-tuning guidelines are only evaluated using BERT, whereas the failure analyses apply also to RoBERTa and AlBERT. (Does that mean that AlBERT and RoBERTa benefit less from the proposed guidelines?)

More comments:

The authors make careful word choices throughout. I'm not sure that I like this sentence though - "there is no clear motivation for why preserving the  original masked language modeling performance after fine-tuning is important". This obviously depends on the task. The end task could involve  slot-filling for example, with fine-tuning used for domain adaptation.

review_title: An interesting and potentially impactful analysis on the fine-tuning procedures of BERT and variants

Decision: Accept (Poster)