Mixout: Effective Regularization to Finetune Large-scale Pretrained Language Models

Abstract: In natural language processing, it has been observed recently that generalization could be greatly improved by finetuning a large-scale language model pretrained on a large unlabeled corpus. Despite its recent success and wide adoption, finetuning a large pretrained language model on a downstream task is prone to degenerate performance when there are only a small number of training instances available. In this paper, we introduce a new regularization technique, to which we refer as “mixout”, motivated by dropout. Mixout stochastically mixes the parameters of two models. We show that our mixout technique regularizes learning to minimize the deviation from one of the two models and that the strength of regularization adapts along the optimization trajectory. We empirically evaluate the proposed mixout and its variants on finetuning a pretrained language model on downstream tasks. More specifically, we demonstrate that the stability of finetuning and the average accuracy greatly increase when we use the proposed approach to regularize finetuning of BERT on downstream tasks in GLUE.

Reviews and author responses:

The authors introduce a new regularization technique for the specific task of finetuning models. It's inspired by dropout and stochastically mixes source and target weights in order to avoid moving the parameters towards 0.

The authors provide a theoretical justification as to why mixout would do useful things in the convex case and then demonstrate empirically that using it achieves good accuracies on some downstream, finetuned, non-convex-loss-utilizing tasks. Their experiments incorporate both small models with good analysis (ie, sec 4) as well as larger, real-world models (sect 5). The paper is in general well-written.

I have a few concerns that I would like to see addressed:

1a. For starters, all the theoretical motivation describes a particular way in which mixout is supposed to aid in downstream tasks for the case of convex functions, but there are no experiments that match their assumptions and which demonstrate this is the actual behavior we see. It would be nice to see results which demonstrate the theory.

1b. In few of the prsented empirical experiments is it the case that the use of mixout by itself is useful.  Why does mixout have to be coupled with other regularization techniques? There is little analysis given here, either empirical or theoretical. 

2. Why are there only 4 GLUE tasks reported? Devlin 2018 reports on all but WNLI.

3. The choice of hyperparameters for GLUE in sect 5 is a bit misleading. Devlin 2018 chose those parameters to get the maximum scores on downstream tasks; the metric they use is max score. However, the authors want to instead discuss the average score against a set of random restarts, perhaps because the max scores using their method aren't terribly different from the baselines. 

Therefore, a more extensive hyperparameter sweep should have been run for the baselines: they should have been re-tuned for the average score if that is the metric the authors wish to use. Instead, the authors only used one task, RTE, to find baseline hyperparameters.

Dear all Reviewers,

we thank you for your valuable reviews and comments. According to your suggestions, we have made more discussions and experiments to improve our paper. In particular, we added/changed the following contents:

1-R1) We have discussed the time usage to finetune BERT_LARGE with the proposed mixout compared to dropout on RTE (Section 6.3 colored in brown & Supplemental E).
2-R2) We have modified the caption of Figure 3 to clarify our experimental setups (Figure 3 colored in orange). 
3-R4) We have performed the least squares regression to see the behavior of mixout when the loss function is strongly convex (Section 3.1 colored in olive & Supplemental D).
4-R4) We have run additional experiments to check whether dropout(0.1) is optimal in terms of mean dev score (Section 5 colored in olive & Supplemental F).  

We hope our responses can help address your concerns and questions.

Thank you for your comments!

“For 20 random restarts, are they the same across regularization setups, i.e., for each dot in Figure 3, is there an orange dot that has the same initialization?”

Yes, we use the same initialization across regularization setups. That is, the initialization of the given blue dot corresponds to each dot colored differently (orange, green, and red).

“What will the extreme case, mixout(w_pre, 1.0), behave? According to Figure 1, it would always use w_pre and end up not learning on a target task. If so, would it introduce a cliff in Fig 4?”

Yes, it introduces a cliff in Figure 4 when we use mixout(w_pre, 1.0). For Figure 4, we used mixout(w_pre, p) for all layers except the additional output layer because the additional output layer was not pretrained and therefore did not have w_pre. So the trainable parameters when we use mixout(w_pre, 1.0) are only the parameters of the additional output layer. We performed the experiment for this extreme case and present the distribution of dev scores from 20 random restarts in https://ibb.co/v3s0c00 . As shown in this link, the dev scores obtained by mixout(w_pre, 1.0) drops significantly. We omitted such extreme cases (dropout(1.0) and mixout(w_pre, 1.0)) to emphasize the monotonic trend of the mix/drop probability in our manuscript.

“3. The choice of hyperparameters for GLUE in sect 5 is a bit misleading. Devlin 2018 chose those parameters to get the maximum scores on downstream tasks; the metric they use is max score. However, the authors want to instead discuss the average score against a set of random restarts, perhaps because the max scores using their method aren't terribly different from the baselines. Therefore, a more extensive hyperparameter sweep should have been run for the baselines: they should have been re-tuned for the average score if that is the metric the authors wish to use. Instead, the authors only used one task, RTE, to find baseline hyperparameters.”

We agree with you. It could be unfair to compare the average dev score between mixout(w_pre) and dropout since Devlin et al. (2018) selected the drop probability 0.1 to get the maximum dev score on each downstream task. We therefore explore the effect of drop probability for dropout when we finetune BERT_LARGE on MRPC, STS-B, and CoLA. All experimental setups for these experiments are the same as Section 6.3. As shown in https://ibb.co/ykrRMxY , we had the highest average dev score on MRPC/STS-B with dropout(0.1). We obtained the highest average dev score on CoLA with dropout(0.0), but we got almost the same average dev score with dropout(0.1) compared to dropout(0.0). From these results, we demonstrate that the hyperparameter for the baseline in our paper is almost optimal for the average dev score.   

References
[1] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.


Thank you for your comments!

“1a. For starters, all the theoretical motivation describes a particular way in which mixout is supposed to aid in downstream tasks for the case of convex functions, but there are no experiments that match their assumptions and which demonstrate this is the actual behavior we see. It would be nice to see results which demonstrate the theory.”

Thank you for your insightful suggestion. Since the theoretical findings of mixout depend on the strong convexity of loss functions, it is nice to demonstrate the behavior of mixout for a strongly convex loss function. We performed the least squares regression with the synthetic dataset. For randomly given a and b, we generated observation y satisfying y = ax + b + e where e is Gaussian noise. We set the model to y_hat = w1 x + w2 (That is, the model parameter w is given by (w1, w2)) and train it with mixout(target, p) where ‘target’ is the target model parameter (u, 0). By Corollary 1.1, the final solution w* should converge to the target model parameter (u, 0) rather than the true model parameter (a, b) as the mix probability p increases. As shown in https://ibb.co/TvyHwP5 , mixout(target, p) indeed regularizes learning to minimize the deviation from the target model parameter (u, 0), and the strength of regularization increases as p increases. This ensures that our theoretical finding about the proposed mixout is valid for strongly convex functions. We will include this result in the next draft and thank you for bringing this to our attention.

“1b. In few of the presented empirical experiments is it the case that the use of mixout by itself is useful.  Why does mixout have to be coupled with other regularization techniques? There is little analysis given here, either empirical or theoretical.”

Thank you for your comment. We have been working with the reason why the proposed mixout has to be coupled with other regularization techniques to improve maximum dev scores. We hypothesize that BERT_LARGE (Total 300M parameters) is over-parametrized for a small training set (RTE, MRPC, STS-B, and CoLA), and more regularization is needed to avoid overfitting on those tasks. Hence, we add an additional regularization technique to regularize our finetuning runs. It is interesting to characterize the necessary strength of regularization for BERT_LARGE, and this would be great future work.
  
“2. Why are there only 4 GLUE tasks reported? Devlin 2018 reports on all but WNLI.”

The main purpose of our paper is to improve finetuning stability via proper regularization. We thus validated our regularization techniques on those 4 GLUE tasks where finetuning results were unstable. More specifically, Devlin et al. (2018) reported that BERT_LARGE needed several random restarts to avoid the unstable finetuning results on small training sets in GLUE. Phang et al. (2018) observed that such finetuning instability was shown when the number of training examples was less than 10,000. Since RTE, MRPC, CoLA, and STS-B have less than 10,000 training instances, we selected them to validate the proposed mixout.

In Section 6.1, we performed the ablation experiment to explore the effectiveness of the proposed mixout on SST-2 which is one of the remaining GLUE tasks with sufficient training examples (greater than 10,000). As shown in Table 2,  we did not observe differences in terms of mean and maximum dev scores between the proposed mixout and Devlin et al. (2018)’s regularization technique. From this, we conclude that the dev scores obtained by the proposed mixout do not differ from those obtained by Devlin et al. (2018)’s when sufficient training instances are available. We therefore omitted finetuning BERT_LARGE on the GLUE tasks with more than 10,000 training data in Section 5. We will improve our presentation to be clearer in the next version.    

References
[1] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.

[2] Jason Phang, Thibault F´evry, and Samuel R Bowman. Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks. arXiv preprint arXiv:1811.01088, 2018.

Thank you for your comments!

“The time usage of the regularization is not discussed. It seems the method needs to maintain two copies of parameters, it would be much better if the author can provide the time usage of the experiments.”

We strongly agree with you. The time usage discussion of the proposed mixout is useful information for the readers. We re-performed the experiments in Section 6.3  to compare the time usage of dropout and that of the proposed mixout. It took about 843 seconds to finetune BERT_LARGE with the proposed mixout. On the other hand, it took about 636 seconds to finetune BERT_LARGE with dropout. As you anticipated, the proposed mixout spent 32.5% more time than dropout. However, we emphasize at least 15 finetuning runs among 20 random restarts have chance-level accuracies on RTE with dropout(p) for all p (blue in Figure 4) while only 4 finetuning runs out of 20 random restarts are unusable with mixout(w_pre, 0.8). This fact gives a good reason to finetune with the proposed mixout although it yields the additional time usage compared to dropout.    


This paper introduces a new regularization technique refered as “mixout”, motivated by dropout. Mixout stochastically mixes the parameters of two models. Experiments shows the stability of finetuning and the method greatly improve the average accuracy.

I really like the proposed idea, and the paper is easy to understand and follow, and the experiments are well designed. 

The time usage of the regularization is not discussed. It seems the method needs to maintain two copies of parameters, it would be much better if the author can provide the time usage of the experiments. 

This paper introduces a new regularization technique “mixout” for fine-tuning BERT. Mixout technique mixes the parameters of two models — the pretrained model and the dropout model. Because it keeps pretrained model parameters in consideration all the time, it effectively prevent catastrophic forgetting. Empirical results show that mixout can stabilize fine-tuning BERT on tasks with small training examples (which has been shown to be difficult).

I’d like to accept this paper based on the extensive and detailed experiments and promising results. For example, the theory has been supported by experiment findings on handwriting dataset (computer vision) while the main contribution is on natural language tasks. The authors not only conducted experiments on tasks with less examples (the main focus of this paper) but also on a task with sufficient training examples. For each model regularization configuration, 20 random starts are used to report mean and best performance. This not only makes the results more reliable but also provides deeper insights.

Minor clarification questions:

For 20 random restarts, are they the same across regularization setups, i.e., for each dot in Figure 3, is there an orange dot that has the same initialization?

What will the extreme case, mixout(w_pre, 1.0), behave? According to Figure 1, it would always use w_pre and end up not learning on a target task. If so, would it introduce a cliff in Fig 4? 


Meta-review: This paper presents mixout, a regularization method that stochastically mixes parameters of a pretrained language model and a target language model. Experiments on GLUE show that the proposed technique improves the stability and accuracy of finetuning a pretrained BERT on several downstream tasks.

The paper is well written and the proposed idea is applicable in many settings. The authors have addressed reviewers concerns' during the rebuttal period and all reviewers are now in agreement that this paper should be accepted.

I think this paper would be a good addition to ICLR and recommend to accept it.


Decision: accept-poster