Mixout: Effective Regularization to Finetune Large-scale Pretrained Language Models
Authors: Cheolhyoung Lee
Kyunghyun Cho
Wanmo Kang
Venue:  (2020)
Keywords: regularization
finetuning
dropout
dropconnect
adaptive L2-penalty
BERT
pretrained language model

Abstract: In natural language processing, it has been observed recently that generalization could be greatly improved by finetuning a large-scale language model pretrained on a large unlabeled corpus. Despite its recent success and wide adoption, finetuning a large pretrained language model on a downstream task is prone to degenerate performance when there are only a small number of training instances available. In this paper, we introduce a new regularization technique, to which we refer as “mixout”, motivated by dropout. Mixout stochastically mixes the parameters of two models. We show that our mixout technique regularizes learning to minimize the deviation from one of the two models and that the strength of regularization adapts along the optimization trajectory. We empirically evaluate the proposed mixout and its variants on finetuning a pretrained language model on downstream tasks. More specifically, we demonstrate that the stability of finetuning and the average accuracy greatly increase when we use the proposed approach to regularize finetuning of BERT on downstream tasks in GLUE.

Published as a conference paper at ICLR 2020
MIXOUT: EFFECTIVE REGULARIZATION TO FINETUNE LARGE-SCALE PRETRAINED LANGUAGE MODELS
Cheolhyoung Lee"
Kyunghyun Cho
Wanmo Kang" wazmm wwwgglaaiit..... aac. kr
lee.kaist.ackkk
kyunghyun, htt.:y/....ooy....oo edd
ABSTRACT
In natural language processing, it has been observed recently that generalization could be greatly improved by finetuning a large-scale language model pretrained on large unlabeled corpus. Despite its recent success and wide adoption, finetun- ing a large pretrained language model on a downstream task is prone to degenerate performance when there are only small number of training instances available. In this paper, we introduce a new regularization technique, to which we refer as "mixout", motivated by dropout. Mixout stochastically mixes the parameters of two models. We show that our mixout technique regularizes learning to minimize the deviation from one of the two models and that the strength of regularization adapts along the optimization trajectory. We empirically evaluate the proposed mixout and its variants on finetuning a pretrained language model on downstream tasks. More specifically, we demonstrate that the stability of finetuning and the av- erage accuracy greatly increase when we use the proposed approach to regularize finetuning of BERT on downstream tasks in GLUE
INTRODUCTION
Transfer learning has been widely used for the tasks in natural language processing (NLP) Collobert et al. [201]] Devlin et al. 2018: Yang et al.) [019 Liu et al. 2019 Phang et al. 2018). In particu- lar, Devlin et al. (2018) recently demonstrated the effectiveness of finetuning large-scale language model pretrained on large, unannotated corpus on wide range of NL.P tasks including question an- swering and language inference. They have designed two variants of models, BERTLARGE (340M parameters) and BERTBASE (110M parameters), Although BERTLA outperforms BERT BASE generally, was observed that finetuning sometimes fails when a target dataset has fewer than 10,000 training instances (Devlin al. 2018 Phang al. 2018).
When finctuning a big, pretrained language model, dropout (Srivastava et al.) 2014) has been used regularization technique to prevent co-adaptation of neurons Vaaswani al. 2017 Devlin et al. 2019). We provide a theoretical understanding of dropout and its variants, such 2018 Yang et al. Gaussian dropout (Wang & Manning 2013), variational dropout (Kingma al. 2015). and drop- connect (Wan et al. 2013), as an adaptive -=-penalty toward the origin (all zero parameters 0) and generalize dropout by considering target model parameter u (instead of the origin), which we refer as mixout(u)) We illustrate aixout(u) in Figure [ To be specific, mixout(u) replaces all outgoing parameters from a randomly selected neuron to the corresponding parameters of u. mixout(u) avoids optimization from diverging away from u through an adaptive L L'-penalty to- ward u. Unlike mixout(u), dropout encourages a move toward the origin which deviates away from u since dropout is equivalent to mixout(O).
We conduct experiments empirically validating the effectiveness of the proposed mixout(wped) where Wpre denotes pretrained model parameter. To validate our theoretical findings. we train a fully connected network on EMNIST Digits (Cohen et al., (017) and finetune on MNIST. We observe that finetuning solution of mixout/ (xre) deviates less from Wpre in the -senes than
Department of Mathematical Sciences, KAIST, Daejeon, 34141, Republic of Korea 1 New York University
I Facebook AI Research
5 CIFAR Azrieli Global Scholar

---

Published as a conference paper at ICLR 2020
(a) Vanilla network at
(b) Dropout network at
(c) mixout(u) network at
Figure I: Illustration of mixout(u). Suppose that u and to are target model parameter and current model parameter, respectively. (a): We first memorize the parameters of the vanilla network at u. (b): In the dropout network, we randomly choose an input neuron to be dropped (a dotted neuron) with probability of p. That is, all outgoing parameters from the dropped neuron are eliminated (dotted connections). (c): In the mixout(u) network, the eliminated parameters in (b) are replaced by the corresponding parameters in (a). In other words, the mixout (w) network at w is the mixture of the vanilla network at and the dropout network w with probability of p.
that of dropout. In the main experiment, we finetune BERTLARGE with mizout(w,,p (Wpre) small training sets of GLUE (Wang et al. 2018). We observe that mixout mixre reduces the number of unusable models that fail with the chance-level accuracy and increases the average development (dev) scores for all tasks. In the ablation studies, we perform the following three experiments for finetuning BERTLARGE LAGGE with mi xoxut(prro) (i) the effect of mixout (Wpp)) on a sufficient num- ber of training examples, (ii) the effect of regularization technique for an additional output layer which not pretrained, and (iii) the effect of probability of sixout(Wpre) compared From these ablation studies, we observe that three characteristics of mixout (oouttppree (Wer::: ): (i) finetuning
dropout.
with mixout( (pre)) does not harm model performance even with a sufficient number of training
examples; (ii) It is beneficial to use variant of mixout as a regularization technique for the addi- tional output layer; (iii) The proposed mixout(wpre) is helpful to the average dev score and to the finetuning stability in wider range of its hyperparameter p than dropout.
1.1 RELATED WORK
For lcage-scale pretrained language models Vaawwani et al. 2017 Devlin et al. 2018 Yang et al. 2019), dropout has been used as of several regularization techniques. The theoretical analysis for dropout as an L reggulaarre toward 0 was explored by Wan et al. 2013 where the ori- gin. They provided sharp characterization of dropout for simplified setting (generalized linear model). Mianjy & Arora (2019) gave a formal and complete characterization of dropout in deep linear networks with squared loss nuclear norm regularization toward O. However, neither Wan et al. 2013) nor Mianyy & Arora 2019 gives theoretical analysis for the extension of dropout which uses point other than
Wiese et al. (017). Kirkpatrick et al. 2017 and Schwarz et al. (2018) used I.2-penalty toward pretrained model parameter improve performance. They focused on preventing catastrophic forgetting to enable their models to learn multiple tasks sequentially. They however do not discuss nor demonstrate the effect of T,2 T,2-penalty toward the pretrained model parameter on the stability of finetuning. |Bronn et al.] (2017) introduced tuneout, which is special case of mixout. They applied various regularization techniques including dropout, tuneout, and L²-penalty toward pretrained model parameter to finetune neural machine translation. They however do not demonstrate empirical significance of tuneout compared to other regularization techniques nor theoretical justification.
2 PRELIMINARIES AND NOTATIONS
Norms and Loss Functions Unless explicitly stated, norm refers to -o-mo.. A loss function a neural network is written as L(w) = Σ C,(w) (w), where w a trainable model parameter. L, is "a per-example loss function" computed the i-th data point.
2

---

Published as a conference paper at ICLR 2020
Strong Convexity
A differentiable function f is strongly convex if there exists > 0 such that f(v) 2 f(x) + V)xx) (y - x
m ll - 1-12
(1)
for all a and y.
Weight Decay We refer as "wdecay(u,, 2)" to minimizing
((w) + 2 [| ull?
instead of the original loss function ((w) where is regularization coefficient. Usual weight decay of À is equivalent to wdocay(0, A).
Probability for Dropout and Dropeonnect Dropout (Srivastava al. 2014) is regularization technique selecting a neuron to drop with probability of p. Dropconnect (Wan et al. 2013) chooses
parameter drop with probability p. To emphasize their hyperparameter p. we write dropout and dropeonnect with a drop probability of p as "dropout(p)" and "dropconnect(p) respectively. dropout(p) a special case of dropconnect(p) if we simultaneously drop the parameters outgoing from each dropped neuron.
Inverted Dropout and Dropconnect In the of dropout(p)) neuron is retained with proba- bility of 1 1-p during training. If we denote the weight parameter that neuron as w during training, then we use (1 p)w for that weight parameter at time (Srivastava al. 2014). This ensures that the expected output of a neuron is the same as the actual output at test time. In this paper, dropout(p) refers to inverted (popout(p) which uses w p) instead of ur during training. By doing we do not need to compute the output separately at test time. Similarly, dropeonncct(p) refers to inverted dropconnect(p).
ANALYSIS OF DROPOUT AND ITS GENERALIZATION
We start theoretical analysis by investigating dropconnect which is general form of dropout and then apply the result derived from dropconnect to dropout. The iterative SGD equation for dropconnect (p) with learning rate of
(++1) = (t) (1B(*) VL (EB()) B Bl)r(()
1= 0, 1, 2,
(2)
( and ' are mutually independent Bernoulli(1 p) random variables with drop probability of p for all i and We regard equation 2 as finding solution to the minimization problem below:
where (()) diag (1)) Bidh
min E. ((BB))
w
where diag(B; B2, ..., Ba) and B''s are mutually independent Bernoulli(1 - -p) random variables with a drop probability of for all
Gaussian dropout (Wang Manning 2013) and variational dropout (Kingma et al. 2015) use other random masks to improve dropout rather than Bernoulli random masks. To explain these variants of dropout as well, we set a random mask matrix M = diag M2. ... Ma) to satisfy EM, = and Var(M,) = ² for all i. Now define random mixture function with respect to from and M as
(ww; u, M) - - M)u Mw p)u),
(4)
and minimization problem with "mixconnect(u,, μ, 02)"
min E u. M)).
We view dropconnect(p) equation 3 special case of equation where 0 and M B.
We investigate how mixconnect(u, 14, d²) differs from the vanilla minimization problem
min EL(w).
(6)
w
the loss function C is strongly convex, we can derive a lower bound of EC (f(w; (w; u, M)) as in
Theorem [
3

---

Published as a conference paper at ICLR 2020
Theorem 1. Assume that the loss function L is strongly convex. Suppose that random mixture function with respect to w from u and M is given by (w: u, M. in equation f where M is (Mag(M,, M2. Ma) satisfying EM, μ and Var(M, o² for all i. Then, there exists
m > such that
mo² 11 u11?
( u, M)) 2 [(w) +
2p²²
for all w (Proof in Supplement
Theorem shows that minimizing the lh.ss of equation 7 minimizes the r.h.s. of equation 7 when the r.h.s. a sharp lower limit of the s. The strong convexity of &. means that is bounded from below by quadratic function, and the inequality of equation comes from the strong convexity. Hence, the equality holds if is quadratic, and mixconnect(t.. μ,, o²) is an I,2 r.g-rauulrrizer with regularization coefficient of mo² //2.
3.1 MIXCONNECT TO MIXOUT
We propose mixout special case of mixconnect, which is motivated by the relationship between dropout and dropconnect. We assume that
(N1)
(N1) (Na)
(N2)
(Nx)
(N&)
wd, wi
Wdd
20
dx
(N.) is the jth parameter outgoing from the neuron N.. We set the corresponding M to where wj
M = diag (M))
M(N)), (Ma)),
(
(
M(N)) (8)
where (MM = μ and Var(A((N)) o² for all i. In this paper. we set ( to Bernoulli1 -1 for all and mixout(u) hereafter refers to this correlated version of mixconnect with Bernoulli random masks. We write it as "mixout(u, p)" when we emphasize the mix probability p.
Corollary 1.1. Assume that the loss function C is strongly convex. We denote the random mix- ture function of mixout(u, u. p). which is equivalent to that of mixconnectt
- P, p²), as Φ (w: u, M) where M is defined in equation 8 Then, there exists 0 such that
EL(((w: u. B)) 2 L(w) + 2(1 w ull?
mp
(9)
for all w.
Corollary 1.1 is straightforward result from Theorem As the mix probability p in equation
increases to I, the L'-regularization coefficient of mp/ (I (I--[] p) increases to infinity. It means that of mixout(u,, p) can adjust the strength of L L'-penalty toward in optimization. mixout (u) differs from wdeeca( (u) since the regularization coefficient of mimxou((uu (u) depends on m determined by the current model parameter
mixout(u, p) indeed regularizes learning to minimize the deviation from u. We validate this by performing least squares regression in Supplement(D)
2014) applied dropout to fully connected layers only. We generalize Theorem [ to the case in which mixout is only applied specific layers, and it can be done by constructing M in particular way. We demonstrate this approach in Supplement B and show that mixout for specific layers adaptively 2.2-penalizes their parameters.
We often apply dropout to specific layers. For instance, [Sioonyan & Zisserman
3.2
MIXOUT FOR PRETRAINED MODELS
Hoffer et al. (2017) have empirically shown that
(10) where wi is model parameter after the t-th SGD step. When training from scratch, usually sample an initial model parameter wp from a normal/uniform distribution with mean 0 and small variance. Since wo is close to the origin, wi is away from the origin only with large by equa- tion 10 When finetuning, we initialize our model parameter from a pretrained model parameter Wpre- Since we usually obtain Wrre by training from scratch on large pretraining dataset, Wpre often far away from the origin. By Corollary 1.1 dropout L -e-eallizess the model parameter for deviating away from the origin rather than Wpre- To explicitly prevent the deviation from w pres we
l|w; woll ~ log
instead propose use mix mixout( (oope
4

---

Published as a conference paper at ICLR 2020
4 VERIFICATION OF THEORETICAL RESULTS FOR MIXOUT ON MNIST
Wiese et al. 2017) have highlighted that wdecay(Wpor is an effective regularization technique to avoid catastrophic forgetting during finetuning. Because mi sixout(upp) keeps the finetuned model to stay in the vicinity of the pretrained model similarly to wdecay (eree)))))))..... we suspect that the pro- posed mixout(Wprd) has a similar effect of alleviating the issue of catastrophic forgetting. To empir- ically verify this claim, we pretrain 784-300- 1--3000000-111 10 fully-connected network on EMNIST Digits (Cohen et al.] (2017). and finetune it on MNIST. For more detailed description of the model architec- ture and datasets, Supplement C.T
In the pretraining stage, we run five random experiments with a batch size of 32 for (1. 2, ................................ 200 training epochs. We use Adam (Kingma & Ba 2014) with a learning rate of 10 A1 0.9, 88 = 0.999, wdecay((0, 0.01), learning rate warm-up over the first 10% steps of the total steps. and linear decay of the learning rate after the warm-up. We use (O.pout(O.1) 1) for all layers except the input and output layers. We select wre whose validation accuracy on EMNIST Digits is best (0.992) in all experiments.
For finetuning, most of the model hyperparameters are kept same as in pretraining, with the excep- tion of the learning rate, number of training epochs, and regularization techniques. We train with a learning rate of 5 x 10-6 for 5 training epochs. We replace dropout(p) with nixout(Wpre: p). We do not use any other regularization technique such as wdecay (0) and wdccay( pree). We mon- itor ||wnn "rrell validation accuracy on MNIST, and validation accuracy on EMNIST Digits to compare mixout(wpre p) (p) across 10 random restarts
2.5
Validation accuracy
Validation accurapy
0.70
2.0
0.96
1.5
Drapout Mixout
0.65
0.94
0.60
1.0
Dropout
Dropout
0.55
0.5
0.92
Mixout
Mixout
0.0
0.4
0.0
0.2
0.4 Probability
0.0
0.6
0.8
Proba bility
Probability
(a)
Wp
Wpre ||
(b) MNIST
(c) EMNIST Digits
We present W| Wprell validation accuracy on MNIST (target task), and validation accuracy on EMNIST Digits (source task), as the function of the probability p where wn and wpee are the model parameter after finetuning and the pretrained model parameter, respectively, We report mean (curve) + std. (shaded area) across 10 random restarts, (a): mixout mixout(upre p) Le-penalizes the deviation from Wpre and this penalty becomes strong as p increases. However, with dropout Wn becomes away from Wpre as increases. (b): After finetuning on MNIST, both mi miuttttttttte (pre: p) and dropout(p) result in high validation accuracy on MNIST for p E {0.1, 0.2, 0.3). (c): Valida- tion accuracy of dropout(p) on EMNIST Digits drops more than that of mixout (pee p) for all p. mixout pre: p) minimizes the deviation from wree and memorizes the source task better than dropout(p) for all p.
Figure 2:
As shown in Figure 2 (a), after finetuning with mixout (2 p). the deviation from Wpee is min- imized in the Le-sense. This result verifies Corollary 1.1 We demonstrate that the validation accuracy of mixout(wpre. p) has greater robustness to the choice of p than that of dropout (p). In Figure 2 (b), both dropout(p) and mixout(Wpps p) result in high validation accuracy on the target task (MNIST) for p E {0.1, 0.2, 0.3), although mixout (roouut(wpgg (Wpre: p) is much more robust with respect to the choice of the mix probability p. In Figure 2
(c), the validation accuracy of mixout(wpre) p) on the source task (EMNIST Digits) drops from the validation accuracy of the model at Wpre (0.992) to approximately 0.723 regardless of On the other hand, the validation accuracy of (ropout(p) on the source task respectively drops by 0.041, 0.074 and 0.105 which are more than those of sixout(Wper p) for E (0.1, 0.2, 0.3}.
"n is model parameter after finetuning. Using the same pretrained model parameter Wpre but perform different finetuning data shuffling.
5

---

Published as a conference paper at ICLR 2020
5 FINETUNING A PRETRAINED LANGUAGE MODEL WITH MIXOUT
In order to experimentally validate the effectiveness of mixout, we finetune LERTLARGE LARGE on sub- set of GLUE (Wang al. 2018) tasks (RTE, MRPC, CoLA, and STS-B) with cixout(Wpre). We choose them because Phang et al. (2018) have observed that it was unstable to finetune BERTLARE LARGE on these four tasks. We use the publicly available pretrained model released by Devlin et al. (0188). ported into PyTorch by HuggingFaced M We use the learning setup and hyperparameters recom- mended by Devlin et al. (2018). We use Adam with learning rate of x 10 81 0.9, 82 = 0.999, learning rate warmup over the first 10% steps of the total steps, and linear decay of the leamnnng rate after the warmup finishes. We train with batch size of 32 for 3 training epochs. Since the pretrained LERTEARGE is the sentence encoder, we have to create an additional output layer, which is not pretrained. We initialize each parameter of it with N(0, 0.02²). We describe our experimental setup further in Supplement
The original regularization strategy used in Devlin et al. (2018) for finetuning LARGE is using both dropout(O.1) and wdecay(0. 0.01) for all layers except layer normalization and in- termediate layers activated by GELU (Hendrycks & Gimpel 2016). We however cannot use mi mixout(W (prpe) nor wdecay(uppe) for the additional output layer which was not pretrained and therefore does not have pre We do not use any regularization for the additional output layer when finetuning BERTLARGE with mixout(wpre and wdecay (WDpe) For the other layers, we replace dropout and wdecaa((,, 0.01) with mixout mixout(Wpre) and vrdecay(Wpre) respectively.
Phang et al. (2018) have reported that large pretrained models (e.g., BERTLARGE) are prone to degenerate performance when finetuned on a task with a small number of training examples, and that multiple random restarte are required to obtain usable model better than random prediction. To compare finetuning stability of the regularization techniques, we need to demonstrate the distribution of model performance. We therefore train BERT BERTLARGE LaRGE with each regularization strategy on each task with 20 random restarts. We validate each random restart on the dev set to observe the behaviour of the proposed mixout and finally evaluate on the test set for generalization. We present the test score of our proposed regularization strategy on each task in Supplement
We finetune SERTLARGE with mixout (Wppe> {0.7, 0.8, 0.9)) on RTE, MRPC, CoLA, and STS- B. For the baselines, we finetune BERT with both (00pout(0.1) and wdecay(0., 0.01) as well as with wdecay(w pre (0.01, 0.04, 0.07, 0.10)).. These choices are made based on the exper- iments in Section 6.3 and Supplement F In Section 6.3 we observe that finetuning BERT LARGE with mixout (ure, p) on RTE is significantly more stable with p E {0.7, 0.8, 0.9} while fine- tuning with (ropout(p) becomes unstable p increases. In Supplement F we demonstrate that dropout(O.1) is almost optimal for all the tasks in terms of mean dev score although Devlin et al. (018) selected it to improve the maximum dev score.
In Figure we plot the distributions of the dev scores from 20 random restarts when finetuning BERTLARGE with various regularization strategies each task. For conciseness, we only show four regularization strategics; Devlin et al. (2018)ss: ss: both dropout(0.1) and wdecay(O, 0.01), Wiese et al.] (2017)'s: wdecay (Weee 0.01). ) ours: mixout(Wpre. 0.7), and ours+ Wiese et al.
2017)'s s: both mixout(wp.. 0.7) and wdecay (Wpre: 0.01). As shown in Figure 3 (a-c), we ob- serve many finetuning runs that fail with the chance-level accuracy when we finetune BERTIARGE
with both dropout(O.. (0.1) and wdecay (0, 0.01) on RTE, MRPC, and CoLA. We also have a bunch of degenerate model configurations when we use wdleaa
0.01) without mixout (Wpre- 0.7).
Unlike existing regularization strategies, when we use mixout pree 0.7) as regularization tech- nique with or without wdecay (Wrre: 0.01) for finetuning LARGE, the number of degenerate model configurations that fail with chance-level accuracy significantly decreases. For example, in Figure 3 (c), we have only one degenerate model configuration when finetuning BERTLARGE larGe with mixout w pre. 0.7) on CoLA while we observe respectively seven and six degenerate models with Devlin et al. 2018)'s 's and Wiese et al. (2017)'s 's regularization strategies.
https : / s3 amazonaws com models s ingginngfcc. - co. / bert
bert unngeeuncased pooddd-yttrhhhhhoo pyttophhtotohhollllllleeop (.bi
*Using the same pretrained model parameter Wpre but each random restart differs from the others by shuf- fling target data and initializing the additional output layer differently.

---

Published as a conference paper at ICLR 2020
92.5
70
90.0
Dev score
Dev score
87.5
60
85.0
*....****
82.5
80.0
Devlin
Wiese
Our
Our+W
Devlin
Wiese
Our
Our+W
(a) RTE (Accuracy)
(b) MRPC (F1 accuracy)
100
60
80
Dev score
Dev score
40
60
40
20
20
0
Devlin
Wiese
Our
Our+W
Devlin Wiese
Our
Our+W
(c) CoLA (Mattew's 's correlation)
(d) STS-B (Spearman correlation)
Figure 3: Distribution of dev scores on each task from 20 random restarts when finetuning BERT LARGE with Devlin al. (2018)'s: both (0.1) and wdecay (0. 0.01). Wiese et 2017 s: wdecay (ppre: 0.01), ours: mixout(wpr. 0.7). and ours+Wiesee et al. (017))ss ss: both mixout(un (urpe, 0.7) and wdecay (upre 0.01). We write them as Devlin (blue). Wiese (orange). Our (green), and Our+W (red), respectively. We use the same set of 20 random initializations across all the regularization setups. Error intervals show std. For all the tasks, the number of finetuning runs that fail with the chance-level accuracy is significantly reduced when we use our regularization mixout Wpre. 0.7) regardless of using wdecay (rr,, 0.01).
In Figure (a), we further improve the stability of finetuning BERTHARGE by using both mixout( (Wre 0.7) and wdecay 0.01). Figure 3 (d) shows respectively and one degener- ate model configurations with Devlin et al. (2018) 's and Wiese et al. (2017)'s, 's but we do not have any degenerate resulting model with ours and ours+ Wiese et al. (217)'s In Figure c). we observe that the number of degenerate model configurations increases when we use wdecay (Wpre: 0.01) additionally to mixout(wpes. 0.7). In short, applying our proposed mixout significantly stabi- lizes the finetuning results of BERTLARGE small training sets regardless of whether we use wwecaa(wpre.. 0.01).
In Table we report the average and the best dev scores across 20 random restarts for each task with various regularization strategies. The average dev scores with mixout(Wpee) (0.7, 0.8, 0.9}) increase for all the tasks. For instance, the mean dev score of finetuning with mixout(uue (Wpre: 0.8) on CoLA is 57.9 which is 49.2% increase over 38.8 obtained by finetuning with both dropout(p) and wdecay (0, 0.01). We observe that using wdecay(wpp.) (0.01, 0.04, 0.07, 0.10)) also improves the average dev scores for most tasks compared to using both dropout(p) and wdecay(0, 0.01). We however observe that finetuning with mixout(w, (Wper {0.7, 0.8, 0.9)) outperforms that with wdecay( {0.01, 0.04, 0.07, 0.103)) on average. This confirms that mixout (Wpre) has a dif- ferent effect for finetuning BERTLARGE compared to wdecay (Wppe) since sixout(wper is an adaptive T.Z-regularizer along the optimization trajectory.

---

Published as a conference paper at ICLR 2020
Since finetuning large pretrained language model such as BERT LARGE small training set fre- quently fails, the final model performance has often been reported as the maximum dev score (Devlin et al. 2018 Phang al. 2018) among a few random restarts. We thus report the best dev score for each setting in Table According to the best dev scores as well, mixout(Wpro (0.7, 0.8, 0.9)) improves performance for all the tasks compared to using both (popout(p) and wdecay(0, 0.01). For instance, using mixout (rr,, 0.9) improves the maximum dev score by 0.9 compared to using both (popout(p) and wdecay(0, 0.01) on MRPC. Unlike the average dev scores, the best dev scores achieved by using wdecay(w (Wpee. {0.01, 0.04, 0.07, 0.10)) are better than those achieved by using mixout(w (Wrpe: f0.7, 0.8, 0.9)) except RTE on which was better to use (u {0.7, 0.8, 0.9}) than wdecay(upre, (Wpre, {0.01, 0.04, 0.07, 0.10)).
Table 1: Mean (max) dev scores across 20 random restarts when finetuning BERT LARGE with various regularization strategies on each task. We show the following baseline results on the first and second cells: Devlin et al. (2018)ss regularization strategy (both (p) and wdecay(0, 0.01)) and Wiese et al. 2017) 's regularization strategy (wdecay (UPpee {0.01, 0.04, 0.07, 0.10))). In the {0.7, 0.8, 0.9)). The results with both mixout pres {0.7, 0.8, 0.9}.) and wdecay prep 0.01) are also presented in the fourth cell. Bold marks the best of each statistics within each column. The mean dev scores greatly increase for all the tasks when we use mixout(w; {0.7, 0.8, 0.9)).
third cell, we demonstrate finetuning results with only mixout(w
TECHNIQUE 1
TECHNIQUE2
RTE
MRPC
CoLA
STS-B
(0.)
wdecay (0, 0.01)
56.5 (73.6)
83.4 (90.4)
38.8 (63.3)
82.4 (90.3)
wdecay pre, 0.01)
(71.5)
86.2
(65.6)
85.4 (90.5)
wdecay(w (Wper 0.04)
51.5 (70.8)
851.(91.5)
35.4 (64.7)
80.7 (90.6)
wdecay pre. 0.07)
57.0 (70.4)
85.8
(63.9)
89.6 (90.3)
wdecay W pre. 0.10)
54.6 (71.1)
(91.)
45.6663.8)
84.3 (90.1)
mixout w pre. 0.7)
61.6 (74.0)
87.1
57.4 (62.1)
89.6 (90.3)
mixout prey 0.8)
(7.00)
89.0 (90.7)
(63.8)
89.4 (90.3)
mixout W pre: 0.9)
88.2 (7..))
(63.4)
89.4 (90.0)
mixout pre, 0.7)
wdecay prey 0.01)
65.3 (74.4)
87.8
(64.0)
89.6 (90.6)
mixout pre. 0.8)
wdecay Upre 0.01)
62.8 (74.0)
86.3 (90.9)
58.3 (65.1)
89.7 (90.3)
mixout pre. 0.9)
wdecay W pre. 0.01)
65.0 (75.5)
88.6 (91.3)
(65.1)
89.5 (90.0)
We investigate the effect of combining both mixout(wpee) and wdocaa(Wpre) to see whether they are complementary. We finetune BERTLARGE with both mixout(Wpees {0.7, 0.8, 0.9)) and wdecay (Wpre, 0.01). This leads not only to the improvement in the average dev scores but also in the best dev scores compared to using wdecay (Wpre: {0.01, 0.04, 0.07, 0.103) and us- ing both dropout(p) and wdecay(O. 0.01). The experiments in this section confirm that using mixout (wpre) as one of several regularization techniques prevents finetuning instability and yields gains in dev scores.
6 ABLATION STUDY
In this section, we perform ablation experiments to better understand mixout mixout(wpre) Unless explic- itly stated, all experimental setups are the same as in Section S
6.1 MIXOUT WITH A SUFFICIENT NUMBER OF TRAINING EXAMPLES
We showed the effectiveness of the proposed mixout finetuning with only a few training examples in Section 5 In this section, we investigate the effectiveness of the proposed mixout in the case of larger finetuning set. Since it has been stable to finetune BERTLARGE on sufficient number of training examples (Devlin et al.) 2018 Phang et al. 2018), we expect to see the change in the behaviour of mixout(Wpre) when we use it to finetune BERT L.AGE on larger training set.
We train BERT LARGE by using both mixout(Wppr) 0.7) and wdeeay (Wppee 0.01) with 20 random restarts on 2 We also train BERTLARGE by using both (p) and wd 0.01) with 20 random restarts SST-2 as the baseline. In Table 2 we report the mean and maximum of
For the description of SST-2, see Supplement C.2
8

---

Published as a conference paper at ICLR 2020
SST-2 dev scores across 20 random restarts with each regularization strategy. We observe that there is little difference between their mean and maximum dev scores on a larger training set, although using both mixout( ( pres 0.7) and wdecay(Wpp. pre: 0.01) outperformed using both dropout(p) and wdecay(0, 0.01) on the small training sets in Section 5
Mean (max) SST-2 dev scores across 20 random restarts when finetuning LERTLARGE with each regularization strategy. Bold marks the best of each statistics within each column. For a large training set, both mean and maximum dev scores are similar to each other.
Table 2:
TECHNIQUE1 1
TECHNIQUE 2
SST-2
(0.) mixout(w (wrre> 0.7)
wdecay (O, 0.01) eddecayWpper 0.01)
93.4 (94.0)
93.5 (94.3)
6.2 EFFECT OF A REGULARIZATION TECHNIQUE FOR AN ADDITIONAL OUTPUT LAYER
In this section, we explore the effect of a regularization technique for an additional output layer. There are two regularization techniques available for the additional output layer: dropout(p) and mixout(wy) Wo. p) where wy is its randomly initialized parameter. Either of these strategies differs from the earlier experiments in Section 5 where we did not put any regularization for the additional output layer.
Table 3: We present mean (max) dev scores across 20 random restarts with various regularization techniques for the additional output layers (ADDITIONAL) when finetuning BERTLARGE on each task. For all cases, we apply mixout mixout(Wpre. 0.7) to the pretrained layers (PRETRAINED). The first row corresponds to the setup in Section 5 In the second row, we apply mixout Wo, 0.7) to the additional output layer where wy is its randomly initialized parameter. The third row shows the results obtained by applying dropout(0.7) )) to the additional output layer. In the fourth row, demonstrate the best of each result from all the regularization strategies shown in Table I Bold marks the best of each statistics within each column. We obtain additional gains in dev scores by varying the regularization technique for the additional output layer.
PRETRAINED ADDITIONAL
RTE
MRPC
CoLA
STS-B
mixout( Wpre: 0.7)
(74.0)
87.1 (91.7)
57.4 (62.7)
89.6 (90.3)
mixout( pre: 0.7)
mixout(wo, 0.7)
66.5 (75.5)
88.1 (92.4)
58.7 (65.6)
89.7 (90.6)
mixout( pre: 0.7)
dropout(O..
57.2 (70.8)
85.9 (92.5)
48.9 (64.3)
89.2 (89.8)
The best of each result from Table
65.3 (75.5)
89.0 (91.8)
(65.6)
89.7 (90.6)
We report the average and best dev scores across 20 random restarts when finetuning BERTLARGE with miixuu((wpre: 0.7) while varying the regularization technique for the additional output layer in Table 3 We observe that using mixout(wo, 0.7) for the additional output layer improves both the average and best dev score on RTE, CoLA, and STS-B. In the case of MRPC. we have the highest best-dev score by using d00pout(0.7) for the additional output layer while the highest mean dev score is obtained by using mi mixout( Wo. 0.7) for it. In Section 3.2 we discussed how mix mixout( (w)) does not differ from dropout when the layer randomly initialized, since we sample wo from w whose mean and variance are O and small, respectively. Although the additional output layer is randomly initialized, we observe the significant difference between dropout and mixout( (wo) in this layer. We conjecture that l/w -0) Oll is not sufficiently small because El| w Ol| is proportional to the dimensionality of the layer (2,048). We therefore expect mixout(w,, to behave differently from dropout even for the case of training from scratch.
In the last row of Table 3 we present the best of the corresponding result from Table I We have the highest mean and best dev scores when we respectively use mixout( (Wpre: 0.7) and mixout( Wo, 0.7) for the pretrained layers and the additional output layer on RTE, CoLA, and STS- B. The highest mean dev score on MRPC is obtained by using mixout(wpre. 0.8) for the pretrained layers which is one of the results in Table [ We have the highest best dev score on MRPC when use mi mixout(w
0.7) and dropout(0 for the pretrained layers and the additional output
IIn this experiment, we use neither (d) nor (

---

Published as a conference paper at ICLR 2020
layer. respectively. The experiments in this section reveal that using mixout(wp) for randomly initialized layer of a pretrained model is one of the regularization schemes to improve the average dev score and the best dev score.
6.3 EFFECT OF MIX PROBABILITY FOR MIXOUT AND DROPOUT
We explore the effect of the hyperparameter p when finetuning BERT LARGE with mixout(# p) and dpopout(p). We train BERTLA LARGE with mixout(Wpre {0.0, 0.1, ..., 0.9)) on RTE with 20 random restarts. also train BERTLARGE after replacing exxoutwpraa p) by dropout with 20 random restarts. We do not any regularization technique for the additional output layer. Because we use neither wdecay(O) nor wdecay (Wpee) in this section, dropout(O.0) and mi mixout wpee 0.0) equivalent to finetuning without regularization.
Mixout
70
Dropout
65
60
50
45
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0.0 0.1 Probability
0.2
0.3 0.4 0.5 0.6
0.7
0.8
0.9
Figure Distribution of RTE dev scores (Accuracy) from 20 random restarts when finetun- ing LERTLARGE with dropout(p) (orange) or mixout (xxuutwwpre. p) (blue). Error intervals show mtd We do not use (decay(O) nor wdecay pre)). In the case of mixout (W p). the number of usable models after finetuning with mixout(w, {0.7, 0.8, 0.9}) is significantly more than the number usable models after finetuning with dropout(p) for all p.
It is not helpful to vary p for dropout(p) while mixout(Wpre. p) helps significantly in wide range of Figure 4 shows distributions of RTE dev scores across 20 random restarts when finctuning BERTLARGE with dropout(p) and mixout(Wpre. p) for p {0.0, 0.1,
0.9]. The mean dev score of finetuning BERTLARGE with mixout(Wp.. pre p) increases as p increases. On the other hand, the mean dev score of finetuning BERTLARGE with (popout(p) decreases as p increases. is less than 0.4, finetuning with mixout(uu,p..m pre: p) does not improve the finetuning results of using dropout (( 0.1, 0.23). We however observe that mixout(Wp.. {0.7, 0.8, 0.9}) yields better average dev scores than (ropoul(p) for all and significantly reduces the number of finetuning runs that fail with the chance-level accuracy.
We notice that the proposed mixout spends more time than dropout from the experiments in this section. takes longer to finetune model with the proposed mixout than with the original dropout, although this increase is not significant especially considering the waste of time from failed finetun- ing runs using dropout. In Supplement El we describe more in detail the difference between mixout and dropout in terms of wall-clock time.
CONCLUSION
The special case of our approach, mixrut(Wpre), is one of several regularization techniques mod- ifying a finetuning procedure to prevent catastrophic forgetting. Unlike wdecay(w,,) (Wpre) proposed earlier by Wiese et al. (2017), mixout (Wrre) is an adaptive r.-rrgularizer toward Wpre in the sense that its regularization coefficient adapts along the optimization path. Due to this difference, the pro- posed mixout improves the stability of finetuning a big, pretrained language model even with only few training examples of target task. Furthermore, approach improves finetuning results in terms of the average accuracy and the best accuracy over multiple runs. We emphasize that our approach can be used with any pretrained language models such as RoBERTa (Liu et al.) (2119) and XLNet (Yang et al. 2019), since mixout does not depend on model architectures, and leave it future work.
experiments have revealed the proposed
10

---

Published as a conference paper at ICLR 2020
ACKNOWLEDGMENTS
The first and third authors' work was supported by the National Research Foundation of Korea (NRF) grants funded by the Korea government (MOE, MSIT) NRF-2017RIA2240111 2019R TAAAAAAIOO A022142),24) The second author thanks support by AdeptMind, eBay, TenCent, NVIDIA and CIFAR and was partly supported by Samsung Electronics (Improving Deep Learning using Latent Structure).
NRF-
REFERENCES
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv: 1607.0660 2016.
Antonio Valerio Miceli Barone, Barry Haddow, Ulrich Germann, and Rico Sennrich. Regularization techniques for fine-tuning in neural machine translation. arXiv preprint arXiv:1707.09920, 2017.
Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. Semeval-2017 task
I: Semantic textual momnllarty-mlltlinngee and cross- tross-lingual focused evaluation. arXiv preprint arXiv:1708.00055 2017.
Gregory Cohen, Saeed Afshar, Jonathan Tapson, and André van Schaik. Emnist: an extension of mnist to handwritten letters. arXiv preprint 17X1v:170..05373, 2017.
Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. Natural language processing (almost) from scratch. Journal of machine learning research, 12(Aug):2493-255) 2537,,2557,25 20
Ido Dagan, Oren Glickman, and Bernardo Magnini. The PASCAL recognising textual entailment challenge. In Machine learning challenges. evaluating predictive uncertainty, visual object clas- sification, and recognising tectual entailment, pp. 177-190. Springer. 2006.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint
2018.
William B Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the International Workshop on Paraphrasing, 2005.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint
arXiv:1606.08411 2016.
Elad Hoffer, Itay Hubara, and Daniel Soudry. Train longer, generalize better: closing the generaliza- tion gap in large batch training of neural networks. In Advances in Neural Information Processing Systems, pp. 1731-1741, 2017.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Durk P Kingma, Tim Salimans, and Max Welling. Variational dropout and the local reparameteri- zation trick. In Advances in Neural Information Processing Systems, pp. 2575-2583, 2015.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Barkkiskaa al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521 -326,, 2017.
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 19t..1907.19999 2019.
Poorya Mianjy and Raman Arora. On dropout and nuclear norm regularization. In Kamalika Chaud- huri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Ma- chine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 4575 4584, Long Beach, California, USA, 09- 15 Jun 2019. PMLR. URL http : : proceedings mir preas/v97/miant 9a. html
11

---

Published as a conference paper at ICLR 2020
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference on machine learning (ICML-10), pp. 807-814, 2010.
Jason Phang, Thibault Févry, and Samuel R Bowman. Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks. arXiv preprint arXiv: 2018.
Jonathan Schwarz, Jelena Luketina, Wojeiech M Czarnecki, Agnieszka Strabsk-Barwwing Yee Whye Teh, Razvan Pascanu, and Raia Hadsell. Progress & compress: A scalable frame- work for continual learning, arXiv preprint 1xXiv:8805006300 2018.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1556, 2014.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of EMNLP, pp. 1631-1642, 2013.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, llya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1):1929- 1958, 2014.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Li Wan, Matthew Zeiler, Sixin Zhang, Yann Le Cun, and Rob Fergus. Regularization of neural networks using dropeonnect. In International conference on machine learning, pp. 1058-1066, 2013.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint 1XX4::880.004455 2018.
Sida Wang and Christopher Manning. Fast dropout training. In international conference on machine learning, pp. 118-126, 2013.
Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. Neural network acceptability judgments. arXiv preprint /805. 12471, 2018.
Georg Wiese, Dirk Weissenborn, and Mariana Neves. Neural domain adaptation for biomedical question answering. arXiv preprint arXiv:1706.03611 2017.
Zhilin Yang, Zihang Dai, Yiming Yang. Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint 18829,190..08237, 2019.
12

---

Published as a conference paper at ICLR 2020
SUPPLEMENTARY MATERIAL
A PROOFS FOR THEOREM
Theorem [ Assume that the loss function L is strongly convex. Suppose that a random mixture function with respect from and M is given by
Φ(w; M) pe ((T Myu + M - (1 p)))),
where M diag(M1, M2. there exists m >> 0 such that
Ma) satissfyng EM; μ and Var(M)) o² for all Then,
mo²
(L( u, M)) 2 f(w) + 242 1/00 u11?
(1I)
for all w.
Proof. Since & is strongly convex, there exist >> 0 such that
(L(w(w; u, M)) IEC w + (www; u, MM w)
> L(w) vL(w) (itw)][[[e [[ u, M) w] + m E o(w; M) - -wl1?
2
(12)
for all by equation
Recall that EE;; = μ and Var(M;) x² for all i. Then, we have
E[w(w;; u, M) w]
(13)
and
1
B|| [(w; M) w|/ E
w
uM m1))
[m
u4)
IM x1)
4
w
u11?
(14)
By using equation 13 and equation 14
we rewrite equation 12 as
EL(((w: u, M)) 2 [(w) + mo²
||w
u112.
24²
B APPLYING TO SPECIFIC LAYERS
We often apply dropout specific layers. For instance, (Sionyyan & Zisserman dropout to fully connected layers only. We generalize Theorem to the case in which mixconnect is only applied specific layers, and it can be done by constructing M in a particular way. To better characterize mixconnect applied to specific layers, we define the index set I as I = {a M, = 1}. Furthermore, we use and ū denote (ux) and (uz).g respectively. Then, we generalize equation 7 to
(0044) applied
mo² 1 u/|2
BL(((w; M)) 2 L(w) +
(15)
2n
From equation 15 applying mix mixconnect(uu u, μ, a²) is to adaptive wdecay () on the weight parameter of the specific layers w. Similarly, we can regard applying mixout(u, (u, p) to specific layers adaptive wdecay(ū)..
13

---

Published as a conference paper at ICLR 2020
C EXPERIMENTAL DETAILS
C.I FROM EMNIST DIGITS TO MNIST
Model Architecture The model architecture in Section 4 is a 984-300-100-11 fully connected network with a softmax output layer. For each hidden layer, we add layer normalization (Ba al.) 2016) right after the ReLU (Nair & Hinton 2010) nonlinearity. We initialize each parameter with NO, 0.022) and each bias with
Regularization In the pretraining stage, we use dropout(0.1) and wdecay (0, 0.01). We apply (ropout(0.1) to all hidden layers. That is, we do not drop neurons the input and output layers. wdccay(0, (0, 0.01) does not penalize the parameters for bias and layer normalization. When finetune our model on MNIST, we replace (ropout(p) with mixout(Wpre p). We use neither wdecay (0) nor (alecayypre)) for finetuning.
Dataset For pretraining, we train our model on EMNIST Digits. This dataset has 280,000 char- acters into 10 balanced classes. These characters are compatible with MNIST characters. EMNIST Digits provides 240,000 characters for training and 40,000 characters for test. We use 240,000 char- acters provide for training and split these into the training set (216,000 characters) and validation set (24,000 characters). For finetuning, we train our model on MNIST. This has 70,000 characters into 10 balance classes. MNIST provide 60,000 characters for training and 10,000 characters for test. We use 60,000 characters given for training and split these into the training set (54,000 characters) and validation set (6,000 characters).
Data Preprocessing We only normalization after scaling pixel values into [o, 1]. We do use any data augmentation.
C.2 FINETUNING BERT ON PARTIAL GLUE TASKS
Model Architecture Because the model architecture of LARGE is identical the original (Devlin et al., 2018). we omit its exhaustive description. Briefly, BERT has 24 layers, 1024 hidden size, and 16 self-attention heads (total 340M parameters). We use the publicly available pre- trained model released by Devlin et al. (2018), ported into PyToorch by HuggingFace We initialize each weight parameter and bias for an additional output layer with (0,, 0.02?) and 0, respectively.
Regularization In the finetuning stage, Devlin et al. (2018) used wdecay(0. 0.01) for all param- eters except bias and layer normalization. They apply dropout (0.1) to all layers except each hidden layer activated by GELU (Hendrycks & Gimpel 2016) and layer normalization. We substitute wdecay (WPpe) and mixout (Wpre) for w0 0.01) and dropout respectively..
Dataset We use subset of GLUE (Wang et al. is as the following:
2018 tasks. The brief description for each dataset
RTE (2,500 training examples): Binary entailment task (Dagan et al. 2006)
MRPC (3,700 training examples): Semantic similarity (Dolan & Brocketi 2005
CoLA (8,500 training examples): Acceptability classification (Warstadt al. 2018
STS-B (7,000 training examples): Semantic textual similarity (Cer et al. 2017)
SST-2 (67,000 training examples): Binary sentiment classification (Socher et al. 2013
In this paper, we reported F1 accuracy scores for MRPC, Mattew's correlation scores for CoLA, Spearman correlation scores for STS-B, and accuracy scores for the other tasks.
Data Preprocessing Face 8
We use the publicly available implementation of BERT LARGE by Hugging-
https : s3 amazonaws com models huggingface co / bert
bert www.sed
14

---

Published as a conference paper at ICLR 2020
C.3 TEST RESULTS ON GLUE TASKS
We expect that using mixout stabilizes finetuning results of BERTTARGE on a small training set. To show this, we demonstrated distributions of dev scores from 20 random restarts on RTE, MRPC, CoLA, and STS-B in Figure 3 We further obtained the highest average/best dev score on each task in Table 3 To confirm the generalization of the our best model on the dev set, we demonstrate the test results scored by the evaluation serve F7) in Table
Table 4: We present the test score when finetuning BERTLARGE with each regularization strat- egy on each task. The first row shows the test scores obtained by using both dropout(p) and wdocay(0, (0, 0.01). These results in the first row are reported by Devlin et al. (2018). They used the learning rate of {2 10-5, 3 x 10-5, 5 epochs with multiple random restarts. They selected the best model on each dev set. In the second row, we demonstrate the test scores obtained by using the proposed mixout in Section 6.2 using mixout (W. pre, 0.7) for the pretrained layers and mixout(wo, 0.7) for the additional output layer where wo is its randomly initialized weight parameter. We used the learning rate of x 10 and a batch size of 32 for 3 epochs with 20 random restarts. We submitted the best model each dev set. The third row shows that the test scores obtained by using both dropout(p) and wdccay (0, 0.01) with same experimental setups of the second row. Bold marks the best within each column. The proposed mixout improves the test scores except MRPC compared to the original regularization
10-5, 5 x 10- and batch size of 32 for
strategy proposed by Devlin al. (2018).
STRATEGY
RTE
MRPC
CoLA
STS-B
Devlin et al. 2018
70.1
89.3
60.5
86.5
mixout(Wpre.. 0.7) & mixout Wo: 0.7)
70.2
89.1
62.1
87.3
(popout(p) wdeeay 0.01)
68.2
88.3
59.6
86.0
For all the tasks except MRPC, the test scores obtained by the proposed mixou TO
are better than those reported by Devlin et al. 20188.. We explored the behaviour of finetuning BERTLARGE with mixout by using the learning rate of 2 x 10-5 while Devlin et al.
2018 obtained their results
by using the learning rate of (2 x 10-5, x 10-5 4 x 10 5 test scores obtained by the regularization strategy of Devlin al. (2018) when the learning rate is 2 x 10-5, The results in this section show that the best model on the dev set generalizes well, and all the experiments based on dev scores in this paper are proper validate the effectiveness of the proposed mixout. For the remaining GLUE tasks such as SST-2 with sufficient number of training instances, we observed that using mixout does not differs from using dropout in Section 6.1 We therefore omit the test results on the other tasks in GLUE.
10 We thus present the
D VERIFICATION OF COROLLARY 1.1 WITH LEAST SQUARES REGRESSION
Corollary 1.1 shows that mixout(u, (u., p) regularizes learning to minimize the deviation from the tar- get model parameter u. and the strength of regularization increases as p increases when the loss func- tion is strongly convex. In order to validate this, we explore the behavior of least squares regression with mizout(u, p) on synthetic dataset. For randomly given wī and w². generated an obser- vation y satisfying That is, the model parameter w given by (W1, W2). We randomly pick u as a target model param- eter for mi mixout(u, (u. p) and perform least squares regression with mixout(u, {0.0, 0.3, 0.6, 0.9)). As shown in Figure 5 w converges to the target model parameter u rather than the true model parameter w" (wf, wx) as the mix probability p increases.
w x + w +e where c is Gaussian noise. We set the model to ÿ = wi + W2-
https:: / / wwbenchmark. com/ eaderboard
The regularization strategy in Section 6.2 using mixout( (Wpre. 0.7) for the pretrained layers and mixout(w., 0.7) for the additional output where wn randomly initialized weight parameter.
15

---

Published as a conference paper at ICLR 2020
400
400
200
-200
-200
-000
true target
mixout/target, 0.0)
-400
true target
mixouttarget, 0.3)
Observation
Observation
-40
-20
20
40
40
20
20
(a) mixout (ueouutu,ue 0.0)
(b) mixout (u, 0.3)
400
200
200
0
-200
-200
mixoutttarget, 0.6)
mixouttbarget, 0.9)
target
Observation
target
Observation
-0
40
(c) mixout(u.. (u,, 0.6)
(d) mixout u,, 0.9)
Figure 5: Behavior of mixout(u, p) for strongly convex loss function. We plot the line obtained by least squares regression with mixout(u, {0.0, 0.3, 0.6, 0.9}) (each green line) on a synthetic dataset (blue dots) generated by the true line (each blue dotted line). As p increases, the regression line (each green line) converges to the target line generated by the target model parameter (each orange dotted line) rather than the true line (each blue dotted line).
E TIME USAGE OF MIXOUT COMPARED TO DROPOUT
We recorded the training time of the experiment in Section to compare the time usage of mixout and that of dropout. It took about 843 seconds to finetune BERTLARGE with mixout(Wpre). On the other hand, it took about 636 seconds to finetune BERT with dropout. mixout(Wpre) spends 32.5% more time than dropout since mixout (wrputtt(wr)))))) needs an additional computation with the pretrained model parameter Wpre- However, as shown in Figure at least 15 finetuning runs among 20 random restarts fail with the chance-level accuracy RTE with dropout(p) for all while only 4 finetuning runs out of 20 random restarts are unusable with mixout (Wpre, 0.8). From this result, it is reasonable finetune with the proposed mixout although this requires additional time usage compared to dropout.
F EXTENSIVE HYPERPARAMETER SEARCH FOR DROPOUT
Devlin et al. 2018 finetuned BERT LARGE with (0opout(0.1) all GLUE (Wang et al. 2018) lasks. They chose to improve the maximum dev score on each downstream task, but we have reported not only the maximum dev score but also the mean dev score to quantitatively compare various regularization techniques in our paper. In this section, we explore the effect of the hyper- parameter p when finetuning BERTLARGE with dropout(p) RTE, MRPC, CoLA, and STS-B to show (topout(O.1) is optimal terms mean dev score. All experimental setups for these experiments the same Section 6.3.
16

---

Published as a conference paper at ICLR 2020
92.5
65
Dey score
87.5
Dev
85.0
50
80.0
Drop probability
Drap proba
RTE (Accuracy)
(h) MRPC (FI accuracy)
80
Dev score
Dev 40
Drop probability
Drop probability
(c) CoLA (Mattew's correlation)
(d) STS-B (Spearman correlation)
Figure 6: Distribution of dev scores on each task from 20 random restarts when finetuning LERGLARGE with dropout (f0.0, 0.1, ... 0.5}). Error intervals show meanistd. When we use dropout(0.1), we have the highest average dev scores on MRPC and STS-B and the eecond-highed average dev scores on RTE and CoLA. These results show that dropout(0.1) almost optimal for all tasks in terms of mean dev score.
As shown in F we have the highest average dev score on MRPC with dropout(0.1) as well as on STS-B. We obtain the highest average dev scores with dropout (0.0) RTE and CoLA, but get the second-highest average dev scores with dropout (0.1) on them. These experiments confirm that the drop probability 0.1 is almost optimal for the highest average dev score on each task.
17


Reviews:
[{"review_id": "S1elsTST9S", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a new regularization technique “mixout” for fine-tuning BERT. Mixout technique mixes the parameters of two models — the pretrained model and the dropout model. Because it keeps pretrained model parameters in consideration all the time, it effectively prevent catastrophic forgetting. Empirical results show that mixout can stabilize fine-tuning BERT on tasks with small training examples (which has been shown to be difficult).\n\nI’d like to accept this paper based on the extensive and detailed experiments and promising results. For example, the theory has been supported by experiment findings on handwriting dataset (computer vision) while the main contribution is on natural language tasks. The authors not only conducted experiments on tasks with less examples (the main focus of this paper) but also on a task with sufficient training examples. For each model regularization configuration, 20 random starts are used to report mean and best performance. This not only makes the results more reliable but also provides deeper insights.\n\nMinor clarification questions:\n\nFor 20 random restarts, are they the same across regularization setups, i.e., for each dot in Figure 3, is there an orange dot that has the same initialization?\n\nWhat will the extreme case, mixout(w_pre, 1.0), behave? According to Figure 1, it would always use w_pre and end up not learning on a target task. If so, would it introduce a cliff in Fig 4? \n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "HkgxWvcTKS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a new regularization technique refered as “mixout”, motivated by dropout. Mixout stochastically mixes the parameters of two models. Experiments shows the stability of finetuning and the method greatly improve the average accuracy.\n\nI really like the proposed idea, and the paper is easy to understand and follow, and the experiments are well designed. \n\nThe time usage of the regularization is not discussed. It seems the method needs to maintain two copies of parameters, it would be much better if the author can provide the time usage of the experiments. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "BJxftmXRqS", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors introduce a new regularization technique for the specific task of finetuning models. It's inspired by dropout and stochastically mixes source and target weights in order to avoid moving the parameters towards 0.\n\nThe authors provide a theoretical justification as to why mixout would do useful things in the convex case and then demonstrate empirically that using it achieves good accuracies on some downstream, finetuned, non-convex-loss-utilizing tasks. Their experiments incorporate both small models with good analysis (ie, sec 4) as well as larger, real-world models (sect 5). The paper is in general well-written.\n\nI have a few concerns that I would like to see addressed:\n\n1a. For starters, all the theoretical motivation describes a particular way in which mixout is supposed to aid in downstream tasks for the case of convex functions, but there are no experiments that match their assumptions and which demonstrate this is the actual behavior we see. It would be nice to see results which demonstrate the theory.\n\n1b. In few of the prsented empirical experiments is it the case that the use of mixout by itself is useful.  Why does mixout have to be coupled with other regularization techniques? There is little analysis given here, either empirical or theoretical. \n\n2. Why are there only 4 GLUE tasks reported? Devlin 2018 reports on all but WNLI.\n\n3. The choice of hyperparameters for GLUE in sect 5 is a bit misleading. Devlin 2018 chose those parameters to get the maximum scores on downstream tasks; the metric they use is max score. However, the authors want to instead discuss the average score against a set of random restarts, perhaps because the max scores using their method aren't terribly different from the baselines. \n\nTherefore, a more extensive hyperparameter sweep should have been run for the baselines: they should have been re-tuned for the average score if that is the metric the authors wish to use. Instead, the authors only used one task, RTE, to find baseline hyperparameters.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents mixout, a regularization method that stochastically mixes parameters of a pretrained language model and a target language model. Experiments on GLUE show that the proposed technique improves the stability and accuracy of finetuning a pretrained BERT on several downstream tasks.

The paper is well written and the proposed idea is applicable in many settings. The authors have addressed reviewers concerns' during the rebuttal period and all reviewers are now in agreement that this paper should be accepted.

I think this paper would be a good addition to ICLR and recommend to accept it.

Decision: Accept (Poster)