MIXOUT: EFFECTIVE REGULARIZATION TO FINETUNE LARGE-SCALE PRETRAINED LANGUAGE MODELS
Year: 2020

Abstract:
In natural language processing, it has been observed recently that generalization could be greatly improved by finetuning a large-scale language model pretrained on a large unlabeled corpus. Despite its recent success and wide adoption, finetuning a large pretrained language model on a downstream task is prone to degenerate performance when there are only a small number of training instances available. In this paper, we introduce a new regularization technique, to which we refer as “mixout”, motivated by dropout. Mixout stochastically mixes the parameters of two models. We show that our mixout technique regularizes learning to minimize the deviation from one of the two models and that the strength of regularization adapts along the optimization trajectory. We empirically evaluate the proposed mixout and its variants on finetuning a pretrained language model on downstream tasks. More specifically, we demonstrate that the stability of finetuning and the average accuracy greatly increase when we use the proposed approach to regularize finetuning of BERT on downstream tasks in GLUE.

heading: 1 INTRODUCTION
text: Transfer learning has been widely used for the tasks in natural language processing (NLP) (Collobert et al., 2011; Devlin et al., 2018; Yang et al., 2019; Liu et al., 2019; Phang et al., 2018). In particular, Devlin et al. (2018) recently demonstrated the effectiveness of finetuning a large-scale language model pretrained on a large, unannotated corpus on a wide range of NLP tasks including question answering and language inference. They have designed two variants of models, BERTLARGE (340M parameters) and BERTBASE (110M parameters). Although BERTLARGE outperforms BERTBASE generally, it was observed that finetuning sometimes fails when a target dataset has fewer than 10,000 training instances (Devlin et al., 2018; Phang et al., 2018).
When finetuning a big, pretrained language model, dropout (Srivastava et al., 2014) has been used as a regularization technique to prevent co-adaptation of neurons (Vaswani et al., 2017; Devlin et al., 2018; Yang et al., 2019). We provide a theoretical understanding of dropout and its variants, such as Gaussian dropout (Wang & Manning, 2013), variational dropout (Kingma et al., 2015), and dropconnect (Wan et al., 2013), as an adaptive L2-penalty toward the origin (all zero parameters 0) and generalize dropout by considering a target model parameter u (instead of the origin), to which we refer as mixout(u). We illustrate mixout(u) in Figure 1. To be specific, mixout(u) replaces all outgoing parameters from a randomly selected neuron to the corresponding parameters of u. mixout(u) avoids optimization from diverging away from u through an adaptive L2-penalty toward u. Unlike mixout(u), dropout encourages a move toward the origin which deviates away from u since dropout is equivalent to mixout(0).
We conduct experiments empirically validating the effectiveness of the proposed mixout(wpre) where wpre denotes a pretrained model parameter. To validate our theoretical findings, we train a fully connected network on EMNIST Digits (Cohen et al., 2017) and finetune it on MNIST. We observe that a finetuning solution of mixout(wpre) deviates less from wpre in the L2-sense than
∗Department of Mathematical Sciences, KAIST, Daejeon, 34141, Republic of Korea †New York University ‡Facebook AI Research §CIFAR Azrieli Global Scholar
that of dropout. In the main experiment, we finetune BERTLARGE with mixout(wpre) on small training sets of GLUE (Wang et al., 2018). We observe that mixout(wpre) reduces the number of unusable models that fail with the chance-level accuracy and increases the average development (dev) scores for all tasks. In the ablation studies, we perform the following three experiments for finetuning BERTLARGE with mixout(wpre): (i) the effect of mixout(wpre) on a sufficient number of training examples, (ii) the effect of a regularization technique for an additional output layer which is not pretrained, and (iii) the effect of probability of mixout(wpre) compared to dropout. From these ablation studies, we observe that three characteristics of mixout(wpre): (i) finetuning with mixout(wpre) does not harm model performance even with a sufficient number of training examples; (ii) It is beneficial to use a variant of mixout as a regularization technique for the additional output layer; (iii) The proposed mixout(wpre) is helpful to the average dev score and to the finetuning stability in a wider range of its hyperparameter p than dropout.

heading: 1.1 RELATED WORK
text: For large-scale pretrained language models (Vaswani et al., 2017; Devlin et al., 2018; Yang et al., 2019), dropout has been used as one of several regularization techniques. The theoretical analysis for dropout as an L2-regularizer toward 0 was explored by Wan et al. (2013) where 0 is the origin. They provided a sharp characterization of dropout for a simplified setting (generalized linear model). Mianjy & Arora (2019) gave a formal and complete characterization of dropout in deep linear networks with squared loss as a nuclear norm regularization toward 0. However, neither Wan et al. (2013) nor Mianjy & Arora (2019) gives theoretical analysis for the extension of dropout which uses a point other than 0.
Wiese et al. (2017), Kirkpatrick et al. (2017), and Schwarz et al. (2018) used L2-penalty toward a pretrained model parameter to improve performance. They focused on preventing catastrophic forgetting to enable their models to learn multiple tasks sequentially. They however do not discuss nor demonstrate the effect of L2-penalty toward the pretrained model parameter on the stability of finetuning. Barone et al. (2017) introduced tuneout, which is a special case of mixout. They applied various regularization techniques including dropout, tuneout, and L2-penalty toward a pretrained model parameter to finetune neural machine translation. They however do not demonstrate empirical significance of tuneout compared to other regularization techniques nor its theoretical justification.

heading: 2 PRELIMINARIES AND NOTATIONS
text: Norms and Loss Functions Unless explicitly stated, a norm ‖ · ‖ refers to L2-norm. A loss function of a neural network is written as L(w) = 1n ∑n i=1 Li(w), where w is a trainable model parameter. Li is “a per-example loss function” computed on the i-th data point.
Strong Convexity A differentiable function f is strongly convex if there exists m > 0 such that
f(y) ≥ f(x) +∇f(x)>(y − x) + m 2 ‖y − x‖2, (1)
for all x and y.
Weight Decay We refer as “wdecay(u, λ)” to minimizing
L(w) + λ 2 ‖w − u‖2,
instead of the original loss function L(w) where λ is a regularization coefficient. Usual weight decay of λ is equivalent to wdecay(0, λ).
Probability for Dropout and Dropconnect Dropout (Srivastava et al., 2014) is a regularization technique selecting a neuron to drop with a probability of p. Dropconnect (Wan et al., 2013) chooses a parameter to drop with a probability of p. To emphasize their hyperparameter p, we write dropout and dropconnect with a drop probability of p as “dropout(p)” and “dropconnect(p)”, respectively. dropout(p) is a special case of dropconnect(p) if we simultaneously drop the parameters outgoing from each dropped neuron.
Inverted Dropout and Dropconnect In the case of dropout(p), a neuron is retained with a probability of 1−p during training. If we denote the weight parameter of that neuron as w during training, then we use (1 − p)w for that weight parameter at test time (Srivastava et al., 2014). This ensures that the expected output of a neuron is the same as the actual output at test time. In this paper, dropout(p) refers to inverted dropout(p) which uses w/(1 − p) instead of w during training. By doing so, we do not need to compute the output separately at test time. Similarly, dropconnect(p) refers to inverted dropconnect(p).

heading: 3 ANALYSIS OF DROPOUT AND ITS GENERALIZATION
text: We start our theoretical analysis by investigating dropconnect which is a general form of dropout and then apply the result derived from dropconnect to dropout. The iterative SGD equation for dropconnect(p) with a learning rate of η is
w(t+1) = w(t) − ηB(t)∇L (( EB(t)1 )−1 B(t)w(t) ) , t = 0, 1, 2, · · · , (2)
where B(t) = diag(B(t)1 , B (t) 2 , · · · , B (t) d ) and B (t) i ’s are mutually independent Bernoulli(1− p) random variables with a drop probability of p for all i and t. We regard equation 2 as finding a solution to the minimization problem below:
min w
EL ( (EB1)−1Bw ) , (3)
where B = diag(B1, B2, · · · , Bd) and Bi’s are mutually independent Bernoulli(1− p) random variables with a drop probability of p for all i.
Gaussian dropout (Wang & Manning, 2013) and variational dropout (Kingma et al., 2015) use other random masks to improve dropout rather than Bernoulli random masks. To explain these variants of dropout as well, we set a random mask matrix M = diag(M1, M2, · · · , Md) to satisfy EMi = µ and Var(Mi) = σ2 for all i. Now we define a random mixture function with respect to w from u and M as
Φ(w; u,M) = µ−1 ( (I −M)u + Mw − (1− µ)u ) , (4)
and a minimization problem with “mixconnect(u, µ, σ2)” as
min w
EL ( Φ(w; u,M) ) . (5)
We can view dropconnect(p) equation 3 as a special case of equation 5 where u = 0 and M = B. We investigate how mixconnect(u, µ, σ2) differs from the vanilla minimization problem
min w
EL(w). (6) If the loss function L is strongly convex, we can derive a lower bound of EL ( Φ(w; u,M) ) as in Theorem 1:
Theorem 1. Assume that the loss function L is strongly convex. Suppose that a random mixture function with respect to w from u and M is given by Φ(w; u,M) in equation 4 where M is diag(M1, M2, · · · , Md) satisfying EMi = µ and Var(Mi) = σ2 for all i. Then, there exists m > 0 such that
EL ( Φ(w; u,M) ) ≥ L(w) + mσ 2
2µ2 ‖w − u‖2, (7)
for all w (Proof in Supplement A).
Theorem 1 shows that minimizing the l.h.s. of equation 7 minimizes the r.h.s. of equation 7 when the r.h.s. is a sharp lower limit of the l.h.s. The strong convexity of L means that L is bounded from below by a quadratic function, and the inequality of equation 7 comes from the strong convexity. Hence, the equality holds if L is quadratic, and mixconnect(u, µ, σ2) is an L2-regularizer with a regularization coefficient of mσ2/µ2.

heading: 3.1 MIXCONNECT TO MIXOUT
text: We propose mixout as a special case of mixconnect, which is motivated by the relationship between dropout and dropconnect. We assume that
w = ( w
(N1) 1 , · · · , w (N1) d1 , w (N2) 1 , · · · , w (N2) d2 , · · · · · · , w(Nk)1 , · · · , w (Nk) dk
) ,
where w(Ni)j is the jth parameter outgoing from the neuron Ni. We set the corresponding M to M = diag ( M (N1), · · · , M (N1), M (N2), · · · , M (N2), · · · · · · ,M (Nk), · · · , M (Nk) ) , (8)
where EM (Ni) = µ and Var(M (Ni)) = σ2 for all i. In this paper, we setM (Ni) to Bernoulli(1−p) for all i and mixout(u) hereafter refers to this correlated version of mixconnect with Bernoulli random masks. We write it as “mixout(u, p)” when we emphasize the mix probability p. Corollary 1.1. Assume that the loss function L is strongly convex. We denote the random mixture function of mixout(u, p), which is equivalent to that of mixconnect(u, 1− p, p− p2), as Φ(w; u,M) where M is defined in equation 8. Then, there exists m > 0 such that
EL ( Φ(w; u,B) ) ≥ L(w) + mp
2(1− p) ‖w − u‖2, (9)
for all w.
Corollary 1.1 is a straightforward result from Theorem 1. As the mix probability p in equation 9 increases to 1, the L2-regularization coefficient of mp/(1− p) increases to infinity. It means that p of mixout(u, p) can adjust the strength of L2-penalty toward u in optimization. mixout(u) differs from wdecay(u) since the regularization coefficient of mixout(u) depends onm determined by the current model parameter w. mixout(u, p) indeed regularizes learning to minimize the deviation from u. We validate this by performing least squares regression in Supplement D.
We often apply dropout to specific layers. For instance, Simonyan & Zisserman (2014) applied dropout to fully connected layers only. We generalize Theorem 1 to the case in which mixout is only applied to specific layers, and it can be done by constructing M in a particular way. We demonstrate this approach in Supplement B and show that mixout for specific layers adaptively L2-penalizes their parameters.

heading: 3.2 MIXOUT FOR PRETRAINED MODELS
text: Hoffer et al. (2017) have empirically shown that ‖wt −w0‖ ∼ log t, (10)
where wt is a model parameter after the t-th SGD step. When training from scratch, we usually sample an initial model parameter w0 from a normal/uniform distribution with mean 0 and small variance. Since w0 is close to the origin, wt is away from the origin only with a large t by equation 10. When finetuning, we initialize our model parameter from a pretrained model parameter wpre. Since we usually obtain wpre by training from scratch on a large pretraining dataset, wpre is often far away from the origin. By Corollary 1.1, dropout L2-penalizes the model parameter for deviating away from the origin rather than wpre. To explicitly prevent the deviation from wpre, we instead propose to use mixout(wpre).

heading: 4 VERIFICATION OF THEORETICAL RESULTS FOR MIXOUT ON MNIST
text: Wiese et al. (2017) have highlighted that wdecay(wpre) is an effective regularization technique to avoid catastrophic forgetting during finetuning. Because mixout(wpre) keeps the finetuned model to stay in the vicinity of the pretrained model similarly to wdecay(wpre), we suspect that the proposed mixout(wpre) has a similar effect of alleviating the issue of catastrophic forgetting. To empirically verify this claim, we pretrain a 784-300-100-10 fully-connected network on EMNIST Digits (Cohen et al., 2017), and finetune it on MNIST. For more detailed description of the model architecture and datasets, see Supplement C.1.
In the pretraining stage, we run five random experiments with a batch size of 32 for {1, 2, · · · , 20} training epochs. We use Adam (Kingma & Ba, 2014) with a learning rate of 10−4, β1 = 0.9, β2 = 0.999, wdecay(0, 0.01), learning rate warm-up over the first 10% steps of the total steps, and linear decay of the learning rate after the warm-up. We use dropout(0.1) for all layers except the input and output layers. We select wpre whose validation accuracy on EMNIST Digits is best (0.992) in all experiments.
For finetuning, most of the model hyperparameters are kept same as in pretraining, with the exception of the learning rate, number of training epochs, and regularization techniques. We train with a learning rate of 5 × 10−5 for 5 training epochs. We replace dropout(p) with mixout(wpre, p). We do not use any other regularization technique such as wdecay(0) and wdecay(wpre). We monitor ‖wft −wpre‖2,1 validation accuracy on MNIST, and validation accuracy on EMNIST Digits to compare mixout(wpre, p) to dropout(p) across 10 random restarts.2
As shown in Figure 2 (a), after finetuning with mixout(wpre, p), the deviation from wpre is minimized in the L2-sense. This result verifies Corollary 1.1. We demonstrate that the validation accuracy of mixout(wpre, p) has greater robustness to the choice of p than that of dropout(p). In Figure 2 (b), both dropout(p) and mixout(wpre, p) result in high validation accuracy on the target task (MNIST) for p ∈ {0.1, 0.2, 0.3}, although mixout(wpre, p) is much more robust with respect to the choice of the mix probability p. In Figure 2 (c), the validation accuracy of mixout(wpre, p) on the source task (EMNIST Digits) drops from the validation accuracy of the model at wpre (0.992) to approximately 0.723 regardless of p. On the other hand, the validation accuracy of dropout(p) on the source task respectively drops by 0.041, 0.074 and 0.105 which are more than those of mixout(wpre, p) for p ∈ {0.1, 0.2, 0.3}.
1wft is a model parameter after finetuning. 2Using the same pretrained model parameter wpre but perform different finetuning data shuffling.

heading: 5 FINETUNING A PRETRAINED LANGUAGE MODEL WITH MIXOUT
text: In order to experimentally validate the effectiveness of mixout, we finetune BERTLARGE on a subset of GLUE (Wang et al., 2018) tasks (RTE, MRPC, CoLA, and STS-B) with mixout(wpre). We choose them because Phang et al. (2018) have observed that it was unstable to finetune BERTLARGE on these four tasks. We use the publicly available pretrained model released by Devlin et al. (2018), ported into PyTorch by HuggingFace.3 We use the learning setup and hyperparameters recommended by Devlin et al. (2018). We use Adam with a learning rate of 2 × 10−5, β1 = 0.9, β2 = 0.999, learning rate warmup over the first 10% steps of the total steps, and linear decay of the learning rate after the warmup finishes. We train with a batch size of 32 for 3 training epochs. Since the pretrained BERTLARGE is the sentence encoder, we have to create an additional output layer, which is not pretrained. We initialize each parameter of it withN (0, 0.022). We describe our experimental setup further in Supplement C.2.
The original regularization strategy used in Devlin et al. (2018) for finetuning BERTLARGE is using both dropout(0.1) and wdecay(0, 0.01) for all layers except layer normalization and intermediate layers activated by GELU (Hendrycks & Gimpel, 2016). We however cannot use mixout(wpre) nor wdecay(wpre) for the additional output layer which was not pretrained and therefore does not have wpre. We do not use any regularization for the additional output layer when finetuning BERTLARGE with mixout(wpre) and wdecay(wpre). For the other layers, we replace dropout(0.1) and wdecay(0, 0.01) with mixout(wpre) and wdecay(wpre), respectively.
Phang et al. (2018) have reported that large pretrained models (e.g., BERTLARGE) are prone to degenerate performance when finetuned on a task with a small number of training examples, and that multiple random restarts4 are required to obtain a usable model better than random prediction. To compare finetuning stability of the regularization techniques, we need to demonstrate the distribution of model performance. We therefore train BERTLARGE with each regularization strategy on each task with 20 random restarts. We validate each random restart on the dev set to observe the behaviour of the proposed mixout and finally evaluate it on the test set for generalization. We present the test score of our proposed regularization strategy on each task in Supplement C.3.
We finetune BERTLARGE with mixout(wpre, {0.7, 0.8, 0.9}) on RTE, MRPC, CoLA, and STSB. For the baselines, we finetune BERTLARGE with both dropout(0.1) and wdecay(0, 0.01) as well as with wdecay(wpre, {0.01, 0.04, 0.07, 0.10}). These choices are made based on the experiments in Section 6.3 and Supplement F. In Section 6.3, we observe that finetuning BERTLARGE with mixout(wpre, p) on RTE is significantly more stable with p ∈ {0.7, 0.8, 0.9} while finetuning with dropout(p) becomes unstable as p increases. In Supplement F, we demonstrate that dropout(0.1) is almost optimal for all the tasks in terms of mean dev score although Devlin et al. (2018) selected it to improve the maximum dev score.
In Figure 3, we plot the distributions of the dev scores from 20 random restarts when finetuning BERTLARGE with various regularization strategies on each task. For conciseness, we only show four regularization strategies; Devlin et al. (2018)’s: both dropout(0.1) and wdecay(0, 0.01), Wiese et al. (2017)’s: wdecay(wpre, 0.01), ours: mixout(wpre, 0.7), and ours+Wiese et al. (2017)’s: both mixout(wpre, 0.7) and wdecay(wpre, 0.01). As shown in Figure 3 (a–c), we observe many finetuning runs that fail with the chance-level accuracy when we finetune BERTLARGE with both dropout(0.1) and wdecay(0, 0.01) on RTE, MRPC, and CoLA. We also have a bunch of degenerate model configurations when we use wdecay(wpre, 0.01) without mixout(wpre, 0.7).
Unlike existing regularization strategies, when we use mixout(wpre, 0.7) as a regularization technique with or without wdecay(wpre, 0.01) for finetuning BERTLARGE, the number of degenerate model configurations that fail with a chance-level accuracy significantly decreases. For example, in Figure 3 (c), we have only one degenerate model configuration when finetuning BERTLARGE with mixout(wpre, 0.7) on CoLA while we observe respectively seven and six degenerate models with Devlin et al. (2018)’s and Wiese et al. (2017)’s regularization strategies.
3https : / / s3 . amazonaws . com / models . huggingface . co / bert / bert-large-uncased-pytorch_model.bin
4Using the same pretrained model parameter wpre but each random restart differs from the others by shuffling target data and initializing the additional output layer differently.
In Figure 3 (a), we further improve the stability of finetuning BERTLARGE by using both mixout(wpre, 0.7) and wdecay(wpre, 0.01). Figure 3 (d) shows respectively two and one degenerate model configurations with Devlin et al. (2018)’s and Wiese et al. (2017)’s, but we do not have any degenerate resulting model with ours and ours+Wiese et al. (2017)’s. In Figure 3 (b, c), we observe that the number of degenerate model configurations increases when we use wdecay(wpre, 0.01) additionally to mixout(wpre, 0.7). In short, applying our proposed mixout significantly stabilizes the finetuning results of BERTLARGE on small training sets regardless of whether we use wdecay(wpre, 0.01).
In Table 1, we report the average and the best dev scores across 20 random restarts for each task with various regularization strategies. The average dev scores with mixout(wpre, {0.7, 0.8, 0.9}) increase for all the tasks. For instance, the mean dev score of finetuning with mixout(wpre, 0.8) on CoLA is 57.9 which is 49.2% increase over 38.8 obtained by finetuning with both dropout(p) and wdecay(0, 0.01). We observe that using wdecay(wpre, {0.01, 0.04, 0.07, 0.10}) also improves the average dev scores for most tasks compared to using both dropout(p) and wdecay(0, 0.01). We however observe that finetuning with mixout(wpre, {0.7, 0.8, 0.9}) outperforms that with wdecay(wpre, {0.01, 0.04, 0.07, 0.10}) on average. This confirms that mixout(wpre) has a different effect for finetuning BERTLARGE compared to wdecay(wpre) since mixout(wpre) is an adaptive L2-regularizer along the optimization trajectory.
Since finetuning a large pretrained language model such as BERTLARGE on a small training set frequently fails, the final model performance has often been reported as the maximum dev score (Devlin et al., 2018; Phang et al., 2018) among a few random restarts. We thus report the best dev score for each setting in Table 1. According to the best dev scores as well, mixout(wpre, {0.7, 0.8, 0.9}) improves performance for all the tasks compared to using both dropout(p) and wdecay(0, 0.01). For instance, using mixout(wpre, 0.9) improves the maximum dev score by 0.9 compared to using both dropout(p) and wdecay(0, 0.01) on MRPC. Unlike the average dev scores, the best dev scores achieved by using wdecay(wpre, {0.01, 0.04, 0.07, 0.10}) are better than those achieved by using mixout(wpre, {0.7, 0.8, 0.9}) except RTE on which it was better to use mixout(wpre, {0.7, 0.8, 0.9}) than wdecay(wpre, {0.01, 0.04, 0.07, 0.10}).
We investigate the effect of combining both mixout(wpre) and wdecay(wpre) to see whether they are complementary. We finetune BERTLARGE with both mixout(wpre, {0.7, 0.8, 0.9}) and wdecay(wpre, 0.01). This leads not only to the improvement in the average dev scores but also in the best dev scores compared to using wdecay(wpre, {0.01, 0.04, 0.07, 0.10}) and using both dropout(p) and wdecay(0, 0.01). The experiments in this section confirm that using mixout(wpre) as one of several regularization techniques prevents finetuning instability and yields gains in dev scores.

heading: 6 ABLATION STUDY
text: In this section, we perform ablation experiments to better understand mixout(wpre). Unless explicitly stated, all experimental setups are the same as in Section 5.

heading: 6.1 MIXOUT WITH A SUFFICIENT NUMBER OF TRAINING EXAMPLES
text: We showed the effectiveness of the proposed mixout finetuning with only a few training examples in Section 5. In this section, we investigate the effectiveness of the proposed mixout in the case of a larger finetuning set. Since it has been stable to finetune BERTLARGE on a sufficient number of training examples (Devlin et al., 2018; Phang et al., 2018), we expect to see the change in the behaviour of mixout(wpre) when we use it to finetune BERTLARGE on a larger training set.
We train BERTLARGE by using both mixout(wpre, 0.7) and wdecay(wpre, 0.01) with 20 random restarts on SST-2.5 We also train BERTLARGE by using both dropout(p) and wdecay(0, 0.01) with 20 random restarts on SST-2 as the baseline. In Table 2, we report the mean and maximum of
5For the description of SST-2, see Supplement C.2.
SST-2 dev scores across 20 random restarts with each regularization strategy. We observe that there is little difference between their mean and maximum dev scores on a larger training set, although using both mixout(wpre, 0.7) and wdecay(wpre, 0.01) outperformed using both dropout(p) and wdecay(0, 0.01) on the small training sets in Section 5.

heading: 6.2 EFFECT OF A REGULARIZATION TECHNIQUE FOR AN ADDITIONAL OUTPUT LAYER
text: In this section, we explore the effect of a regularization technique for an additional output layer. There are two regularization techniques available for the additional output layer: dropout(p) and mixout(w0, p) where w0 is its randomly initialized parameter. Either of these strategies differs from the earlier experiments in Section 5 where we did not put any regularization for the additional output layer.
We report the average and best dev scores across 20 random restarts when finetuning BERTLARGE with mixout(wpre, 0.7) while varying the regularization technique for the additional output layer in
Table 3.6 We observe that using mixout(w0, 0.7) for the additional output layer improves both the average and best dev score on RTE, CoLA, and STS-B. In the case of MRPC, we have the highest best-dev score by using dropout(0.7) for the additional output layer while the highest mean dev score is obtained by using mixout(w0, 0.7) for it. In Section 3.2, we discussed how mixout(w0) does not differ from dropout when the layer is randomly initialized, since we sample w0 from w whose mean and variance are 0 and small, respectively. Although the additional output layer is randomly initialized, we observe the significant difference between dropout and mixout(w0) in this layer. We conjecture that ‖w0 − 0‖ is not sufficiently small because E‖w − 0‖ is proportional to the dimensionality of the layer (2,048). We therefore expect mixout(w0) to behave differently from dropout even for the case of training from scratch.
In the last row of Table 3, we present the best of the corresponding result from Table 1. We have the highest mean and best dev scores when we respectively use mixout(wpre, 0.7) and mixout(w0, 0.7) for the pretrained layers and the additional output layer on RTE, CoLA, and STSB. The highest mean dev score on MRPC is obtained by using mixout(wpre, 0.8) for the pretrained layers which is one of the results in Table 1. We have the highest best dev score on MRPC when we use mixout(wpre, 0.7) and dropout(0.7) for the pretrained layers and the additional output
6In this experiment, we use neither wdecay(0) nor wdecay(wpre).
layer, respectively. The experiments in this section reveal that using mixout(w0) for a randomly initialized layer of a pretrained model is one of the regularization schemes to improve the average dev score and the best dev score.

heading: 6.3 EFFECT OF MIX PROBABILITY FOR MIXOUT AND DROPOUT
text: We explore the effect of the hyperparameter pwhen finetuning BERTLARGE with mixout(wpre, p) and dropout(p). We train BERTLARGE with mixout(wpre, {0.0, 0.1, · · · , 0.9}) on RTE with 20 random restarts. We also train BERTLARGE after replacing mixout(wpre, p) by dropout(p) with 20 random restarts. We do not use any regularization technique for the additional output layer. Because we use neither wdecay(0) nor wdecay(wpre) in this section, dropout(0.0) and mixout(wpre, 0.0) are equivalent to finetuning without regularization.
It is not helpful to vary p for dropout(p) while mixout(wpre, p) helps significantly in a wide range of p. Figure 4 shows distributions of RTE dev scores across 20 random restarts when finetuning BERTLARGE with dropout(p) and mixout(wpre, p) for p ∈ {0.0, 0.1, · · · , 0.9}. The mean dev score of finetuning BERTLARGE with mixout(wpre, p) increases as p increases. On the other hand, the mean dev score of finetuning BERTLARGE with dropout(p) decreases as p increases. If p is less than 0.4, finetuning with mixout(wpre, p) does not improve the finetuning results of using dropout({0.0, 0.1, 0.2}). We however observe that mixout(wpre, {0.7, 0.8, 0.9}) yields better average dev scores than dropout(p) for all p, and significantly reduces the number of finetuning runs that fail with the chance-level accuracy.
We notice that the proposed mixout spends more time than dropout from the experiments in this section. It takes longer to finetune a model with the proposed mixout than with the original dropout, although this increase is not significant especially considering the waste of time from failed finetuning runs using dropout. In Supplement E, we describe more in detail the difference between mixout and dropout in terms of wall-clock time.

heading: 7 CONCLUSION
text: The special case of our approach, mixout(wpre), is one of several regularization techniques modifying a finetuning procedure to prevent catastrophic forgetting. Unlike wdecay(wpre) proposed earlier by Wiese et al. (2017), mixout(wpre) is an adaptive L2-regularizer toward wpre in the sense that its regularization coefficient adapts along the optimization path. Due to this difference, the proposed mixout improves the stability of finetuning a big, pretrained language model even with only a few training examples of a target task. Furthermore, our experiments have revealed the proposed approach improves finetuning results in terms of the average accuracy and the best accuracy over multiple runs. We emphasize that our approach can be used with any pretrained language models such as RoBERTa (Liu et al., 2019) and XLNet (Yang et al., 2019), since mixout does not depend on model architectures, and leave it as future work.

heading: ACKNOWLEDGMENTS
text: The first and third authors’ work was supported by the National Research Foundation of Korea (NRF) grants funded by the Korea government (MOE, MSIT) (NRF-2017R1A2B4011546, NRF2019R1A5A1028324). The second author thanks support by AdeptMind, eBay, TenCent, NVIDIA and CIFAR and was partly supported by Samsung Electronics (Improving Deep Learning using Latent Structure).

heading: A PROOFS FOR THEOREM 1
text: Theorem 1. Assume that the loss function L is strongly convex. Suppose that a random mixture function with respect to w from u and M is given by
Φ(w; u,M) = µ−1 ( (I −M)u + Mw − (1− µ)u ) ,
where M is diag(M1, M2, · · · , Md) satisfying EMi = µ and Var(Mi) = σ2 for all i. Then, there exists m > 0 such that
EL ( Φ(w; u,M) ) ≥ L(w) + mσ 2
2µ2 ‖w − u‖2, (11)
for all w.
Proof. Since L is strongly convex, there exist m > 0 such that
EL ( Φ(w; u,M) ) = EL ( w + ( Φ(w; u,M)−w )) ≥ L(w) +∇L(w)>E[Φ(w; u,M)−w] + m
2 E‖Φ(w; u,M)−w‖2,
(12)
for all w by equation 1. Recall that EMi = µ and Var(Mi) = σ2 for all i. Then, we have
E[Φ(w; u,M)−w] = 0, (13)
and
E‖Φ(w; u,M)−w‖2 = E ∥∥∥∥ 1µ (w − u)(M − µI) ∥∥∥∥2 = 1
µ2 d∑ i=1 (wi − ui)2E(Mi − µ)2
= σ2
µ2 ‖w − u‖2. (14)
By using equation 13 and equation 14, we can rewrite equation 12 as
EL ( Φ(w; u,M) ) ≥ L(w) + mσ 2
2µ2 ‖w − u‖2.

heading: B APPLYING TO SPECIFIC LAYERS
text: We often apply dropout to specific layers. For instance, Simonyan & Zisserman (2014) applied dropout to fully connected layers only. We generalize Theorem 1 to the case in which mixconnect is only applied to specific layers, and it can be done by constructing M in a particular way. To better characterize mixconnect applied to specific layers, we define the index set I as I = {i : Mi = 1}. Furthermore, we use w̃ and ũ to denote (wi)i/∈I and (ui)i/∈I, respectively. Then, we generalize equation 7 to
EL ( Φ(w; u,M) ) ≥ L(w) + mσ 2
2µ2 ‖w̃ − ũ‖2. (15)
From equation 15, applying mixconnect(u, µ, σ2) is to use adaptive wdecay(ũ) on the weight parameter of the specific layers w̃. Similarly, we can regard applying mixout(u, p) to specific layers as adaptive wdecay(ũ).

heading: C EXPERIMENTAL DETAILS
text: C.1 FROM EMNIST DIGITS TO MNIST
Model Architecture The model architecture in Section 4 is a 784-300-100-10 fully connected network with a softmax output layer. For each hidden layer, we add layer normalization (Ba et al., 2016) right after the ReLU (Nair & Hinton, 2010) nonlinearity. We initialize each parameter with N (0, 0.022) and each bias with 0.
Regularization In the pretraining stage, we use dropout(0.1) and wdecay(0, 0.01). We apply dropout(0.1) to all hidden layers. That is, we do not drop neurons of the input and output layers. wdecay(0, 0.01) does not penalize the parameters for bias and layer normalization. When we finetune our model on MNIST, we replace dropout(p) with mixout(wpre, p). We use neither wdecay(0) nor wdecay(wpre) for finetuning.
Dataset For pretraining, we train our model on EMNIST Digits. This dataset has 280,000 characters into 10 balanced classes. These characters are compatible with MNIST characters. EMNIST Digits provides 240,000 characters for training and 40,000 characters for test. We use 240,000 characters provide for training and split these into the training set (216,000 characters) and validation set (24,000 characters). For finetuning, we train our model on MNIST. This has 70,000 characters into 10 balance classes. MNIST provide 60,000 characters for training and 10,000 characters for test. We use 60,000 characters given for training and split these into the training set (54,000 characters) and validation set (6,000 characters).
Data Preprocessing We only use normalization after scaling pixel values into [0, 1]. We do not use any data augmentation.
C.2 FINETUNING BERT ON PARTIAL GLUE TASKS
Model Architecture Because the model architecture of BERTLARGE is identical to the original (Devlin et al., 2018), we omit its exhaustive description. Briefly, BERTLARGE has 24 layers, 1024 hidden size, and 16 self-attention heads (total 340M parameters). We use the publicly available pretrained model released by Devlin et al. (2018), ported into PyTorch by HuggingFace.7 We initialize each weight parameter and bias for an additional output layer withN (0, 0.022) and 0, respectively.
Regularization In the finetuning stage, Devlin et al. (2018) used wdecay(0, 0.01) for all parameters except bias and layer normalization. They apply dropout(0.1) to all layers except each hidden layer activated by GELU (Hendrycks & Gimpel, 2016) and layer normalization. We substitute wdecay(wpre) and mixout(wpre) for wdecay(0, 0.01) and dropout(0.1), respectively.
Dataset We use a subset of GLUE (Wang et al., 2018) tasks. The brief description for each dataset is as the following:
• RTE (2,500 training examples): Binary entailment task (Dagan et al., 2006) • MRPC (3,700 training examples): Semantic similarity (Dolan & Brockett, 2005) • CoLA (8,500 training examples): Acceptability classification (Warstadt et al., 2018) • STS-B (7,000 training examples): Semantic textual similarity (Cer et al., 2017) • SST-2 (67,000 training examples): Binary sentiment classification (Socher et al., 2013)
In this paper, we reported F1 accuracy scores for MRPC, Mattew’s correlation scores for CoLA, Spearman correlation scores for STS-B, and accuracy scores for the other tasks.
Data Preprocessing We use the publicly available implementation of BERTLARGE by HuggingFace.8
7https : / / s3 . amazonaws . com / models . huggingface . co / bert / bert-large-uncased-pytorch_model.bin
8https://github.com/huggingface/pytorch-transformers
C.3 TEST RESULTS ON GLUE TASKS
We expect that using mixout stabilizes finetuning results of BERTLARGE on a small training set. To show this, we demonstrated distributions of dev scores from 20 random restarts on RTE, MRPC, CoLA, and STS-B in Figure 3. We further obtained the highest average/best dev score on each task in Table 3. To confirm the generalization of the our best model on the dev set, we demonstrate the test results scored by the evaluation server9 in Table 4.
For all the tasks except MRPC, the test scores obtained by the proposed mixout10 are better than those reported by Devlin et al. (2018). We explored the behaviour of finetuning BERTLARGE with mixout by using the learning rate of 2 × 10−5 while Devlin et al. (2018) obtained their results by using the learning rate of {2 × 10−5, 3 × 10−5, 4 × 10−5, 5 × 10−5}. We thus present the test scores obtained by the regularization strategy of Devlin et al. (2018) when the learning rate is 2 × 10−5. The results in this section show that the best model on the dev set generalizes well, and all the experiments based on dev scores in this paper are proper to validate the effectiveness of the proposed mixout. For the remaining GLUE tasks such as SST-2 with a sufficient number of training instances, we observed that using mixout does not differs from using dropout in Section 6.1. We therefore omit the test results on the other tasks in GLUE.
D VERIFICATION OF COROLLARY 1.1 WITH LEAST SQUARES REGRESSION
Corollary 1.1 shows that mixout(u, p) regularizes learning to minimize the deviation from the target model parameter u, and the strength of regularization increases as p increases when the loss function is strongly convex. In order to validate this, we explore the behavior of least squares regression with mixout(u, p) on a synthetic dataset. For randomly given w∗1 and w ∗ 2 , we generated an observation y satisfying y = w∗1x+w ∗ 2 + where is Gaussian noise. We set the model to ŷ = w1x+w2. That is, the model parameter w is given by (w1, w2). We randomly pick u as a target model parameter for mixout(u, p) and perform least squares regression with mixout(u, {0.0, 0.3, 0.6, 0.9}). As shown in Figure 5, w converges to the target model parameter u rather than the true model parameter w∗ = (w∗1 , w ∗ 2) as the mix probability p increases.
9https://gluebenchmark.com/leaderboard 10The regularization strategy in Section 6.2: using mixout(wpre, 0.7) for the pretrained layers and
mixout(w0, 0.7) for the additional output where w0 is its randomly initialized weight parameter.

heading: E TIME USAGE OF MIXOUT COMPARED TO DROPOUT
text: We recorded the training time of the experiment in Section 6.3 to compare the time usage of mixout and that of dropout. It took about 843 seconds to finetune BERTLARGE with mixout(wpre). On the other hand, it took about 636 seconds to finetune BERTLARGE with dropout. mixout(wpre) spends 32.5% more time than dropout since mixout(wpre) needs an additional computation with the pretrained model parameter wpre. However, as shown in Figure 4, at least 15 finetuning runs among 20 random restarts fail with the chance-level accuracy on RTE with dropout(p) for all p while only 4 finetuning runs out of 20 random restarts are unusable with mixout(wpre, 0.8). From this result, it is reasonable to finetune with the proposed mixout although this requires additional time usage compared to dropout.

heading: F EXTENSIVE HYPERPARAMETER SEARCH FOR DROPOUT
text: Devlin et al. (2018) finetuned BERTLARGE with dropout(0.1) on all GLUE (Wang et al., 2018) tasks. They chose it to improve the maximum dev score on each downstream task, but we have reported not only the maximum dev score but also the mean dev score to quantitatively compare various regularization techniques in our paper. In this section, we explore the effect of the hyperparameter p when finetuning BERTLARGE with dropout(p) on RTE, MRPC, CoLA, and STS-B to show dropout(0.1) is optimal in terms of mean dev score. All experimental setups for these experiments are the same as Section 6.3.
As shown in Figure 6, we have the highest average dev score on MRPC with dropout(0.1) as well as on STS-B. We obtain the highest average dev scores with dropout(0.0) on RTE and CoLA, but we get the second-highest average dev scores with dropout(0.1) on them. These experiments confirm that the drop probability 0.1 is almost optimal for the highest average dev score on each task.

Review summaries:
This paper introduces a new regularization technique “mixout” for fine-tuning BERT. Mixout technique mixes the parameters of two models — the pretrained model and the dropout model. Because it keeps pretrained model parameters in consideration all the time, it effectively prevent catastrophic forgetting. Empirical results show that mixout can stabilize fine-tuning BERT on tasks with small training examples (which has been shown to be difficult).