Title: MIXOUT: EFFECTIVE REGULARIZATION TO FINETUNE LARGE-SCALE PRETRAINED LANGUAGE MODELS

Abstract: In natural language processing, it has been observed recently that generalization could be greatly improved by finetuning a large-scale language model pretrained on a large unlabeled corpus. Despite its recent success and wide adoption, finetuning a large pretrained language model on a downstream task is prone to degenerate performance when there are only a small number of training instances available. In this paper, we introduce a new regularization technique, to which we refer as "mixout", motivated by dropout. Mixout stochastically mixes the parameters of two models. We show that our mixout technique regularizes learning to minimize the deviation from one of the two models and that the strength of regularization adapts along the optimization trajectory. We empirically evaluate the proposed mixout and its variants on finetuning a pretrained language model on downstream tasks. More specifically, we demonstrate that the stability of finetuning and the average accuracy greatly increase when we use the proposed approach to regularize finetuning of BERT on downstream tasks in GLUE.

1. INTRODUCTION:

Transfer learning has been widely used for the tasks in natural language processing (NLP) (Collobert et al., 2011; Devlin et al., 2018; Yang et al., 2019; Liu et al., 2019; Phang et al., 2018) . In particular, Devlin et al. (2018) recently demonstrated the effectiveness of finetuning a large-scale language model pretrained on a large, unannotated corpus on a wide range of NLP tasks including question answering and language inference. They have designed two variants of models, BERT LARGE (340M parameters) and BERT BASE (110M parameters). Although BERT LARGE outperforms BERT BASE generally, it was observed that finetuning sometimes fails when a target dataset has fewer than 10,000 training instances (Devlin et al., 2018; Phang et al., 2018) .

When finetuning a big, pretrained language model, dropout (Srivastava et al., 2014) has been used as a regularization technique to prevent co-adaptation of neurons (Vaswani et al., 2017; Devlin et al., 2018; Yang et al., 2019) . We provide a theoretical understanding of dropout and its variants, such as Gaussian dropout (Wang & Manning, 2013 ), variational dropout (Kingma et al., 2015) , and dropconnect (Wan et al., 2013) , as an adaptive L 2 -penalty toward the origin (all zero parameters 0) and generalize dropout by considering a target model parameter u (instead of the origin), to which we refer as mixout(u). We illustrate mixout(u) in Figure 1 . To be specific, mixout(u) replaces all outgoing parameters from a randomly selected neuron to the corresponding parameters of u. mixout(u) avoids optimization from diverging away from u through an adaptive L 2 -penalty toward u. Unlike mixout(u), dropout encourages a move toward the origin which deviates away from u since dropout is equivalent to mixout(0).

We conduct experiments empirically validating the effectiveness of the proposed mixout(w pre ) where w pre denotes a pretrained model parameter. To validate our theoretical findings, we train a fully connected network on EMNIST Digits (Cohen et al., 2017) and finetune it on MNIST. We observe that a finetuning solution of mixout(w pre ) deviates less from w pre in the L 2 -sense than In the dropout network, we randomly choose an input neuron to be dropped (a dotted neuron) with a probability of p. That is, all outgoing parameters from the dropped neuron are eliminated (dotted connections). (c): In the mixout(u) network, the eliminated parameters in (b) are replaced by the corresponding parameters in (a). In other words, the mixout(u) network at w is the mixture of the vanilla network at u and the dropout network at w with a probability of p.

that of dropout. In the main experiment, we finetune BERT LARGE with mixout(w pre ) on small training sets of GLUE (Wang et al., 2018) . We observe that mixout(w pre ) reduces the number of unusable models that fail with the chance-level accuracy and increases the average development (dev) scores for all tasks. In the ablation studies, we perform the following three experiments for finetuning BERT LARGE with mixout(w pre ): (i) the effect of mixout(w pre ) on a sufficient number of training examples, (ii) the effect of a regularization technique for an additional output layer which is not pretrained, and (iii) the effect of probability of mixout(w pre ) compared to dropout. From these ablation studies, we observe that three characteristics of mixout(w pre ): (i) finetuning with mixout(w pre ) does not harm model performance even with a sufficient number of training examples; (ii) It is beneficial to use a variant of mixout as a regularization technique for the additional output layer; (iii) The proposed mixout(w pre ) is helpful to the average dev score and to the finetuning stability in a wider range of its hyperparameter p than dropout.

1.1. RELATED WORK:

For large-scale pretrained language models (Vaswani et al., 2017; Devlin et al., 2018; Yang et al., 2019) , dropout has been used as one of several regularization techniques. The theoretical analysis for dropout as an L 2 -regularizer toward 0 was explored by Wan et al. (2013) where 0 is the origin. They provided a sharp characterization of dropout for a simplified setting (generalized linear model). Mianjy & Arora (2019) gave a formal and complete characterization of dropout in deep linear networks with squared loss as a nuclear norm regularization toward 0. However, neither Wan et al. (2013) nor Mianjy & Arora (2019) gives theoretical analysis for the extension of dropout which uses a point other than 0. 

2. PRELIMINARIES AND NOTATIONS:

Norms and Loss Functions Unless explicitly stated, a norm • refers to L 2 -norm. A loss function of a neural network is written as

L(w) = 1 n n i=1 L i (w)

, where w is a trainable model parameter. L i is "a per-example loss function" computed on the i-th data point.

Strong Convexity:

EQUATION

for all x and y.

Weight Decay We refer as "wdecay(u, λ)" to minimizing

L(w) + λ 2 w -u 2 ,

instead of the original loss function L(w) where λ is a regularization coefficient. Usual weight decay of λ is equivalent to wdecay(0, λ).

Probability for Dropout and Dropconnect Dropout (Srivastava et al., 2014 ) is a regularization technique selecting a neuron to drop with a probability of p. Dropconnect (Wan et al., 2013) chooses a parameter to drop with a probability of p. To emphasize their hyperparameter p, we write dropout and dropconnect with a drop probability of p as "dropout(p)" and "dropconnect(p)", respectively. dropout(p) is a special case of dropconnect(p) if we simultaneously drop the parameters outgoing from each dropped neuron.

Inverted Dropout and Dropconnect In the case of dropout(p), a neuron is retained with a probability of 1-p during training. If we denote the weight parameter of that neuron as w during training, then we use (1 -p)w for that weight parameter at test time (Srivastava et al., 2014) . This ensures that the expected output of a neuron is the same as the actual output at test time. In this paper, dropout(p) refers to inverted dropout(p) which uses w/(1 -p) instead of w during training. By doing so, we do not need to compute the output separately at test time. Similarly, dropconnect(p) refers to inverted dropconnect(p).

3. ANALYSIS OF DROPOUT AND ITS GENERALIZATION:

We start our theoretical analysis by investigating dropconnect which is a general form of dropout and then apply the result derived from dropconnect to dropout. The iterative SGD equation for dropconnect(p) with a learning rate of η is

EQUATION

where

B (t) = diag(B (t) 1 , B (t) 2 , • • • , B (t) d ) and B (t)

i 's are mutually independent Bernoulli(1 -p) random variables with a drop probability of p for all i and t. We regard equation 2 as finding a solution to the minimization problem below:

EQUATION

where

B = diag(B 1 , B 2 , • • • , B d

) and B i 's are mutually independent Bernoulli(1 -p) random variables with a drop probability of p for all i.

Gaussian dropout (Wang & Manning, 2013) and variational dropout (Kingma et al., 2015) use other random masks to improve dropout rather than Bernoulli random masks. To explain these variants of dropout as well, we set a random mask matrix

M = diag(M 1 , M 2 , • • • , M d ) to satisfy EM i = µ

and Var(M i ) = σ 2 for all i. Now we define a random mixture function with respect to w from u and M as

EQUATION

and a minimization problem with "mixconnect(u, µ, σ 2 )" as min w EL Φ(w; u, M ) .

(5)

We can view dropconnect(p) equation 3 as a special case of equation 5 where u = 0 and M = B. We investigate how mixconnect(u, µ, σ 2 ) differs from the vanilla minimization problem min w EL(w).

If the loss function L is strongly convex, we can derive a lower bound of EL Φ(w; u, M ) as in Theorem 1:

Theorem 1. Assume that the loss function L is strongly convex. Suppose that a random mixture function with respect to w from u and M is given by Φ(w; u, M ) in equation 4 where

M is diag(M 1 , M 2 , • • • , M d ) satisfying EM i = µ and Var(M i ) = σ 2 for all i.

Then, there exists m > 0 such that

EQUATION

)

for all w (Proof in Supplement A).

Theorem 1 shows that minimizing the l.h.s. of equation 7 minimizes the r.h.s. of equation 7 when the r.h.s. is a sharp lower limit of the l.h.s. The strong convexity of L means that L is bounded from below by a quadratic function, and the inequality of equation 7 comes from the strong convexity.

Hence, the equality holds if L is quadratic, and mixconnect(u, µ, σ 2 ) is an L 2 -regularizer with a regularization coefficient of mσ 2 /µ 2 .

3.1. MIXCONNECT TO MIXOUT:

We propose mixout as a special case of mixconnect, which is motivated by the relationship between dropout and dropconnect. We assume that

w = w (N1) 1 , • • • , w (N1) d1 , w (N2) 1 , • • • , w (N2) d2 , • • • • • • , w (N k ) 1 , • • • , w (N k ) d k ,

where w

(Ni) j

is the jth parameter outgoing from the neuron N i . We set the corresponding M to

EQUATION

where EM (Ni) = µ and Var(M (Ni) ) = σ 2 for all i. In this paper, we set M (Ni) to Bernoulli(1 -p) for all i and mixout(u) hereafter refers to this correlated version of mixconnect with Bernoulli random masks. We write it as "mixout(u, p)" when we emphasize the mix probability p. Corollary 1.1. Assume that the loss function L is strongly convex. We denote the random mixture function of mixout(u, p), which is equivalent to that of mixconnect(u, 1 -p, p -p 2 ), as Φ(w; u, M ) where M is defined in equation 8. Then, there exists m > 0 such that

EQUATION

for all w.

Corollary 1.1 is a straightforward result from Theorem 1. As the mix probability p in equation 9 increases to 1, the L 2 -regularization coefficient of mp/(1 -p) increases to infinity. It means that p of mixout(u, p) can adjust the strength of L 2 -penalty toward u in optimization. mixout(u) differs from wdecay(u) since the regularization coefficient of mixout(u) depends on m determined by the current model parameter w. mixout(u, p) indeed regularizes learning to minimize the deviation from u. We validate this by performing least squares regression in Supplement D.

We often apply dropout to specific layers. For instance, Simonyan & Zisserman (2014) applied dropout to fully connected layers only. We generalize Theorem 1 to the case in which mixout is only applied to specific layers, and it can be done by constructing M in a particular way. We demonstrate this approach in Supplement B and show that mixout for specific layers adaptively L 2 -penalizes their parameters. Hoffer et al. (2017) have empirically shown that w t -w 0 ∼ log t, (10) where w t is a model parameter after the t-th SGD step. When training from scratch, we usually sample an initial model parameter w 0 from a normal/uniform distribution with mean 0 and small variance. Since w 0 is close to the origin, w t is away from the origin only with a large t by equation 10. When finetuning, we initialize our model parameter from a pretrained model parameter w pre . Since we usually obtain w pre by training from scratch on a large pretraining dataset, w pre is often far away from the origin. By Corollary 1.1, dropout L 2 -penalizes the model parameter for deviating away from the origin rather than w pre . To explicitly prevent the deviation from w pre , we instead propose to use mixout(w pre ). Wiese et al. (2017) have highlighted that wdecay(w pre ) is an effective regularization technique to avoid catastrophic forgetting during finetuning. Because mixout(w pre ) keeps the finetuned model to stay in the vicinity of the pretrained model similarly to wdecay(w pre ), we suspect that the proposed mixout(w pre ) has a similar effect of alleviating the issue of catastrophic forgetting. To empirically verify this claim, we pretrain a 784-300-100-10 fully-connected network on EMNIST Digits (Cohen et al., 2017) , and finetune it on MNIST. For more detailed description of the model architecture and datasets, see Supplement C.1.

3.2. MIXOUT FOR PRETRAINED MODELS:



4. VERIFICATION OF THEORETICAL RESULTS FOR MIXOUT ON MNIST:

In the pretraining stage, we run five random experiments with a batch size of 32 for {1, 2, • • • , 20} training epochs. We use Adam (Kingma & Ba, 2014) with a learning rate of 10 -4 , β 1 = 0.9, β 2 = 0.999, wdecay(0, 0.01), learning rate warm-up over the first 10% steps of the total steps, and linear decay of the learning rate after the warm-up. We use dropout(0.1) for all layers except the input and output layers. We select w pre whose validation accuracy on EMNIST Digits is best (0.992) in all experiments.

For finetuning, most of the model hyperparameters are kept same as in pretraining, with the exception of the learning rate, number of training epochs, and regularization techniques. We train with a learning rate of 5 × 10 -5 for 5 training epochs. We replace dropout(p) with mixout(w pre , p). We do not use any other regularization technique such as wdecay(0) and wdecay(w pre ). We monitor w ft -w pre2 ,1 validation accuracy on MNIST, and validation accuracy on EMNIST Digits to compare mixout(w pre , p) to dropout(p) across 10 random restarts. , validation accuracy on MNIST (target task), and validation accuracy on EMNIST Digits (source task), as the function of the probability p where w ft and w pre are the model parameter after finetuning and the pretrained model parameter, respectively. We report mean (curve) ± std. (shaded area) across 10 random restarts. (a): mixout(w pre , p) L 2 -penalizes the deviation from w pre , and this penalty becomes strong as p increases. However, with dropout(p), w ft becomes away from w pre as p increases. (b): After finetuning on MNIST, both mixout(w pre , p) and dropout(p) result in high validation accuracy on MNIST for p ∈ {0.1, 0.2, 0.3}. (c): Validation accuracy of dropout(p) on EMNIST Digits drops more than that of mixout(w pre , p) for all p. mixout(w pre , p) minimizes the deviation from w pre and memorizes the source task better than dropout(p) for all p.

As shown in Figure 2 (a), after finetuning with mixout(w pre , p), the deviation from w pre is minimized in the L 2 -sense. This result verifies Corollary 1.1. We demonstrate that the validation accuracy of mixout(w pre , p) has greater robustness to the choice of p than that of dropout(p). In Figure 2 (b), both dropout(p) and mixout(w pre , p) result in high validation accuracy on the target task (MNIST) for p ∈ {0.1, 0.2, 0.3}, although mixout(w pre , p) is much more robust with respect to the choice of the mix probability p. In Figure 2 (c), the validation accuracy of mixout(w pre , p) on the source task (EMNIST Digits) drops from the validation accuracy of the model at w pre (0.992) to approximately 0.723 regardless of p. On the other hand, the validation accuracy of dropout(p) on the source task respectively drops by 0.041, 0.074 and 0.105 which are more than those of mixout(w pre , p) for p ∈ {0.1, 0.2, 0.3}.

5. FINETUNING A PRETRAINED LANGUAGE MODEL WITH MIXOUT:

In order to experimentally validate the effectiveness of mixout, we finetune BERT LARGE on a subset of GLUE (Wang et al., 2018) tasks (RTE, MRPC, CoLA, and STS-B) with mixout(w pre ). We choose them because Phang et al. (2018) have observed that it was unstable to finetune BERT LARGE on these four tasks. We use the publicly available pretrained model released by Devlin et al. (2018) , ported into PyTorch by HuggingFace. 3 We use the learning setup and hyperparameters recommended by Devlin et al. (2018) . We use Adam with a learning rate of 2 × 10 -5 , β 1 = 0.9, β 2 = 0.999, learning rate warmup over the first 10% steps of the total steps, and linear decay of the learning rate after the warmup finishes. We train with a batch size of 32 for 3 training epochs. Since the pretrained BERT LARGE is the sentence encoder, we have to create an additional output layer, which is not pretrained. We initialize each parameter of it with N (0, 0.02 2 ). We describe our experimental setup further in Supplement C.2.

The original regularization strategy used in Devlin et al. (2018) for finetuning BERT LARGE is using both dropout(0.1) and wdecay(0, 0.01) for all layers except layer normalization and intermediate layers activated by GELU (Hendrycks & Gimpel, 2016) . We however cannot use mixout(w pre ) nor wdecay(w pre ) for the additional output layer which was not pretrained and therefore does not have w pre . We do not use any regularization for the additional output layer when finetuning BERT LARGE with mixout(w pre ) and wdecay(w pre ). For the other layers, we replace dropout(0.1) and wdecay(0, 0.01) with mixout(w pre ) and wdecay(w pre ), respectively. Phang et al. (2018) have reported that large pretrained models (e.g., BERT LARGE ) are prone to degenerate performance when finetuned on a task with a small number of training examples, and that multiple random restarts4 are required to obtain a usable model better than random prediction. To compare finetuning stability of the regularization techniques, we need to demonstrate the distribution of model performance. We therefore train BERT LARGE with each regularization strategy on each task with 20 random restarts. We validate each random restart on the dev set to observe the behaviour of the proposed mixout and finally evaluate it on the test set for generalization. We present the test score of our proposed regularization strategy on each task in Supplement C.3.

We finetune BERT LARGE with mixout(w pre , {0.7, 0.8, 0.9}) on RTE, MRPC, CoLA, and STS-B. For the baselines, we finetune BERT LARGE with both dropout(0.1) and wdecay(0, 0.01) as well as with wdecay(w pre , {0.01, 0.04, 0.07, 0.10}). These choices are made based on the experiments in Section 6.3 and Supplement F. In Section 6.3, we observe that finetuning BERT LARGE with mixout(w pre , p) on RTE is significantly more stable with p ∈ {0.7, 0.8, 0.9} while finetuning with dropout(p) becomes unstable as p increases. In Supplement F, we demonstrate that dropout(0.1) is almost optimal for all the tasks in terms of mean dev score although Devlin et al. (2018) selected it to improve the maximum dev score.

In Figure 3 , we plot the distributions of the dev scores from 20 random restarts when finetuning BERT LARGE with various regularization strategies on each task. For conciseness, we only show four regularization strategies; Devlin et al. (2018) 's: both dropout(0.1) and wdecay(0, 0.01), Wiese et al. (2017) 's: wdecay(w pre , 0.01), ours: mixout(w pre , 0.7), and ours+Wiese et al.

(2017)'s: both mixout(w pre , 0.7) and wdecay(w pre , 0.01). As shown in Figure 3 (a-c ), we observe many finetuning runs that fail with the chance-level accuracy when we finetune BERT LARGE with both dropout(0.1) and wdecay(0, 0.01) on RTE, MRPC, and CoLA. We also have a bunch of degenerate model configurations when we use wdecay(w pre , 0.01) without mixout(w pre , 0.7).

Unlike existing regularization strategies, when we use mixout(w pre , 0.7) as a regularization technique with or without wdecay(w pre , 0.01) for finetuning BERT LARGE , the number of degenerate model configurations that fail with a chance-level accuracy significantly decreases. For example, in Figure 3 2017)'s: both mixout(w pre , 0.7) and wdecay(w pre , 0.01). We write them as Devlin (blue), Wiese (orange), Our (green), and Our+W (red), respectively. We use the same set of 20 random initializations across all the regularization setups. Error intervals show mean±std. For all the tasks, the number of finetuning runs that fail with the chance-level accuracy is significantly reduced when we use our regularization mixout(w pre , 0.7) regardless of using wdecay(w pre , 0.01).

In Figure 3 (a), we further improve the stability of finetuning BERT LARGE by using both mixout(w pre , 0.7) and wdecay(w pre , 0.01). 2017)'s. In Figure 3 (b, c ), we observe that the number of degenerate model configurations increases when we use wdecay(w pre , 0.01) additionally to mixout(w pre , 0.7). In short, applying our proposed mixout significantly stabilizes the finetuning results of BERT LARGE on small training sets regardless of whether we use wdecay(w pre , 0.01).

In Table 1 , we report the average and the best dev scores across 20 random restarts for each task with various regularization strategies. The average dev scores with mixout(w pre , {0.7, 0.8, 0.9}) increase for all the tasks. For instance, the mean dev score of finetuning with mixout(w pre , 0.8) on CoLA is 57.9 which is 49.2% increase over 38.8 obtained by finetuning with both dropout(p) and wdecay(0, 0.01). We observe that using wdecay(w pre , {0.01, 0.04, 0.07, 0.10}) also improves the average dev scores for most tasks compared to using both dropout(p) and wdecay(0, 0.01). We however observe that finetuning with mixout(w pre , {0.7, 0.8, 0.9}) outperforms that with wdecay(w pre , {0.01, 0.04, 0.07, 0.10}) on average. This confirms that mixout(w pre ) has a different effect for finetuning BERT LARGE compared to wdecay(w pre ) since mixout(w pre ) is an adaptive L 2 -regularizer along the optimization trajectory.

Since finetuning a large pretrained language model such as BERT LARGE on a small training set frequently fails, the final model performance has often been reported as the maximum dev score (Devlin et al., 2018; Phang et al., 2018) among a few random restarts. We thus report the best dev score for each setting in Table 1 . According to the best dev scores as well, mixout(w pre , {0.7, 0.8, 0.9}) improves performance for all the tasks compared to using both dropout(p) and wdecay(0, 0.01). For instance, using mixout(w pre , 0.9) improves the maximum dev score by 0.9 compared to using both dropout(p) and wdecay(0, 0.01) on MRPC. Unlike the average dev scores, the best dev scores achieved by using wdecay(w pre , {0.01, 0.04, 0.07, 0.10}) are better than those achieved by using mixout(w pre , {0.7, 0.8, 0.9}) except RTE on which it was better to use mixout(w pre , {0.7, 0.8, 0.9}) than wdecay(w pre , {0.01, 0.04, 0.07, 0.10}).

Table 1 : Mean (max) dev scores across 20 random restarts when finetuning BERT LARGE with various regularization strategies on each task. We show the following baseline results on the first and second cells: Devlin et al. ( 2018)'s regularization strategy (both dropout(p) and wdecay(0, 0.01)) and Wiese et al. ( 2017)'s regularization strategy (wdecay(w pre , {0.01, 0.04, 0.07, 0.10})). In the third cell, we demonstrate finetuning results with only mixout(w pre , {0.7, 0.8, 0.9}). The results with both mixout(w pre , {0.7, 0.8, 0.9}) and wdecay(w pre , 0.01) are also presented in the fourth cell. Bold marks the best of each statistics within each column. The mean dev scores greatly increase for all the tasks when we use mixout(w pre , {0.7, 0.8, 0.9}). We investigate the effect of combining both mixout(w pre ) and wdecay(w pre ) to see whether they are complementary. We finetune BERT LARGE with both mixout(w pre , {0.7, 0.8, 0.9}) and wdecay(w pre , 0.01). This leads not only to the improvement in the average dev scores but also in the best dev scores compared to using wdecay(w pre , {0.01, 0.04, 0.07, 0.10}) and using both dropout(p) and wdecay(0, 0.01). The experiments in this section confirm that using mixout(w pre ) as one of several regularization techniques prevents finetuning instability and yields gains in dev scores.

TECHNIQUE:



6. ABLATION STUDY:

In this section, we perform ablation experiments to better understand mixout(w pre ). Unless explicitly stated, all experimental setups are the same as in Section 5.

6.1. MIXOUT WITH A SUFFICIENT NUMBER OF TRAINING EXAMPLES:

We showed the effectiveness of the proposed mixout finetuning with only a few training examples in Section 5. In this section, we investigate the effectiveness of the proposed mixout in the case of a larger finetuning set. Since it has been stable to finetune BERT LARGE on a sufficient number of training examples (Devlin et al., 2018; Phang et al., 2018) , we expect to see the change in the behaviour of mixout(w pre ) when we use it to finetune BERT LARGE on a larger training set.

We train BERT LARGE by using both mixout(w pre , 0.7) and wdecay(w pre , 0.01) with 20 random restarts on SST-2. 5 We also train BERT LARGE by using both dropout(p) and wdecay(0, 0.01) with 20 random restarts on SST-2 as the baseline. In Table 2 , we report the mean and maximum of SST-2 dev scores across 20 random restarts with each regularization strategy. We observe that there is little difference between their mean and maximum dev scores on a larger training set, although using both mixout(w pre , 0.7) and wdecay(w pre , 0.01) outperformed using both dropout(p) and wdecay(0, 0.01) on the small training sets in Section 5.

Table 2 : Mean (max) SST-2 dev scores across 20 random restarts when finetuning BERT LARGE with each regularization strategy. Bold marks the best of each statistics within each column. For a large training set, both mean and maximum dev scores are similar to each other.

TECHNIQUE 1:

TECHNIQUE 2 SST-2 dropout(0.1) wdecay(0, 0.01) 93.4 (94.0) mixout(w pre , 0.7) wdecay(w pre , 0.01) 93.5 (94.3)

6.2. EFFECT OF A REGULARIZATION TECHNIQUE FOR AN ADDITIONAL OUTPUT LAYER:

In this section, we explore the effect of a regularization technique for an additional output layer. There are two regularization techniques available for the additional output layer: dropout(p) and mixout(w 0 , p) where w 0 is its randomly initialized parameter. Either of these strategies differs from the earlier experiments in Section 5 where we did not put any regularization for the additional output layer.

Table 3 : We present mean (max) dev scores across 20 random restarts with various regularization techniques for the additional output layers (ADDITIONAL) when finetuning BERT LARGE on each task. For all cases, we apply mixout(w pre , 0.7) to the pretrained layers (PRETRAINED). The first row corresponds to the setup in Section 5. In the second row, we apply mixout(w 0 , 0.7) to the additional output layer where w 0 is its randomly initialized parameter. The third row shows the results obtained by applying dropout(0.7) to the additional output layer. In the fourth row, we demonstrate the best of each result from all the regularization strategies shown in Table 1 . Bold marks the best of each statistics within each column. We obtain additional gains in dev scores by varying the regularization technique for the additional output layer.

PRETRAINED:

ADDITIONAL RTE MRPC CoLA STS-B mixout(w pre , 0.7) -61.6 (74.0) 87.1 (91.1) 57.4 (62.1) 89.6 (90.3) mixout(w pre , 0.7) mixout(w 0 , 0.7) 66.5 (75.5) 88.1 (92.4) 58.7 (65.6) 89.7 (90.6) mixout(w pre , 0.7) dropout(0.7) 57.2 (70.8) 85.9 (92.5) 48.9 (64.3) 89.2 (89.8) The best of each result from Table 1 65.3 (75.5) 89.0 (91.8) 58.3 (65.6) 89.7 (90.6)

We report the average and best dev scores across 20 random restarts when finetuning BERT LARGE with mixout(w pre , 0.7) while varying the regularization technique for the additional output layer in Table 3 . 6 We observe that using mixout(w 0 , 0.7) for the additional output layer improves both the average and best dev score on RTE, CoLA, and STS-B. In the case of MRPC, we have the highest best-dev score by using dropout(0.7) for the additional output layer while the highest mean dev score is obtained by using mixout(w 0 , 0.7) for it. In Section 3.2, we discussed how mixout(w 0 ) does not differ from dropout when the layer is randomly initialized, since we sample w 0 from w whose mean and variance are 0 and small, respectively. Although the additional output layer is randomly initialized, we observe the significant difference between dropout and mixout(w 0 ) in this layer. We conjecture that w 0 -0 is not sufficiently small because E w -0 is proportional to the dimensionality of the layer (2,048). We therefore expect mixout(w 0 ) to behave differently from dropout even for the case of training from scratch.

In the last row of Table 3 , we present the best of the corresponding result from Table 1 . We have the highest mean and best dev scores when we respectively use mixout(w pre , 0.7) and mixout(w 0 , 0.7) for the pretrained layers and the additional output layer on RTE, CoLA, and STS-B. The highest mean dev score on MRPC is obtained by using mixout(w pre , 0.8) for the pretrained layers which is one of the results in Table 1 . We have the highest best dev score on MRPC when we use mixout(w pre , 0.7) and dropout(0.7) for the pretrained layers and the additional output layer, respectively. The experiments in this section reveal that using mixout(w 0 ) for a randomly initialized layer of a pretrained model is one of the regularization schemes to improve the average dev score and the best dev score.

6.3. EFFECT OF MIX PROBABILITY FOR MIXOUT AND DROPOUT:

We explore the effect of the hyperparameter p when finetuning BERT LARGE with mixout(w pre , p) and dropout(p). We train BERT LARGE with mixout(w pre , {0.0, 0.1, • • • , 0.9}) on RTE with 20 random restarts. We also train BERT LARGE after replacing mixout(w pre , p) by dropout(p) with 20 random restarts. We do not use any regularization technique for the additional output layer. Because we use neither wdecay(0) nor wdecay(w pre ) in this section, dropout(0.0) and mixout(w pre , 0.0) are equivalent to finetuning without regularization.

0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Error intervals show mean±std. We do not use wdecay(0) nor wdecay(w pre ). In the case of mixout(w pre , p), the number of usable models after finetuning with mixout(w pre , {0.7, 0.8, 0.9}) is significantly more than the number of usable models after finetuning with dropout(p) for all p.

It is not helpful to vary p for dropout(p) while mixout(w pre , p) helps significantly in a wide range of p. Figure 4 shows distributions of RTE dev scores across 20 random restarts when finetuning BERT LARGE with dropout(p) and mixout(w pre , p) for p ∈ {0.0, 0.1, • • • , 0.9}. The mean dev score of finetuning BERT LARGE with mixout(w pre , p) increases as p increases. On the other hand, the mean dev score of finetuning BERT LARGE with dropout(p) decreases as p increases. If p is less than 0.4, finetuning with mixout(w pre , p) does not improve the finetuning results of using dropout({0.0, 0.1, 0.2}). We however observe that mixout(w pre , {0.7, 0.8, 0.9}) yields better average dev scores than dropout(p) for all p, and significantly reduces the number of finetuning runs that fail with the chance-level accuracy.

We notice that the proposed mixout spends more time than dropout from the experiments in this section. It takes longer to finetune a model with the proposed mixout than with the original dropout, although this increase is not significant especially considering the waste of time from failed finetuning runs using dropout. In Supplement E, we describe more in detail the difference between mixout and dropout in terms of wall-clock time.

7. CONCLUSION:

The special case of our approach, mixout(w pre ), is one of several regularization techniques modifying a finetuning procedure to prevent catastrophic forgetting. Unlike wdecay(w pre ) proposed earlier by Wiese et al. (2017) , mixout(w pre ) is an adaptive L 2 -regularizer toward w pre in the sense that its regularization coefficient adapts along the optimization path. Due to this difference, the proposed mixout improves the stability of finetuning a big, pretrained language model even with only a few training examples of a target task. Furthermore, our experiments have revealed the proposed approach improves finetuning results in terms of the average accuracy and the best accuracy over multiple runs. We emphasize that our approach can be used with any pretrained language models such as RoBERTa (Liu et al., 2019) and XLNet (Yang et al., 2019) , since mixout does not depend on model architectures, and leave it as future work.

SUPPLEMENTARY MATERIAL:

A PROOFS FOR THEOREM 1

Theorem 1. Assume that the loss function L is strongly convex. Suppose that a random mixture function with respect to w from u and M is given by

Φ(w; u, M ) = µ -1 (I -M )u + M w -(1 -µ)u ,

where

M is diag(M 1 , M 2 , • • • , M d ) satisfying EM i = µ

and Var(M i ) = σ 2 for all i. Then, there exists m > 0 such that

EQUATION

for all w.

Proof. Since L is strongly convex, there exist m > 0 such that

EQUATION

)

for all w by equation 1. Recall that EM i = µ and Var(M i ) = σ 2 for all i. Then, we have

EQUATION

and

EQUATION

)

By using equation 13 and equation 14, we can rewrite equation 12 as

EL Φ(w; u, M ) ≥ L(w) + mσ 2 2µ 2 w -u 2 .

B APPLYING TO SPECIFIC LAYERS:

We often apply dropout to specific layers. For instance, Simonyan & Zisserman (2014) applied dropout to fully connected layers only. We generalize Theorem 1 to the case in which mixconnect is only applied to specific layers, and it can be done by constructing M in a particular way. To better characterize mixconnect applied to specific layers, we define the index set I as I = {i : M i = 1}. Furthermore, we use w and ũ to denote (w i ) i / ∈I and (u i ) i / ∈I , respectively. Then, we generalize equation 7 to

EQUATION

)

From equation 15, applying mixconnect(u, µ, σ 2 ) is to use adaptive wdecay( ũ) on the weight parameter of the specific layers w. Similarly, we can regard applying mixout(u, p) to specific layers as adaptive wdecay( ũ).

C.3 TEST RESULTS ON GLUE TASKS:

We expect that using mixout stabilizes finetuning results of BERT LARGE on a small training set.

To show this, we demonstrated distributions of dev scores from 20 random restarts on RTE, MRPC, CoLA, and STS-B in Figure 3 . We further obtained the highest average/best dev score on each task in Table 3 . To confirm the generalization of the our best model on the dev set, we demonstrate the test results scored by the evaluation server9 in Table 4 .

Table 4 : We present the test score when finetuning BERT LARGE with each regularization strategy on each task. The first row shows the test scores obtained by using both dropout(p) and wdecay(0, 0.01). These results in the first row are reported by Devlin et al. (2018) . They used the learning rate of {2 × 10 -5 , 3 × 10 -5 , 4 × 10 -5 , 5 × 10 -5 } and a batch size of 32 for 3 epochs with multiple random restarts. They selected the best model on each dev set. In the second row, we demonstrate the test scores obtained by using the proposed mixout in Section 6.2: using mixout(w pre , 0.7) for the pretrained layers and mixout(w 0 , 0.7) for the additional output layer where w 0 is its randomly initialized weight parameter. We used the learning rate of 2 × 10 -5 and a batch size of 32 for 3 epochs with 20 random restarts. We submitted the best model on each dev set. For all the tasks except MRPC, the test scores obtained by the proposed mixout10 are better than those reported by Devlin et al. (2018) . We explored the behaviour of finetuning BERT LARGE with mixout by using the learning rate of 2 × 10 -5 while Devlin et al. (2018) obtained their results by using the learning rate of {2 × 10 -5 , 3 × 10 -5 , 4 × 10 -5 , 5 × 10 -5 }. We thus present the test scores obtained by the regularization strategy of Devlin et al. (2018) when the learning rate is 2 × 10 -5 . The results in this section show that the best model on the dev set generalizes well, and all the experiments based on dev scores in this paper are proper to validate the effectiveness of the proposed mixout. For the remaining GLUE tasks such as SST-2 with a sufficient number of training instances, we observed that using mixout does not differs from using dropout in Section 6.1. We therefore omit the test results on the other tasks in GLUE.

D VERIFICATION OF COROLLARY 1.1 WITH LEAST SQUARES REGRESSION:

Corollary 1.1 shows that mixout(u, p) regularizes learning to minimize the deviation from the target model parameter u, and the strength of regularization increases as p increases when the loss function is strongly convex. In order to validate this, we explore the behavior of least squares regression with mixout(u, p) on a synthetic dataset. For randomly given w * 1 and w * 2 , we generated an observation y satisfying y = w * 1 x + w * 2 + where is Gaussian noise. We set the model to ŷ = w 1 x + w 2 . That is, the model parameter w is given by (w 1 , w 2 ). We randomly pick u as a target model parameter for mixout(u, p) and perform least squares regression with mixout(u, {0.0, 0.3, 0.6, 0.9}). As shown in Figure 5 , w converges to the target model parameter u rather than the true model parameter w * = (w * 1 , w * 2 ) as the mix probability p increases. 

E TIME USAGE OF MIXOUT COMPARED TO DROPOUT:

We recorded the training time of the experiment in Section 6.3 to compare the time usage of mixout and that of dropout. It took about 843 seconds to finetune BERT LARGE with mixout(w pre ). On the other hand, it took about 636 seconds to finetune BERT LARGE with dropout. mixout(w pre ) spends 32.5% more time than dropout since mixout(w pre ) needs an additional computation with the pretrained model parameter w pre . However, as shown in Figure 4 , at least 15 finetuning runs among 20 random restarts fail with the chance-level accuracy on RTE with dropout(p) for all p while only 4 finetuning runs out of 20 random restarts are unusable with mixout(w pre , 0.8). From this result, it is reasonable to finetune with the proposed mixout although this requires additional time usage compared to dropout.

F EXTENSIVE HYPERPARAMETER SEARCH FOR DROPOUT Devlin et al. (2018) finetuned BERT LARGE with dropout(0.1) on all GLUE (Wang et al., 2018) tasks. They chose it to improve the maximum dev score on each downstream task, but we have reported not only the maximum dev score but also the mean dev score to quantitatively compare various regularization techniques in our paper. In this section, we explore the effect of the hyperparameter p when finetuning BERT LARGE with dropout(p) on RTE, MRPC, CoLA, and STS-B to show dropout(0.1) is optimal in terms of mean dev score. All experimental setups for these experiments are the same as Section 6.3. As shown in Figure 6 , we have the highest average dev score on MRPC with dropout(0.1) as well as on STS-B. We obtain the highest average dev scores with dropout(0.0) on RTE and CoLA, but we get the second-highest average dev scores with dropout(0.1) on them. These experiments confirm that the drop probability 0.1 is almost optimal for the highest average dev score on each task.

:

wft is a model parameter after finetuning.

Using the same pretrained model parameter wpre but perform different finetuning data shuffling.

https : / / s3 . amazonaws . com / models . huggingface . co / bert / bert-large-uncased-pytorch_model.bin

Using the same pretrained model parameter wpre but each random restart differs from the others by shuffling target data and initializing the additional output layer differently.

For the description of SST-2, see Supplement C.2.

In this experiment, we use neither wdecay(0) nor wdecay(wpre).

https : / / s3 . amazonaws . com / models . huggingface . co / bert / bert-large-uncased-pytorch_model.bin

https://github.com/huggingface/pytorch-transformers

https://gluebenchmark.com/leaderboard

The regularization strategy in Section 6.2: using mixout(wpre, 0.7) for the pretrained layers and mixout(w0, 0.7) for the additional output where w0 is its randomly initialized weight parameter.

Figures and Tables:

Figure 1: Illustration of mixout(u). Suppose that u and w are a target model parameter and a current model parameter, respectively. (a): We first memorize the parameters of the vanilla network at u. (b):In the dropout network, we randomly choose an input neuron to be dropped (a dotted neuron) with a probability of p. That is, all outgoing parameters from the dropped neuron are eliminated (dotted connections). (c): In the mixout(u) network, the eliminated parameters in (b) are replaced by the corresponding parameters in (a). In other words, the mixout(u) network at w is the mixture of the vanilla network at u and the dropout network at w with a probability of p.

Figure2: We present w ft -w pre 2 , validation accuracy on MNIST (target task), and validation accuracy on EMNIST Digits (source task), as the function of the probability p where w ft and w pre are the model parameter after finetuning and the pretrained model parameter, respectively. We report mean (curve) ± std. (shaded area) across 10 random restarts. (a): mixout(w pre , p) L 2 -penalizes the deviation from w pre , and this penalty becomes strong as p increases. However, with dropout(p), w ft becomes away from w pre as p increases. (b): After finetuning on MNIST, both mixout(w pre , p) and dropout(p) result in high validation accuracy on MNIST for p ∈ {0.1, 0.2, 0.3}. (c): Validation accuracy of dropout(p) on EMNIST Digits drops more than that of mixout(w pre , p) for all p. mixout(w pre , p) minimizes the deviation from w pre and memorizes the source task better than dropout(p) for all p.

Figure 3: Distribution of dev scores on each task from 20 random restarts when finetuning BERT LARGE with Devlin et al. (2018)'s: both dropout(0.1) and wdecay(0, 0.01), Wiese et al. (2017)'s: wdecay(w pre , 0.01), ours: mixout(w pre , 0.7), and ours+Wiese et al. (2017)'s: both mixout(w pre , 0.7) and wdecay(w pre , 0.01). We write them as Devlin (blue), Wiese (orange), Our (green), and Our+W (red), respectively. We use the same set of 20 random initializations across all the regularization setups. Error intervals show mean±std. For all the tasks, the number of finetuning runs that fail with the chance-level accuracy is significantly reduced when we use our regularization mixout(w pre , 0.7) regardless of using wdecay(w pre , 0.01).

Figure 3 (d) shows respectively two and one degenerate model configurations with Devlin et al. (2018)'s and Wiese et al. (2017)'s, but we do not have any degenerate resulting model with ours and ours+Wiese et al. (

Figure4: Distribution of RTE dev scores (Accuracy) from 20 random restarts when finetuning BERT LARGE with dropout(p) (orange) or mixout(w pre , p) (blue). Error intervals show mean±std. We do not use wdecay(0) nor wdecay(w pre ). In the case of mixout(w pre , p), the number of usable models after finetuning with mixout(w pre , {0.7, 0.8, 0.9}) is significantly more than the number of usable models after finetuning with dropout(p) for all p.

Figure5: Behavior of mixout(u, p) for a strongly convex loss function. We plot the line obtained by least squares regression with mixout(u, {0.0, 0.3, 0.6, 0.9}) (each green line) on a synthetic dataset (blue dots) generated by the true line (each blue dotted line). As p increases, the regression line (each green line) converges to the target line generated by the target model parameter u (each orange dotted line) rather than the true line (each blue dotted line).

Figure6: Distribution of dev scores on each task from 20 random restarts when finetuning BERT LARGE with dropout({0.0, 0.1, • • • , 0.5}). Error intervals show mean±std. When we use dropout(0.1), we have the highest average dev scores on MRPC and STS-B and the second-highest average dev scores on RTE and CoLA. These results show that dropout(0.1) is almost optimal for all tasks in terms of mean dev score.

Wiese et al. (2017),Kirkpatrick et al. (2017), andSchwarz et al. (2018) used L 2 -penalty toward a pretrained model parameter to improve performance. They focused on preventing catastrophic forgetting to enable their models to learn multiple tasks sequentially. They however do not discuss nor demonstrate the effect of L 2 -penalty toward the pretrained model parameter on the stability of finetuning.Barone et al. (2017) introduced tuneout, which is a special case of mixout. They applied various regularization techniques including dropout, tuneout, and L 2 -penalty toward a pretrained model parameter to finetune neural machine translation. They however do not demonstrate empirical significance of tuneout compared to other regularization techniques nor its theoretical justification.

Table content: <table/>

The third row shows that the test scores obtained by using both dropout(p) and wdecay(0, 0.01) with same experimental setups of the second row. Bold marks the best within each column. The proposed mixout improves the test scores except MRPC compared to the original regularization strategy proposed byDevlin et al. (2018).

Table content: <table><tr><td>STRATEGY</td><td colspan="4">RTE MRPC CoLA STS-B</td></tr><tr><td>Devlin et al. (2018)</td><td>70.1</td><td>89.3</td><td>60.5</td><td>86.5</td></tr><tr><td colspan="2">mixout(w pre , 0.7) &amp; mixout(w 0 , 0.7) 70.2</td><td>89.1</td><td>62.1</td><td>87.3</td></tr><tr><td>dropout(p) + wdecay(0, 0.01)</td><td>68.2</td><td>88.3</td><td>59.6</td><td>86.0</td></tr></table>

Peer review (iclr2020):
This paper introduces a new regularization technique refered as “mixout”, motivated by dropout. Mixout stochastically mixes the parameters of two models. Experiments shows the stability of finetuning and the method greatly improve the average accuracy. I really like the proposed idea, and the paper is easy to understand and follow, and the experiments are well designed. The time usage of the regularization is not discussed. It seems the method needs to maintain two copies of parameters, it would be much better if the author can provide the time usage of the experiments.