Mixout: Effective Regularization to Finetune Large-scale Pretrained Language Models

1 INTRODUCTION . Transfer learning has been widely used for the tasks in natural language processing ( NLP ) ( Collobert et al. , 2011 ; Devlin et al. , 2018 ; Yang et al. , 2019 ; Liu et al. , 2019 ; Phang et al. , 2018 ) . In particular , Devlin et al . ( 2018 ) recently demonstrated the effectiveness of finetuning a large-scale language model pretrained on a large , unannotated corpus on a wide range of NLP tasks including question answering and language inference . They have designed two variants of models , BERTLARGE ( 340M parameters ) and BERTBASE ( 110M parameters ) . Although BERTLARGE outperforms BERTBASE generally , it was observed that finetuning sometimes fails when a target dataset has fewer than 10,000 training instances ( Devlin et al. , 2018 ; Phang et al. , 2018 ) . When finetuning a big , pretrained language model , dropout ( Srivastava et al. , 2014 ) has been used as a regularization technique to prevent co-adaptation of neurons ( Vaswani et al. , 2017 ; Devlin et al. , 2018 ; Yang et al. , 2019 ) . We provide a theoretical understanding of dropout and its variants , such as Gaussian dropout ( Wang & Manning , 2013 ) , variational dropout ( Kingma et al. , 2015 ) , and dropconnect ( Wan et al. , 2013 ) , as an adaptive L2-penalty toward the origin ( all zero parameters 0 ) and generalize dropout by considering a target model parameter u ( instead of the origin ) , to which we refer as mixout ( u ) . We illustrate mixout ( u ) in Figure 1 . To be specific , mixout ( u ) replaces all outgoing parameters from a randomly selected neuron to the corresponding parameters of u. mixout ( u ) avoids optimization from diverging away from u through an adaptive L2-penalty toward u . Unlike mixout ( u ) , dropout encourages a move toward the origin which deviates away from u since dropout is equivalent to mixout ( 0 ) . We conduct experiments empirically validating the effectiveness of the proposed mixout ( wpre ) where wpre denotes a pretrained model parameter . To validate our theoretical findings , we train a fully connected network on EMNIST Digits ( Cohen et al. , 2017 ) and finetune it on MNIST . We observe that a finetuning solution of mixout ( wpre ) deviates less from wpre in the L2-sense than ∗Department of Mathematical Sciences , KAIST , Daejeon , 34141 , Republic of Korea †New York University ‡Facebook AI Research §CIFAR Azrieli Global Scholar that of dropout . In the main experiment , we finetune BERTLARGE with mixout ( wpre ) on small training sets of GLUE ( Wang et al. , 2018 ) . We observe that mixout ( wpre ) reduces the number of unusable models that fail with the chance-level accuracy and increases the average development ( dev ) scores for all tasks . In the ablation studies , we perform the following three experiments for finetuning BERTLARGE with mixout ( wpre ) : ( i ) the effect of mixout ( wpre ) on a sufficient number of training examples , ( ii ) the effect of a regularization technique for an additional output layer which is not pretrained , and ( iii ) the effect of probability of mixout ( wpre ) compared to dropout . From these ablation studies , we observe that three characteristics of mixout ( wpre ) : ( i ) finetuning with mixout ( wpre ) does not harm model performance even with a sufficient number of training examples ; ( ii ) It is beneficial to use a variant of mixout as a regularization technique for the additional output layer ; ( iii ) The proposed mixout ( wpre ) is helpful to the average dev score and to the finetuning stability in a wider range of its hyperparameter p than dropout . 1.1 RELATED WORK . For large-scale pretrained language models ( Vaswani et al. , 2017 ; Devlin et al. , 2018 ; Yang et al. , 2019 ) , dropout has been used as one of several regularization techniques . The theoretical analysis for dropout as an L2-regularizer toward 0 was explored by Wan et al . ( 2013 ) where 0 is the origin . They provided a sharp characterization of dropout for a simplified setting ( generalized linear model ) . Mianjy & Arora ( 2019 ) gave a formal and complete characterization of dropout in deep linear networks with squared loss as a nuclear norm regularization toward 0 . However , neither Wan et al . ( 2013 ) nor Mianjy & Arora ( 2019 ) gives theoretical analysis for the extension of dropout which uses a point other than 0 . Wiese et al . ( 2017 ) , Kirkpatrick et al . ( 2017 ) , and Schwarz et al . ( 2018 ) used L2-penalty toward a pretrained model parameter to improve performance . They focused on preventing catastrophic forgetting to enable their models to learn multiple tasks sequentially . They however do not discuss nor demonstrate the effect of L2-penalty toward the pretrained model parameter on the stability of finetuning . Barone et al . ( 2017 ) introduced tuneout , which is a special case of mixout . They applied various regularization techniques including dropout , tuneout , and L2-penalty toward a pretrained model parameter to finetune neural machine translation . They however do not demonstrate empirical significance of tuneout compared to other regularization techniques nor its theoretical justification . 2 PRELIMINARIES AND NOTATIONS . Norms and Loss Functions Unless explicitly stated , a norm ‖ · ‖ refers to L2-norm . A loss function of a neural network is written as L ( w ) = 1n ∑n i=1 Li ( w ) , where w is a trainable model parameter . Li is “ a per-example loss function ” computed on the i-th data point . Strong Convexity A differentiable function f is strongly convex if there exists m > 0 such that f ( y ) ≥ f ( x ) +∇f ( x ) > ( y − x ) + m 2 ‖y − x‖2 , ( 1 ) for all x and y . Weight Decay We refer as “ wdecay ( u , λ ) ” to minimizing L ( w ) + λ 2 ‖w − u‖2 , instead of the original loss function L ( w ) where λ is a regularization coefficient . Usual weight decay of λ is equivalent to wdecay ( 0 , λ ) . Probability for Dropout and Dropconnect Dropout ( Srivastava et al. , 2014 ) is a regularization technique selecting a neuron to drop with a probability of p. Dropconnect ( Wan et al. , 2013 ) chooses a parameter to drop with a probability of p. To emphasize their hyperparameter p , we write dropout and dropconnect with a drop probability of p as “ dropout ( p ) ” and “ dropconnect ( p ) ” , respectively . dropout ( p ) is a special case of dropconnect ( p ) if we simultaneously drop the parameters outgoing from each dropped neuron . Inverted Dropout and Dropconnect In the case of dropout ( p ) , a neuron is retained with a probability of 1−p during training . If we denote the weight parameter of that neuron as w during training , then we use ( 1 − p ) w for that weight parameter at test time ( Srivastava et al. , 2014 ) . This ensures that the expected output of a neuron is the same as the actual output at test time . In this paper , dropout ( p ) refers to inverted dropout ( p ) which uses w/ ( 1 − p ) instead of w during training . By doing so , we do not need to compute the output separately at test time . Similarly , dropconnect ( p ) refers to inverted dropconnect ( p ) . 3 ANALYSIS OF DROPOUT AND ITS GENERALIZATION . We start our theoretical analysis by investigating dropconnect which is a general form of dropout and then apply the result derived from dropconnect to dropout . The iterative SGD equation for dropconnect ( p ) with a learning rate of η is w ( t+1 ) = w ( t ) − ηB ( t ) ∇L ( ( EB ( t ) 1 ) −1 B ( t ) w ( t ) ) , t = 0 , 1 , 2 , · · · , ( 2 ) where B ( t ) = diag ( B ( t ) 1 , B ( t ) 2 , · · · , B ( t ) d ) and B ( t ) i ’ s are mutually independent Bernoulli ( 1− p ) random variables with a drop probability of p for all i and t. We regard equation 2 as finding a solution to the minimization problem below : min w EL ( ( EB1 ) −1Bw ) , ( 3 ) where B = diag ( B1 , B2 , · · · , Bd ) and Bi ’ s are mutually independent Bernoulli ( 1− p ) random variables with a drop probability of p for all i. Gaussian dropout ( Wang & Manning , 2013 ) and variational dropout ( Kingma et al. , 2015 ) use other random masks to improve dropout rather than Bernoulli random masks . To explain these variants of dropout as well , we set a random mask matrix M = diag ( M1 , M2 , · · · , Md ) to satisfy EMi = µ and Var ( Mi ) = σ2 for all i . Now we define a random mixture function with respect to w from u and M as Φ ( w ; u , M ) = µ−1 ( ( I −M ) u + Mw − ( 1− µ ) u ) , ( 4 ) and a minimization problem with “ mixconnect ( u , µ , σ2 ) ” as min w EL ( Φ ( w ; u , M ) ) . ( 5 ) We can view dropconnect ( p ) equation 3 as a special case of equation 5 where u = 0 and M = B . We investigate how mixconnect ( u , µ , σ2 ) differs from the vanilla minimization problem min w EL ( w ) . ( 6 ) If the loss function L is strongly convex , we can derive a lower bound of EL ( Φ ( w ; u , M ) ) as in Theorem 1 : Theorem 1 . Assume that the loss function L is strongly convex . Suppose that a random mixture function with respect to w from u and M is given by Φ ( w ; u , M ) in equation 4 where M is diag ( M1 , M2 , · · · , Md ) satisfying EMi = µ and Var ( Mi ) = σ2 for all i . Then , there exists m > 0 such that EL ( Φ ( w ; u , M ) ) ≥ L ( w ) + mσ 2 2µ2 ‖w − u‖2 , ( 7 ) for all w ( Proof in Supplement A ) . Theorem 1 shows that minimizing the l.h.s . of equation 7 minimizes the r.h.s . of equation 7 when the r.h.s . is a sharp lower limit of the l.h.s . The strong convexity of L means that L is bounded from below by a quadratic function , and the inequality of equation 7 comes from the strong convexity . Hence , the equality holds if L is quadratic , and mixconnect ( u , µ , σ2 ) is an L2-regularizer with a regularization coefficient of mσ2/µ2 .

Summary: This paper introduces a new regularization technique “mixout” for fine-tuning BERT. Mixout technique mixes the parameters of two models — the pretrained model and the dropout model. Because it keeps pretrained model parameters in consideration all the time, it effectively prevent catastrophic forgetting. Empirical results show that mixout can stabilize fine-tuning BERT on tasks with small training examples (which has been shown to be difficult).