Decoupled Weight Decay Regularization

1 INTRODUCTION . Adaptive gradient methods , such as AdaGrad ( Duchi et al. , 2011 ) , RMSProp ( Tieleman & Hinton , 2012 ) , Adam ( Kingma & Ba , 2014 ) and most recently AMSGrad ( Reddi et al. , 2018 ) have become a default method of choice for training feed-forward and recurrent neural networks ( Xu et al. , 2015 ; Radford et al. , 2015 ) . Nevertheless , state-of-the-art results for popular image classification datasets , such as CIFAR-10 and CIFAR-100 Krizhevsky ( 2009 ) , are still obtained by applying SGD with momentum ( Gastaldi , 2017 ; Cubuk et al. , 2018 ) . Furthermore , Wilson et al . ( 2017 ) suggested that adaptive gradient methods do not generalize as well as SGD with momentum when tested on a diverse set of deep learning tasks , such as image classification , character-level language modeling and constituency parsing . Different hypotheses about the origins of this worse generalization have been investigated , such as the presence of sharp local minima ( Keskar et al. , 2016 ; Dinh et al. , 2017 ) and inherent problems of adaptive gradient methods ( Wilson et al. , 2017 ) . In this paper , we investigate whether it is better to use L2 regularization or weight decay regularization to train deep neural networks with SGD and Adam . We show that a major factor of the poor generalization of the most popular adaptive gradient method , Adam , is due to the fact that L2 regularization is not nearly as effective for it as for SGD . Specifically , our analysis of Adam leads to the following observations : L2 regularization and weight decay are not identical . Contrary to a belief which seems popular among some practitioners , the two techniques are not equivalent . For SGD , they can be made equivalent by a reparameterization of the weight decay factor based on the learning rate ; this is not the case for Adam . In particular , when combined with adaptive gradients , L2 regularization leads to weights with large parameter and/or gradient amplitudes being regularized less than they would be when using weight decay . L2 regularization is not effective in Adam . One possible explanation why Adam and other adaptive gradient methods might be outperformed by SGD with momentum is that common deep learning libraries only implement L2 regularization , not the original weight decay . Therefore , on tasks/datasets where the use of L2 regularization is beneficial for SGD ( e.g. , on many popular image classification datasets ) , Adam leads to worse results than SGD with momentum ( for which L2 regularization behaves as expected ) . Weight decay is equally effective in both SGD and Adam . For SGD , it is equivalent to L2 regularization , while for Adam it is not . Optimal weight decay depends on the total number of batch passes/weight updates . Our empirical analysis of SGD and Adam suggests that the larger the runtime/number of batch passes to be performed , the smaller the optimal weight decay . This effect tends to be neglected because hyperparameters are often tuned for a fixed number of training epochs . As a result , the values of the weight decay found to perform best for short runs do not generalize to much longer runs . The main contribution of this paper is to improve regularization in Adam by decoupling the weight decay from the gradient-based update . In a comprehensive analysis , we show that Adam generalizes substantially better with decoupled weight decay than with L2 regularization , achieving 15 % relative improvement in test error ( see Figures 2 and 3 ) ; this holds true for various image recognition datasets ( CIFAR-10 and ImageNet32x32 ) , training budgets ( ranging from 100 to 1800 epochs ) , and learning rate schedules ( fixed , drop-step , and cosine annealing ; see Figure 1 ) . We demonstrate that our decoupled weight decay renders the optimal settings of the learning rate and the weight decay factor much more independent , thereby easing hyperparameter optimization ( see Figure 2 ) . The main motivation of this paper is to improve Adam to make it competitive w.r.t . SGD with momentum even for those problems where it did not use to be competitive . We hope that as a result , practitioners do not need to switch between Adam and SGD anymore , which in turn should reduce the common issue of selecting dataset/task-specific training algorithms and their hyperparameters . 2 DECOUPLING THE WEIGHT DECAY FROM THE GRADIENT-BASED UPDATE . In the weight decay described by Hanson & Pratt ( 1988 ) , the weights θ decay exponentially as θt+1 = ( 1− λ ) θt − α∇ft ( θt ) , ( 1 ) where λ defines the rate of the weight decay per step and ∇ft ( θt ) is the t-th batch gradient to be multiplied by a learning rate α . For standard SGD , it is equivalent to standard L2 regularization : Proposition 1 ( Weight decay = L2 reg for standard SGD ) . Standard SGD with base learning rate α executes the same steps on batch loss functions ft ( θ ) with weight decay λ ( defined in Equation 1 ) as it executes without weight decay on f regt ( θ ) = ft ( θ ) + λ′ 2 ‖θ‖ 2 2 , with λ ′ = λα . The proofs of this well-known fact , as well as our other propositions , are given in the Appendix A . Due to this equivalence , L2 regularization is very frequently referred to as weight decay , including in popular deep learning libraries . However , as we will demonstrate later in this section , this equivalence does not hold for adaptive gradient methods . One fact that is often overlooked already for the simple case of SGD is that in order for the equivalence to hold , the L2 regularizer λ′ has to be set to λ α , i.e. , if there is an overall best weight decay value λ , the best value of λ ′ is tightly coupled with the learning rate α . In order to decouple the effects of these two hyperparameters , we advocate to decouple the weight decay step as proposed by Hanson & Pratt ( 1988 ) ( Equation 1 ) . Looking first at the case of SGD , we propose to decay the weights simultaneously with the update of θt based on gradient information in Line 9 of Algorithm 1 . This yields our proposed variant of SGD with momentum using decoupled weight decay ( SGDW ) . This simple modification explicitly decouples λ and α ( although some problem-dependent implicit coupling may of course remain as for any two hyperparameters ) . In order to account for a possible scheduling of both α and λ , we introduce a scaling factor ηt delivered by a user-defined procedure SetScheduleMultiplier ( t ) . Now , let ’ s turn to adaptive gradient algorithms like the popular optimizer Adam Kingma & Ba ( 2014 ) , which scale gradients by their historic magnitudes . Intuitively , when Adam is run on a loss function f plus L2 regularization , weights that tend to have large gradients in f do not get regularized as much as they would with decoupled weight decay , since the gradient of the regularizer gets scaled along with the gradient of f . This leads to an inequivalence of L2 and decoupled weight decay regularization for adaptive gradient algorithms : Algorithm 1 SGD with L2 regularization and SGD with decoupled weight decay ( SGDW ) , both with momentum 1 : given initial learning rate α ∈ IR , momentum factor β1 ∈ IR , weight decay/L2 regularization factor λ ∈ IR 2 : initialize time step t ← 0 , parameter vector θt=0 ∈ IRn , first moment vector mt=0 ← 0 , schedule multiplier ηt=0 ∈ IR 3 : repeat 4 : t← t+ 1 5 : ∇ft ( θt−1 ) ← SelectBatch ( θt−1 ) . select batch and return the corresponding gradient 6 : gt ← ∇ft ( θt−1 ) +λθt−1 7 : ηt ← SetScheduleMultiplier ( t ) . can be fixed , decay , be used for warm restarts 8 : mt ← β1mt−1 + ηtαgt 9 : θt ← θt−1 −mt −ηtλθt−1 10 : until stopping criterion is met 11 : return optimized parameters θt Algorithm 2 Adam with L2 regularization and Adam with decoupled weight decay ( AdamW ) 1 : given α = 0.001 , β1 = 0.9 , β2 = 0.999 , = 10−8 , λ ∈ IR 2 : initialize time step t ← 0 , parameter vector θt=0 ∈ IRn , first moment vector mt=0 ← 0 , second moment vector vt=0 ← 0 , schedule multiplier ηt=0 ∈ IR 3 : repeat 4 : t← t+ 1 5 : ∇ft ( θt−1 ) ← SelectBatch ( θt−1 ) . select batch and return the corresponding gradient 6 : gt ← ∇ft ( θt−1 ) +λθt−1 7 : mt ← β1mt−1 + ( 1− β1 ) gt . here and below all operations are element-wise 8 : vt ← β2vt−1 + ( 1− β2 ) g2t 9 : m̂t ← mt/ ( 1− βt1 ) . β1 is taken to the power of t 10 : v̂t ← vt/ ( 1− βt2 ) . β2 is taken to the power of t 11 : ηt ← SetScheduleMultiplier ( t ) . can be fixed , decay , or also be used for warm restarts 12 : θt ← θt−1 − ηt ( αm̂t/ ( √ v̂t + ) +λθt−1 ) 13 : until stopping criterion is met 14 : return optimized parameters θt Proposition 2 ( Weight decay 6= L2 reg for adaptive gradients ) . Let O denote an optimizer that has iterates θt+1 ← θt − αMt∇ft ( θt ) when run on batch loss function ft ( θ ) without weight decay , and θt+1 ← ( 1 − λ ) θt − αMt∇ft ( θt ) when run on ft ( θ ) with weight decay , respectively , with Mt 6= kI ( where k ∈ R ) . Then , for O there exists no L2 coefficient λ′ such that running O on batch loss f regt ( θ ) = ft ( θ ) + λ′ 2 ‖θ‖ 2 2 without weight decay is equivalent to runningO on ft ( θ ) with decay λ ∈ R+ . We decouple weight decay and loss-based gradient updates in Adam as shown in line 12 of Algorithm 2 ; this gives rise to our variant of Adam with decoupled weight decay ( AdamW ) . Having shown that L2 regularization and weight decay regularization differ for adaptive gradient algorithms raises the question of how they differ and how to interpret their effects . Their equivalence for standard SGD remains very helpful for intuition : both mechanisms push weights closer to zero , at the same rate . However , for adaptive gradient algorithms they differ : with L2 regularization , the sums of the gradient of the loss function and the gradient of the regularizer ( i.e. , the L2 norm of the weights ) are adapted , whereas with weight decay , only the gradients of the loss function are adapted ( with the weight decay step separated from the adaptive gradient mechanism ) . With L2 regularization both types of gradients are normalized by their typical ( summed ) magnitudes , and therefore weights x with large typical gradient magnitude s are regularized by a smaller relative amount than other weights . In contrast , weight decay regularizes all weights with the same rate λ , effectively regularizing weights x with large s more than standard L2 regularization does . We demonstrate this formally for a simple special case of adaptive gradient algorithm with a fixed preconditioner : Proposition 3 ( Weight decay = scale-adjusted L2 reg for adaptive gradient algorithm with fixed preconditioner ) . Let O denote an algorithm with the same characteristics as in Proposition 2 , and using a fixed preconditioner matrix Mt = diag ( s ) −1 ( with si > 0 for all i ) . Then , O with base learning rate α executes the same steps on batch loss functions ft ( θ ) with weight decay λ as it executes without weight decay on the scale-adjusted regularized batch loss f sregt ( θ ) = ft ( θ ) + λ′ 2α ∥∥θ √s∥∥2 2 , ( 2 ) where and √ · denote element-wise multiplication and square root , respectively , and λ′ = λα .

Summary: In this paper, the authors investigate a very simple but still very interesting idea of decoupling weight decay and gradient step. It is a well known problem that Adam optimization method leads to worse generalization and stronger overfitting than SGD with momentum on classification tasks despite its faster convergence. The authors tried to find a reason for such behavior. They noticed that while SGD with L2 regularization is equivalent to SGD with weight decay, it is not the case for adaptive methods, such as Adam. The main contributions include the following: