Stable Weight Decay Regularization
Authors: Zeke Xie
Issei Sato
Masashi Sugiyama
Venue:  (2021)
Keywords: Weight Decay
Regularization
Optimization
Deep Learning

Abstract: Weight decay is a popular regularization technique for training of deep neural networks. Modern deep learning libraries mainly use $L_{2}$ regularization as the default implementation of weight decay. \citet{loshchilov2018decoupled} demonstrated that $L_{2}$ regularization is not identical to weight decay for adaptive gradient methods, such as Adaptive Momentum Estimation (Adam), and proposed Adam with Decoupled Weight Decay (AdamW). However, we found that the popular implementations of weight decay, including $L_{2}$ regularization and decoupled weight decay, in modern deep learning libraries usually damage performance. First, the $L_{2}$ regularization is unstable weight decay for all optimizers that use Momentum, such as stochastic gradient descent (SGD). Second, decoupled weight decay is highly unstable for all adaptive gradient methods. We further propose the Stable Weight Decay (SWD) method to fix the unstable weight decay problem from a dynamical perspective. The proposed SWD method makes significant improvements over $L_{2}$ regularization and decoupled weight decay in our experiments. Simply fixing weight decay in Adam by SWD, with no extra hyperparameter, can outperform complex Adam variants, which have more hyperparameters. 

Under review as a conference paper at ICLR 2021
STABLE WEIGHT DECAY REGULARIZATION
Anonymous authors Paper under double-blind review
ABSTRACT
Weight decay is a popular regularization technique for training of deep neural net- works. Modern deep learning libraries mainly use L2 regularization as the default implementation of weight decay. Loshchilov & Hutter (2018) demonstrated that L regularization not identical weight decay for adaptive gradient methods, such as Adaptive Momentum Estimation (Adam), and proposed Adam with Decoupled Weight Decay (AdamW) However, we found that the popular implementations of weight decay, including L2 regularization and decoupled weight decay, in modern deep learning libraries usually damage performance. First, the Le regularization is unstable weight decay for all optimizers that use Momentum, such stochastic gradient descent (SGD). Second, decoupled weight decay is highly unstable for all adaptive gradient methods. We further propose the Stable Weight Decay (SWD) method to fix the unstable weight decay problem from dynamical perspective. The proposed SWD method makes significant improvements over La regularization and decoupled weight decay in our experiments. Simply fixing weight decay in Adam by SWD, with no extra hyperparameter, can outperform complex Adam variants, which have more hyperparameters.
INTRODUCTION
Weight decay is popular and even necessary regularization technique for training deep neural networks that generalize well (Krogh Hertz 1992). People commonly use L2 regularization as "weight decay" for training of deep neural networks and interpret it as a Gaussian prior over the model weights. This true for vanilla SGD. However, [Loshchioov Hutter (208) revealed that, when the learning rate is adaptive, the commonly used L2 regularization is not identical to the vanilla weight decay proposed by Hanson & Pratl (989)):
0, = (1 1o)0 - 1]9t>
where Ay is the weight decay hyperparameter, 0; is the model parameters at f-th step. is the learning rate, and gt is the gradient of the minibatch loss function L(0) 0,- Zhang al. (2018) revealed three different roles of weight decay. However, the quantitative measures for weight decay are still missing.
Adaptive gradient methods that use Adaptive Learning Rate, such as AdaGrad (Duchi et al. 2011). RMSprop (Hinton et al.) 2012), Adadelta (Zeiler, 2012) and Adam (Kingma Ba 2015 are class of most popular methods to accelerate training of deep neural networks. Loshchilov & Hutter (2018) reported that, in adaptive gradient methods, the correct implementation of weight decay should be applied to the weights directly and decoupled from the gradients. We show the different implementations of decoupled weight decay and La regularization for Adam in Algorithm 3
It has been widely observed that adaptive gradient methods usually do not generalize as well as SGD the hidden problems adaptive gradient methods, including AdamW Loshchilov & Hutter (2018), AMSGrad (Reddi al. (2019) and Yogi (Zaheer et al., 2018). A recent line of research, such as AdaBound (Luo et al. [2019). Padam (Chen & Gu 2018), and RAdam (Liu al. 2019). believes controlling the adaptivity of learning rates may improve generalization. This line of research usually introduces extra hyperparameters to control the adaptivity, which requires more efforts in tuning hyperparameters.
(Wilson et al.] (207)). A few Adam variants tried
Although popular optimizers have achieved great empirical success for training of deep neural networks, discover that nonoptimal weight decay implementations have been widely used in

---

Under review as a conference paper at ICLR 2021
Table 1: Test performance comparison of optimizers. We report the mean and the standard deviations (as the subscripts) of the optimal test errors computed over three runs of each experiment. AdamS generalizes better than popular adaptive gradient methods significantly and often compares favorably with the baseline optimizer SGD.
DATASET
Monm
SGD
ADAMI
ADAM
A AMSGRAD
ADAMW
ARARoUND
PADAM
YoGi
RADAM
CIFAR-10
RESNeTI8
6.96p0
5.08
5.65a.c
5.12a.e
VGG16
6.42ccs
6.0
7.31eas
7.34
6.50%
6.70g.u
6.150.0
6.00p.s
6.56
CIFAR-100
DENSENEri21
1981.a
20.52h.36
25.13o.4
24.43a.c
21.55e.4
22.03
21.10..a
22.15m.a
22 22.270.32
GoogLENEr
21.216.39
21.05
26.120.31
25.53
21.25
23.18 28.18
21.82,37
24.24e 18
22.236.55
modern deep learning libraries, including PyTorch (Paszke et al. 2019). TensorFlow (Abadi et al.
2016), MXNet (Chen et al.] [015), and Chainer (Tokui et a (019). The nonoptimal implementations can lead to the unstable weight decay problem during training, which may harm performance seriously, We try to answer when the unstable weight decay problem happens and how to mitigate it. While Loshchilov & Hutter (2018) discovered that weight decay should be decoupled from the gradients, we further discovered that weight decay should be coupled with the effective learning rate. We organize main findings follows.
1. The effect of weight decay can be interpreted as iteration-wisely rescaling the loss landscape and the learning rate the same time. We formulate the weight decay rate based on the rescaling ratio per unit stepsize. We call unstable weight decay if the weight decay rate is not constant during training, seen in Definition Our empirical analysis suggests that the unstable weight decay problem may undesirably damage performance of popular optimizers.
2. L2 regularization unstable weight decay in all optimizers that use Momentum. Most popular optimizers in modern deep learning libraries use Momentum and L2 regularization at the same time, including SGD (with Momentum). Unfortunately, L2 regularization often damages performance in the presence of Momentum.
Decoupled weight decay is unstable weight decay in adaptive gradient methods. All adaptive gradient methods used L2 regularization as weight decay until Loshchilov & Hutter (2018) proposed decoupled weight decay. However, decoupled weight decay only solves part of the unstable weight decay problem. Decoupled weight decay is still unstable in the presence of Adaptive Learning Rate.
Always make weight decay rates stable. We proposed the stable weight decay (SWD) method which applies bias correction factor on decoupled weight decay to make weight decay more stable during training. SWD makes significant improvements over L2 regularization and decouple weight decay in our experiments, We display the test performance in Table [ The Adam with SWD (AdamS) displayed in Algorithm
2 A DYNAMICAL PERSPECTIVE ON WEIGHT DECAY
In this section, we study how weight decay affects learning dynamics and how to quantitatively measure the effect of weight decay from a viewpoint of learning dynamics. We also reveal the hidden weight decay problem in SGD with Momentum.
A dynamical perspective weight decay. We first present a new theoretical tool for understanding the effect weight decay. The vanilla weight decay described by Hanson Pratt (1989) is given by Equation
I A more popular implementation for vanilla SGD in modern deep learning libraries is given by
0, = 01-11\100-11 @L(0,-))
00
where we denote the training loss of one minibatch as L(0) and weight decay should be coupled with the learning rate.
We define new coordinates w, = 01(1 nd)-1, which an deratioo-dependent rescaled system of 0. In the system of w, we may define the loss function of as 1. (w.) T((1 17A) w.) T.(O.).
2

---

Under review as a conference paper at ICLR 2021
Then, may rewrite Equation 2
W = w (1- (L(0)-1) Wt-1 - (1-17) 1)7))1)+)) 21+1,,1111
OL (w
(3)
00
Equation 3 describes learning dynamics in the rescaled coordinates of w. The equivalence of Equation
2 and Equation suggests that learning dynamics with weight decay in the original coordinates of equivalent to learning dynamics without weight decay in the rescaled coordinates of w. Thus, the
effect of weight decay can be interpreted as flattening the loss landscape of 0 by a factor of (1 - ma)) per iteration and increase the learning rate by a factor of (1 m)) per iteration.
The weight decay rate and the total weight decay effect. In dynamics of SGD, the learning rate n may be interpreted as unit stepsize or unit dynamical time. The dynamical time of deep learning dynamics at t-th iteration can be defined as the cumulative sum of stepsizes: T = tyy. Thus, for each time step as dr = n. is (1 - A) that decides the rescaling rate of the dynamical system w per unit dynamical time. We define the weight decay rate as the rescaling rate of the dynamical system w per
unit dynamical time:
R R-1-X, X,
(4)
which indicates the rescaling rate of the coordinates w per unit stepsize or unit dynamical time. We define the total weight decay effect after t iterations as
Pt = (1- m7)
(5)
which indicates the total flattened ratio of the coordinates w after iterations. We naturally have Pr Rº in the continuous-time limit or small y limit. Based on the weight decay rate, we define the key concept, stable weight decay, as Definition]] [
Definition (Stable Weight Decay). The weight decay is stable if the weight decay rate constant during training. Otherwise, we define it unstable weight decay.
Suppose there exists an optimal weight decay rate for training of a deep network. Then, we should fine-tune Ả to have proper weight decay rate and keep the the weight decay rate approximately equal to the optimal weight decay rate during training. Thus, we conjecture that stable weight decay should be desirable property for deep learning. In this paper, we conducted comprehensive empirical analysis to verify the advantage of stable weight decay.
Vanilla weight decay is unstable weight decay in the presence of learning rate schedulers. We first take vanilla SGD as the studied example, where no Momentum is involved. It is easy to see that vanilla SGD with L2 regularization }) || is also given by Equation 2 Suppose the learning rate is fixed in the whole training procedure. Then Equation 2 will be identcal to Equation if we simply choose λo 77λ. However, learning rate decay is quite important for training of deep neural networks. So Equation 2 is not identical to Equation in practice.
Which implementation is better? Based on the measure of the weight decay rate, we propose Proposition ! We leave all proofs in Appendix
Proposition 1. Suppose learning dynamics is governed by Equation I and the learning rate at t-th step is Tj. Then, the weight decay rate at t-th step is given by R
Ao,
Corollary 2.0.1. Suppose all conditions of Proposition Thold. Then, Equation Connation--bbseeee bassed weight decay is unstable weight decay in the presence of learning rate schedulers.
Based on Definition] I and Proposition I we naturally obtain Corollary 2.0. We argue that Equation- based weight decay is better than Equation [ based weight decay, because Equation 2 can make the weight decay rate stable. In Figure 1 we empirically verified that the popular Equation [ based weight decay indeed outperforms the vanilla implementation in Equation [ Fortunately, the Equation based implementation has been widely accepted by modern deep learning libraries.
Le regularization is unstable weight decay in the presence of Momentum. Our previous analysis revealed that L2 regularization is stable weight decay for vanilla SGD. It is also commonly believed that L2 regularization is an optimal weight decay implementation when Adaptive Learning Rate is not involved. Almost all deep learning libraries directly use L2 regularization as the default weight decay implementation. However, we discover that L2 regularization is not identical to weight decay and often harms performance slightly when Momentum is involved.
3

---

Under review as a conference paper at ICLR 2021
0.72
Exp
Varila 50D
Eqlt
Eqp A 1110
0.10
Teat Ciee
0.00
0.50
0.37
15
100
125
1.50
375
200
30"
Epociss
Woaff
We compared Equation based weight decay and Equationn2 based weight decay by training ResNet18 on CIFAR-10 via vanilla SGD.
Figure 1:
We compare the generalization of Vanilla SGD, SGD, SGDW, and SGDS under var- ious weight decay hyperparameters by training VGG16 on CIFAR-10. 10. The optimal performance of SGDS/SGDW is better than the widely used
Figure 2:
We divided the learning rate by 10 at the epoch of
80 and 160, which is popular learning rate sched-
uler. No matter how we choose À for Equation-
Vanilla SGD and SGD. For Vanilla SGD, SGD,
based weight decay, Equation- haased weight
and SGDS, may choose \t, Äs 0.0005
decay shows better test performance. It demon-
to maintain the optimal performance. But we have to re-tune Aw 0.005 for SGDW. Hyperparame- ter Setting: 31 = 0 for Vanilla SGD; 81 = 0.9 for SGD, SGDW, and SGDS. We repeat each simula-
strates that the form -nA0 is a better weight de-
cay implementation than -è in the presence of learning rate schedulers.
tion for three runs. A similar experimental result
for ResNet18 8 is presented in Appendix
5,005
5.885
10
5820
10
5.20
5855
5,400
5,800
5.325
Learino
3.325
51160
4395
4395
4.30
4830
4300
10-1
4,00
10
13:9
10th
10:9
10
13
10
10
Weight Decay
Weight Decay
Figure 3: The test errors of ResNet18 on CIFAR-IO. SGDS has a deeper blue basin near dark points (S 4.83%). The optimal choices of and are very close for SGDS and SGD.
We take Heavy Ball Method (HBM) (Zavriev & Kostyuk 1993), which only uses fixed momentum inertia and dampening coefficients, as the studied example in the presence of Momentum, as HBM- style Momentum methods are widely used by many popular optimizers. SGD implemented in Py (Paszke et al.] 2019) is actually HBM with default hyperparameters. SGD TensorFlow (Abadi et 20166 tollows slightly different style of Momentum, which uses the learning rate inside the updating rule of Momentum, seen in Appendix E. In this paper, we use the popular implementation of SGD in PyTorch, displayed in Algorithm D as the base implementation, it follows the style of
HBM. Note that our theoretical results can be very easily generalized to the SGD implementation in TensorFlow. We write HBM with Le regularization
my -- Bs(g + AL101-1)
(6)
0, 0, 1y11
where 31 and 33 are the hyperparameters. In the common setting of SGD with Momentum, ,1 = 0.9 and B3 1. We propose Proposition which indicates that L2 regularization is unstable in the
presence of Momentum.
Proposition 2. Suppose learning dynamics is governed by SGD with L2 regularization and 31 > Then, the followings hold:
(1) La regularization is not identical to weight decay;
(2) the weight decay rate Tog regularization is not constant during training.
4

---

Under review as a conference paper at ICLR 2021
Fixing weight decay in Momentum. For obtaining stable weight decay rate, propose the SWD correction for HBM as
m, -, Ba9t n2 0c-1 19m4,
0, 0s(1-01) 1-9,
where 8s(1-Ai) the effective learning rate in HIM,, as m, R3 Ba(1- Bly) As (1- converges 1-B:
I-3
.n soon, we use the simplified I-9,7 as the bias correction in practice, where a relatively large weight decay rate in the first dozen iterations can work like model parameter initialization strategy. fg It is easy see that the bias correction factor 1-81 for weight decay is exactly the difference between
our stable weight decay and decoupled weight decay suggested by Loshchilov & Hutter (2018). The pseudocode of SGD with SWD (SGDS) is displayed in Algorithm SWD (HBMS).
It may also be called HBM with
Algorithm 1: SGD/SGDW (HBM/HBMW)
Algorithm 2: SGDS (HBMS)
gt ([( A0.-1;
(((0--)): my - + 83gi
my 8,m-- + Sagi
8. = 01-1 - 17m 1110--1;
8, = 01-1 -mmm 1-8, 110t-1:
Bg
Now, we have three kinds of weight decay hyperparameters. We use AL2 Aw, and Äs denote the Le regularization hyperparameter, the decoupled weight decay hyperparameter, and the stable weight decay hyperparameter, respectively.
SGDS. We empirically verified that the optimal performance of SGDS/SGDW is often better than the widely used vanilla SGD and SGD, seen in Figures and TO We leave more empirical results of SGDS in Appendix C such Table Particularly, report the test performance of SGDS and SGD for ResNet18 8 on CIFAR-10 under various learning rates and weight decay in Figure 3 SGDS has a deeper blue basin than SGD.
Empirical Analysis
It is not well-known that SGDW often outperforms SGD slightly. We believe it is mainly because is necessary for maintaining good performance people rarely know re-tuning Aw based on T-8
9q
% 7). the weight when people switch from SGD to SGDW. As the effective learning rate of SGD is 1-9,
decay rate of SGDW is actually R 1- An rather than Aw we use different settings of 31 and B3 in decoupled weight decay, we will undesirably change the weight decay rate R unless we re-tune Aw However, people usually directly let Aw XL, for SGDW in practice. Figure [ shows that, the optimal dτ and Äg in SGD with 8 = 0.9 and 33 1) are both 0.0005, while the optimal Aw is 0.005 instead. The optimal >ι, and Äs are almost same, while the optimal dw is quite different. Thus, the advantage of SGDS over SGDW can save us from re-tuning the weight decay hyperparameter.
FIXING WEIGHT DECAY IN ADAPTIVE GRADIENT METHODS
In this section, we study weight decay in dynamics of adaptive gradient methods. Loshchilov & Hutter (2018) first pointed that, when the learning rate is adaptive, the commonly used L2 regularization not identical to weight decay.
Algorithm 3: Adam/AdamW
Algorithm 4: AdamS
9t VL(O,-1) 1) + A0-1i
9t VL(O,-));
m, = Bim--1 + (1 -81)gri
m, Bim--1 + (1 -1-11-10 (1)1
D2 = 12 + (1- 32)9?;
8284-1
+ +(1 -B2)9?
mr -T-33
my mr
T-
è. =
G = I
8, = 0111
77 my - 17 Vo.te
6 mean( (0z);
O,-1 Vo.+e
m
2 2-20-11
ve,
5

---

Under review as a conference paper at ICLR 2021
Decoupled weight decay is unstable weight decay in adaptive gradient methods. In the following analysis, we ignore the effect of Momentum and focus on the effect of Adaptive Learning Rate. Thus, AdamW dynamics can be written
0, = (1 11)0011 - -1/;"" - -.T(0(-1)
(8)
00
where is the exponential moving average of the squared gradients in Algorithm 3 and the power notation of a vector means the element-wise power of the vector. We interpret ne, as the effective learning rate of adaptive gradient methods. We clearly see that decoupled weight decay uses the vanilla learning rate rather than the effective learning rate. Thus, the weight decay rate is anisotropic in AdamW. W. We propose Proposition
Proposition 3. Suppose learning dynamics is governed by AdamW. Then, the weight decay rate Rw is given by
Rw = =
(9)
where 1 is the all-ones vector. Thus, decoupled weight decay is unstable weight decay in adaptive gradient methods.
We also note that good property of decoupled weight decay is that the total weight decay effect after iterations is still P: (1 ryA)". In summary, the advantage of AdamW is that the total weight effect p after training is isotropic and does not depend on e, while the disadvantage of Adam W is that makes the weight decay rate anisotropic and unstable during training.
Fixing weight decay adaptive gradient methods. easy to fix weight decay by using the following birrecoree weight decay:
0: (1 17;; x)) - 1pu,"" - -T.
(10)
00
where the weight decay rate R (1 A). Without losing generality, we first assume the dimension- ality of f is one. The bias correction factor e, make the weight decay rate ideally stable during training in one-dimensional dynamics.
However, we still use the element wise bias correction factor u, multi-dimensiinn dynamics. the total weight decay effect p will become highly different along different dimensions: Pe
(1 (1 nv The anisotropic total weight decay effect undesirable. Unfortunately, there is no simple solution to achieving both an ideally stable weight decay rate and an isotropic total weight decay effect the same time for adaptive gradient methods. This may be an internal fault of all optimizers that use Adaptive Learning Rate.
Although there is no ideal stable weight decay for Adam, it is possible to apply an isotropic bias correction term to implement relatively more stable weight decay. We propose the updating rule in presence of Adaptive Learning Rate and SWD as:
@L(0--1) 0, (1 n7, x)0, 17
(1I)
00
where 02 is the mean of all elements of the vector vt. This is exactly the updating rule of AdamS, displayed in Algorithm 4 Here ryū 1 for approximating the effective learning rate. As E à is isotropic, the total weight decay effect is still same along different dimensions. The weight decay rate Rs of AdamS dynamics is given by
Rs - Ao, 2
(12)
Finally, we propose Propositions and for quantitatively measuring the magnitude of the weight decay rate of AdamS and Adam W. As La norm is the most popular measure for the magnitude of vectors, we measure the stability of AdamW and AdamS in terms of T.2 norms the coefficients of A in the corresponding weight decay rate.
Proposition 4. For the weight decay rate of Adam W in Equation 9 the L2 norm of the vector coefficient of X equal the sum of all elements in Ut, which is highly varying during training.

---

Under review as a conference paper at ICLR 2021
0.12
0.5
635
0.13
MkM Ain
0:0
0200
F20
0.00
0:00
624
6:07
622
622
100
125
150
17%
200
25
125
150
175
200
25
90
100
125
200
(a) VGGI6 on CIFAR-I0
(b) ResNer34 on CIFAR-100 -000
(c) DenseNet121 on CIFAR- 100
Aamms
Adams
Adurss
23
Transyy
23
33-1
11-1
33-4
3344
32%
1.50
200
Spadha
Cpadaa
Spocha
Figure 4: The learning curves of AdamS, AdamW, and Adam on CIFAR-10 and CIFAR-100. Top Row: Test curves. Bottom Row: Training curves. AdamS shows significantly better generalization than AdamW and Adam.
Proposition 5. For the weight decay rate of AdamS in Equation 12 the L2 norm of the vector coefficient of X equal to the space dimensionality, which constant during training.
we argue that SWD is more stable than decoupled weight decay in adaptive gradient methods in the sense of the magnitude.
Based on Propositions 4 and 5
EMPIRICAL ANALYSIS AND DISCUSSION
In this section, we empirically study how the SWD-corrected optimizers are compared with conven- tional optimizers. We choose Adam as the base optimizer, and train popular deep models, including R (He et al. 2016), VGG16 (Simonyan & Zisserman, 2014), DenseNet121 (Huang cc al. 2017), GoogLeNet (Szegedy et al. 2015), and Long Short- Memory (LSTM) (Hochreiter & Schmidhuber 1997). on IOCIFAR 100 (Krizhevsky et al., 2009) and Penn TreeBank
(Mareus et al. 1993). The implementation details can be found in Appendix
shows the learning curves of AdamS, AdamW, and Adam on several benchmarks. In our experiments, AdamS always leads lower test errors, and sometimes leads to lower training losses. However, F that, even if with similar or higher training losses, AdamS still generalizes significantly better than AdamW, Adam, and popular Adam variants. Figure displays the learning curves of all adaptive gradient methods. The test performance of other models can be found in Table Simply fixing weight decay in Adam by SWD even outperforms recent Adam variants. Figure 7 further demonstrates that AdamS consistently outperforms Adam and AdamW under various weight decay hyperparameters. According to Figure we also notice that the optimal decoupled weight decay hyperparameter in AdamW can be very different from Lz2 regularization and stable weight decay, but the corresponding optimal weight decay rates of three weight decay implementations are often similar. Finally, AdamS also outperforms AdamW and Adam significantly on the Language Modeling experiment, seen in Figure g We leave the experiments with cosine annealing schedulers and warm restarts (Loshchilov & Hutter 2016) in Appendix F. which also support that AdamS yields superior test performance.
Empirical results.
Popular Adam variants often generalize worse than SGD. Some papers (Luo et al. 2019 Chen & Gu 2018) argued that their proposed Adam variants may generalize as well as SGD. However, found that this argument is contracted with comparative experimental results, such as Table
7

---

Under review as a conference paper at ICLR 2021
0.24
0.32
Adams AdUTT
Adart
0.12
0.31
0.28
A4376
0..9
KAANY
10-1
AMSOUUT
0.22
0.66
RACAET
220
0.22
0.24
0.25
3.28
0.33
0:2
6:34
0.2
25
50
300
350
375
230
30
17.0
10-2
10.5
Decay
Figure 5: The scatter plot of training losses and test errors
Figure 6: The learning curves
Figure 7: The test errors of VGGI6 on CIFAR-IO with var- ious weight decay rates. The displayed weight decay value of Adam has been rescaled by the bias correction factor ss 0.001. A similar experimental result for ResNet34 is presented in Appendix D
of all adaptive gradient methods by training ResNet34 on CIFAR- 100. AdamS outperforms other Adam variants. The test perfor- mance of other models can be
during final 40 epochs of train-
ing ResNet34 on CIFAR-100.
Even with similar or higher train-
ing losses, AdamS still general- izes better than other Adam vari-
found in Table
ants. We leave the scatter plot
on CIFAR-10 in Appendix D
Adars
Adart
1.00
heptelyy
Truabrs Peczleut
1.20
1:2
302
8
02-
312
202
30
25
12.0
202
(a) Test Perplexity
(b) Training Perplexity
Figure 8: Language Modeling. The learning curves of AdamS, AdamW, and Adam for LSTM on Penn TreeBank. AdamS has better test performance than Adam W and Adam for LSTM. The optimal test perplexity of AdamS, AdamW, and Adam are 69.90, 72.88, and 77.01, respectively. Note that the lower perplexity is better.
I In our empirical analysis, these advanced Adam variants may narrow but not completely close the generalization gap between adaptive gradient methods and SGD. SGD with a fair weight decay hyperparameter the baseline performance usually generalizes better than recent adaptive gradient methods. The main problem may lie in weight decay. SGD with weight decay À = 0.0001, a common setting in related papers, is often not good baseline, as À 0.0005 often shows better generalization on CIFAR- 10 and CIFAR- 100, seen in Figures 2 10 and 7 We also conduct comparative experiments 0.0001, while some existing Adam variants may outperform SGD sometimes due to the lower baseline performance of SGD, AdamS shows superior test performance. For example, for ResNet18 on CIFAR-10, the test error of AdamS is lower than SGD by nearly one point and no other Adam variant may compare with AdamS.
with X 0. 0.0001, seen in Table of Appendix D Under the setting
Two metrics for weight decay. The main effect of weight decay for deep learning dynamics can be reflected by these two metrics: the weight decay rate R and the total weight decay effect p. We summary two principles about weight decay critically related to performance. First, we need to make the weight decay rate stable possible during training. Second, we need choose the optimal weight decay hyperparameter to maintain the optimal total weight decay effect. These two principles lead to two linear scaling rules.
Two linear scaling rules.
We report two kinds of linear scaling rules of weight decay and learning rates for selecting the weight decay hyperparameter X. First. Proposition] I and Figure [ indicate, the learning rate scheduler should be applied to both the learning rate and weight decay in the form of -1-10 during training. Second, the initial learning rate and the weight decay hyperparameter À
8

---

Under review as a conference paper at ICLR 2021
10
10
6.26
10-9
6.26
6.04
6.04
6.04
Leening Bate
5.82
5.82
5.82
10-1
5.60
10-1
5.60
10
5.60
5.38
4.54
4.94
4.72
4.72
10.5
4.50
10-1
4.50
10-9
4.50
10
10"
Weight Decay
Weight Decay
Weight Decay
Figure 9: The test errors of R CIFAR-10. AdamS has much deeper and wider basin near dark points (S 4.9%). The optimal test еπтог of AdamS, AdamW, and Adam are 4.52%, 4.90%, and 5.49%, respectively. The displayed weight decay value of AdamW has been rescaled by the bias correction factor = 0.001.
linearly coupled for the total weight decay effect. The expression of the total weight decay effect in Equation 5 mainly depends on the weight decay hyperparameter A, the learning rate and the number of iterations 1. Usually, people do not fine tune the number of iterations Given fixed and the optimal weight decay rate R. the total weight decay effect approximately depends on nı). The second linear scaling rule indicates that, in the procedure
fine-tuning hyperparameters, we should try larger À for smaller initial learning rates to maintain the optimal total weight decay effect. We observe that the blue basin of AdamS and Adam W in Figure both approximately reflect the linear scaling rule, which was also reported by Van Laarhoven (2017); Loshchilov & Hutter (2018); [i et al. (2020). We point that its theoretical mechanism closely relates to the total weight decay effect.
A novel interpretation of weight decay. The effect of weight decay can be interpreted as iteration- wisely flattening the loss landscape and increasing the learning rate at the same time. Thus, it is not surprising that increasing weight decay and increasing learning rate have some similar influence optimization performance (Van Laarhoven 2017 Zhang et al. 2018 Hotfer et al,, 2018). However, simply increasing learning rates cannot replace weight decay in practice. For weight decay, increasing the learning rate by a factor of (1 17 -2 is iteration-vee -iio during training, and flattening the loss landscape by factor of -))) per iteration must happen the same time. This interpretation of weight decay actually controls the adaptivity of learning rates in learning dynamics, as the total weight decay effect p is isotropic along different dimensions. This may explain why stable weight decay can effectively fix the ill-conditioned flat minima selection of adaptive gradient methods revealed by Xie al. (2220b) from diffusion perspective (Xie et al. 2020a).
CONCLUSION
While Loshchilov Hutter (008)) discovered that weight decay should decoupled from the gradients, we further discovered that weight decay should be coupled with the effective learning rate. We proposed the SWD method which applies the bias correction on decoupled weight decay to make the weight decay rate more stable during training. The empirical results demonstrate that SWD makes significant improvements over L2 regularization and decoupled weight decay (when they are different). Particularly, observe that the advantage of existing Adam variants is very weak compared with the performance improvement by simply fixing weight decay. The standard Adam with SWD, with extra hyperparameter, usually outperforms complex Adam variants, such as Padam and AdaBound, which have more hyperparameters. Although our analysis mainly focused on Adam, SWD can be easily combined with other Adam variants, too. The proposed principle and method of stable weight decay expands our understanding towards the role of weight decay in deep learning. It will be interesting to further investigate the theory for the weight decay rate in future.
REFERENCES
Martín Abadi, Paul Barham, Jiammin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow; A system for large-scale

---

Under review as a conference paper at ICLR 2021
machine learning. In 12th {USENIX) symposium operating systems design and implementation ((OSDI) 16), pp. 265-283, 2016.
Jinghui Chen and Quanquan Gu. Closing the generalization gap of adaptive gradient methods in 2018.
training deep neural networks. arXiv preprini
Tianqi Chen, Mu Li, Yutian Li, Min Lin, Naiyan Wang, Minjie Wang, Tianjun Xiao, Bing Xu, Chiyuan Zhang, and Zheng Zhang. Mxnet: A flexible and efficient machine learning library for heterogeneous distributed systems. arXiv preprint arXiv: 111.011111......... 2015.
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research, 12(Jul):121-2155) 2011.
Stephen José Hanson and Lorien Y Pratt. Comparing biases for minimal network construction with back-propagation. Advances in neural information processing systems, pp. 177-185, 1989.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Geoffrey Hinton, Nitish Srivastava, and Kevin Swersky. Neural networks for machine learning lecture 6a overview of mini-batch gradient descent. 2012.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1780- 1997.
Elad Hoffer, Ron Banner, Itay Golan, and Daniel Soudry. Norm matters: efficient and accurate normalization schemes in deep networks. In Advances in Neural Information Processing Systems, pp. 2160-2170, 2018.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kiliann Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-47000 2017.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 3rd International Conference on Learning Representations, ICLR 2015, 2015.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Anders Krogh and John Hertz. simple weight decay can improve generalization. In Advances in neural information processing systems, pp. 950-957, 1992.
Zhiyuan Li, Kaifeng Lyu, and Sanjeev Arora. Reconciling modern deep learning with traditional optimization analyses: The intrinsic learning rate. Advances in Neural Information Processing Systems, 33, 2020.
Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. On the variance of the adaptive learning rate and beyond. In International Conference on Learning Representationss, 2019.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608. 03983, 2016.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, In International Confer- ence on Learning Representations, 2018.
Liangchen Luo, Yuanhao Xiong, Yan Liu, and Xu Sun. Adaptive gradient methods with dynamic bound of learning rate. 7th International Conference on Learning Representations, ICLR 2019, 2019.
Mitchell Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building a large annotated corpus of english: The penn treehank. 1993.
10

---

Under review as a conference paper at ICLR 2021
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. In Advances in neural information processing systems, pp. 8026-8037, 2019.
Sashank J Reddi, Satyen Kale, and Sanjiv Kumar. On the convergence of adam and beyond. 6th International Conference Learning Representations, ICLR 2018, 2019.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1556, 2014.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1 1-9, 2015.
Seiya Tokui, Ryosuke Okuta, Takuya Akiba, Yusuke Nitaani, Toru Ogawa, Shunta Saito, Shuji Suzuki, Kota Uenishi, Brian Vogel, and Hiroyuki Yamazaki Vincent. Chainer: A deep learning framework for accelcrating the research cycle. In Proceedings of the 25th ACM SIGKDD International Conference Knowledge Discovery Data Mining, pp. 2002-2011, 2019.
Twan Van Laarhoven. L2 regularization versus batch and weight normalization. arXiv preprint arXiv:1706.05350, 2017.
Ashia C Wilson, Rebecca Roelofs, Mitchell Stern, Nati Srebro, and Benjamin Recht. The marginal value of adaptive gradient methods in machine learning. In Advances in Neural Information Processing Systems, pp. 4148-4158, 2017.
Zeke Xie, Issei Sato, and Masashi Sugiyama. A diffusion theory for minima selection: Stochastic gradient descent escapes from sharp minima exponentially fast. arXiv preprint arXiv:2002.03495, 2020a.
Zeke Xie, Xinrui Wang, Huishuai Zhang, Issei Sato, and Masashi Sugiyama, Adai: Separating the effects of adaptive learning rate and momentum inertia. arXiv preprint arXiv:2006.15815, 2020b.
Manzil Zaheer, Sashank Reddi, Devendra Sachan, Satyen Kale, and Sanjiv Kumar. Adaptive methods for nonconvex optimization. In Advances in neural information processing systems, pp. 9793-9803, 2018.
Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. Recurrent neural network regularization, arXiv preprint 1409.220..2322 2014.
SK Zavriev and FV Kostyuk. Heavy-ball method in nonconvex optimization problems. Computational Mathematics and Modeling, 4(4):336-34) 1993.
Matthew D Zeiler. Adadelta: adaptive learning rate method. arXiv preprint arXiv: 2012.
Guodong Zhang, Chaoqi Wang, Bowen Xu, and Roger Grosse. Three mechanisms of weight decay regularization. In International Conference on Learning Representations, 2018.
A PROOFS
A.1 PROOF OF PROPOSITION 1
Proof. In the presence of learning rate schedulers, we have the updating rule of Equation based weight decay as
8, = (1-1A 20)0,_1 17-9t-
(13)
According to the definition of the weight decay rate, we have R == Aa at 1-th step. Obviously, the weight decay rate will change with learning rate schedulers.
The proof is complete.
11

---

Under review as a conference paper at ICLR 2021
A.2 PROOF OF PROPOSITION 2
Proof. We first show how L2 regularization and weight decay are different in the presence of Momentum.
We may rewrite HBM with L2 regularization as
m - + Bsge
(14)
0, rmm4 13 1 00-x 0 -1p; 7/ 83 I-23
1-0i
(01-1);
where we denote the exponential moving average of past 0 as (0c-1) - Tー Il
1-8
- 1 O.-
Although (0--1) an approximated value of 0c-1. Le regularization use (0c-1) instead of the standard 0 in weight decay introduces undesirable noise into learning dynamics of SGD with L2 regularization.
As (0:-1) * f.-1. obviously, we may not recover weight decay in Equation 2 from L2 regularization. This naturally means that L2 is not identical weight decay.
The weight decay rate of SGD with L2 is given by
Bi 31
R = 1 AL
(0 (0--1)0,11
(15)
where 1 is the all-ones vector and 0,-1 the element-wise inverse of 8;-1-
The difference between 0 and (0.-1) also indicates that the weight decay rate cannot be constant during training.
The proof is complete.
A.3
PROOF OF PROPOSITION 3
Proof. According to the definition of the weight decay rate, the weight decay rate of AdamW dynamics is
Rw 1 - Au
(16)
where 1 is the all-ones vector.
As w- is adaptive, Rw is not constant during training.
The proof is now complete.
A.4 PROOF OF PROPOSITIONS
4
AND 5
Proof. By Equation 9 and Equation 12 we have the weight decay rate of Adam as
Rw 1 - Av
(17)
and the weight decay rate of AdamS
Ao, 1
Rs
(18)
We note that X in Equation 12 (AdamS) has vector coefficient 02 20g while À in Equation (AdamW) has a vector coefficient as o
It is easy to compute the L2 norms of two vector coefficients; || ue ||2 equals the dimensionality n AdamS dynamics and || ll2 equals the sum of all elements v, in AdamW dynamics.
It means À in AdamS has magnitude-constan coefficient, while À in AdamW has magnitude- varying coefficient.
The proof is complete.
12

---

Under review as a conference paper at ICLR 2021
Table 2: Test performance comparison of Adai, AdaiS, SGD, and SGDS. Stable/Decoupled Weight Decay often outperform L2 regularization for optimizers involving in momentum. We report the mean and the standard deviations (as the subscripts) of the optimal test errors computed over three runs of each experiment.
DATASET
MODEL
ADAIS
ADAI
SGDS
SGD
CIFAR-10
RESNET18
4.
4.740.14
4.69p.09
5.01p.03
VGG16
5.810.07
6.00o.09
6.28o.07
6.42p.02
CIFAR-100
DENSENET121
19.440.21
19.590.38
19.61o.26
19.81
GOOGLENET
20.50
20.55g.32
20.68g.03
21.21o.28
B EXPERIMENTAL DETAILS
B. IMAGE CLASSIFICATION
Data Preprocessing: We perform the common per-pixel reao-mean unit- variance normalization, horizontal random flip, and 32 32 random crops after padding with 4 pixels on each side.
Hyperparameter Settings: We select the optimal learning rate for each experiment from {0.001, 0.01, 0.1, 10) for non-adaptive gradient methods and use the default learning rate for adaptive gradient methods. In the experiments on CIFAF-10 10 and CIFAR-100. 100: = for Adai; r 0.1 for AdaiS; 1 0.1 for SGD and SGDS: r = 0.001 AdamS, Adam, AMSGrad, Adam W, and AdaBound;
0.01 for Padam. For the learning rate schedule, the learning rate is divided by 10 the epoch of (80, 160) for CIFAR-10 and {100, 150) for CIFAR-100, respectively. The batch size is set to 128 for both CIFAR-10 10 and CIFAR-100.
The strength of Le regularization and SWD is default to 0.0005 as the baseline. Considering the linear scaling rule, we choose Aw X1. Thus, the weight decay of AdamW uses Aw 0.5 for CIFAR-10 10 and CIFAR-100. As for SWD in AdaiS, choose Äs 0.005 for achieve similar total weight decay effect as SGD. The basic principle of choosing weight decay strength is to let all optimizers have either a similar weight decay rate R or similar total weight decay effect p the baseline L2 regularization.
We set the momentum hyperparameter 3 0.9 for SGD and SGDS. As for other optimizer hyperparameters, we apply the default hyperparameter settings directly.
We leave the empirical results with the weight decay setting À 0.0001 in Appendix D
The source code of AdamS and SGDS in Supplementary Materials is available to the public.
B.2 LANGUAGE MODELING
We use a classical language model, Long Short-Term Memory (LSTM) (Hochreiter & Schmidhuberi 1997) with 2 layers, 512 embedding dimensions, and 512 hidden dimensions, which has million model parameters and is similar to the "medium LSTM" in Zaremba et al. (2014). Note that our baseline performance is better than the reported baseline performance in Zaremba et al. (2014). The benchmark task is the word-level Penn TreeBank (Mareus al. 1993). We empirically compared AdamS, AdamW, and Adam under the common and same conditions.
Hyperparameter Settings. Batch Size: B 20. BPTT Size: bptt 35. Weight Decay: AL, As 0.00005, and Xw = 0.05. Learning Rate: = 0.001. The dropout probability is set to 0.5. We clipped gradient norm to 1.
13

---

Under review as a conference paper at ICLR 2021
SGDS
SGDW
10-
10-4
10
10-
10
10-0 10
Weight
Figure 10: We compare the generalization of Vanilla SGD, SGD, SGDW, and SGDS with various weight decay hyperparameters by training ResNet18 on CIFAR- 10. The optimal weight decay rates are near 0. 0.0005 for all three weight implementations. The optimal performance of SGDS/SGDW better than Vanilla SGD and SGD. For Vanilla SGD, SGD, and SGDS, we may safely choose dL Äs 0.0005. But have re-tune dw 0.005 for SGDW. Hyperparameter Setting: 31 = for Vanilla SGD; B1 0.9 for SGD, SGDW, and SGDS.
Sois
6.0
560
0.685
Teat E100
0000
0.000
265
0.605
624
0.000
1012
325
75
160
(a) ResNet18 on CIFAR-10
(b) VGG16 on CIFAR-10
(c) GoogLeNet on CIFAR-100
Figure 11: Generalization analysis on SGDS and SGD with Le regularization. Hyperparameter
Setting: As = ÅL 0.0005 and B1 0.9.
0.64
Asa
0.000
A24
0.85
626
0.00
4/
0.0
0.000
75
102
325
150
175
0.6
202
155
375
620
75
160
135
Esxiss
Concs
Epeets
(a) ResNet18 CIFAR-10
(b) VGG16 on CIFAR-10
(c) GoogLeNet CIFAR-100
Figure 12: Generalization analysis on AdaiS and Adai with La regularization. Hyperparameter
Setting: Äs AL, 0.0005.
14

---

Under review as a conference paper at ICLR 2021
1.30
028
(a) ResNet18
(b) VGG16
(c) DenseNet121
(d) GoogLeNet
Figure 13: The learning curves of adaptive gradient methods.
Aviurs AdainW
Troak Loke
10-4
tent
10-1
16-1
Tea: Emto
Tea: 5115
Figure 14: Even if with similar or higher training losses, AdamS still generalizes better than Adam W and Adam. The scatter plot of training losses and test errors during final 50 epochs of training VGG16 on CIFAR-10 10 and 1e on CIFAR-100.
C SUPPLEMENTARY FIGURES AND RESULTS OF SGD WITH MOMENTUM AND ADAI
We compare SGDS, SGDW, vanilla SGD and SGD under various weight decay hyperparameters by training ResNet18 on CIFAR-10. We observe that the optimal performance of SGDS/SGDW is better than vanilla SGD and SGD in Figure [0] The advantage of SGDS over SGDW is that we do not need to fine-tune the weight decay hyperparameters based
&
I-81
We report the learning curves of SGD and SGDS in Figure 11 SGDS compares favorably with SGD,
Based on a diffusion theoretical framework (Xie et al. 2020a), Xie et al. (020b) proposed Adaptive Inertia Estimation (Adai) that uses adaptive momentum inertia instead of Adaptive Learning Rate to help training. Adaptive inertia can be regarded as an pvrrii-aadppivv variant of HBM. The previous analysis on HBM can be easily generalized Adai. We display Adai with SWD (AdaiS) in Algorithm 6 We report the learning curves of Adai and AdaiS in Figure 12] which may verify the generalization advantage of AdaiS over Adai.
D SUPPLEMENTARY FIGURES AND RESULTS OF ADAPTIVE GRADIENT METHODS
We report the learning curves of all adaptive gradient methods in Figure [3 They shows that vanilla Adam with SWD can outperform other complex variants of Adam.
Figure 14 displays the scatter plot of training losses and test errors during final 40 epochs of training DenseNet121 21 CIFAR-100.
Figure 13 displays the test performance AdamS, Adam W, and Adam under various weight decay hyperparameteers of ResNet34 on CIFAR-100.
We train ResNet18 on CIFAR-10 10 for 900 epochs to explore the performance limit of AdamS, AdamW, and Adam in Figure 16
15

---

Under review as a conference paper at ICLR 2021
0.255
Acam
0.25
0233
0223
0.35
Figure 15: We compare the generalization of Adam, Adam and AdamS with various weight decay rates by training ResNet34 CIFAR-100. The displayed weight decay of AdamW in the figure has been rescaled by the bias correction factor zs 0.001. The optimal test performance of AdamS significantly better than AdamW and Adam.
Adums Adum
0.04
300
600
020
500
1pocss
Figure 16: We train ResNet18 on CIFAR-10 for 900 epochs to explore the performance limit of
AdamS, Adam W. and Adam. The learning rate is divided by 10 at the epoch of 300 and 600. AdamS achieves the most optimal test error, 4.70%.
Table displays the test performance with = 0.0001, which is a common weight decay setting in related papers.
Finally, we present the learning curves for language modeling experiments in Figure As we expect, AdamS outperforms AdamW and Adam again.
E ADDITIONAL ALGORITHMS
Algorithm is the TensorFlow implementation for SGD.
Algorithm 6 is the implementation of Adai, AdaiW, and AdaiS. As So 1-2,
is always in Adai, AdaiW
identical to AdaiS.
16

---

Under review as a conference paper at ICLR 2021
Table 3: Test performance comparison of optimizers with dL, As 0.0001 and Aw 0.1. AdamS still show better test performance than popular adaptive gradient methods and SGD.
DATASET
MODEL
SGD
ADAMS
AMSGRAD
ADAMW
ADABOUND
PADAM
YaGI
RADAM
CIFAR-10
RESNETI8
5.58
1.69
6.08
5.72
5.33
6.87
5.83
5.13
5.81
VGG16
6.45
6.74
6.08
6.73
CIFAR-100
DENSRNETI21
20.98
21.35
24.39
22.80
22.23
24.23
22.26
22.40
22.40
GOOGLENET
21.89
21.60
24.60
24.05
21.71
25.03
26.69
22.56
22.35
We note that the implementation of AMSGrad in Algorithm 7 is the popular implementation in PyTorch. We use the PyTorch implementation in our paper, as it widely used in practice.
Algorithm 5: SGD TensorFlow
9t VL + A8;-1;
m, = 81m-11 199s
8,- 0,-1 m
Algorithm 6: Adai AAdaiS=Adaii
9t ( + A8
C2 B2t1-1 + B2)9?: 24 1-
6
6 me)n((;);
B1 = (1 8o 8 CCClpp( e):
m = -1m + (1 B1z)gr
mi -- 8In"
0. 0-1 nm4 11
Algorithm AMSGrad/AMSGradW
Algorithm 8: AMSGradS
9t VL(6,-1) + A8
Jt VL(0(-1);
m, = 81m4-1 + (1 81)g
m 81m-11 1 + (1 31)gi
02 = $20--1 + (1 - B2)9?
0t = 320 - B2)9
m4 1-
m4 m:
1-21
Umax = max(Ee (U,, Umax):
Umar = max (Ve, Umaz);
0 Tmar
Emaz:
1-3
I-3
0₂ P-11 Jo-te my 17201-1;
mean(ts) (z);
= 0:-1 vbete m. 2
A0
/e.
F EXPERIMENTS WITH COSINE ANNEALING SCHEDULERS AND WARM RESTARTS
In this section, conducted comparative experiments on AdamS, AdamW, and Adam in the presence of cosine annealing schedulers and warm restarts proposed by Loshchilov & Hutter (2016). We set the learning rate scheduler with recommended setting of Loshchilov & Hutter (2016): To 14 and Tmaa 2. The number of total epochs is 210. Thus, we trained each deep network for four runs of warm restarts, where the four runs have 14, 28, 56, and 112 epochs, respectively. Other hyperparameters and details are displayed in Appendix B
Our experimental results in Figures 17 and 18 suggest that AdamS consistently outperforms AdamW and Adam in the presence of cosine annealing schedulers and warm restarts. It demonstrates that, with various learning rate schedulers, the advantage of SWD may generally hold.
17

---

Under review as a conference paper at ICLR 2021
baans Aeartil
Aeam
2.2
3.15
2.0
330
3.05
2.06
Epecta
Epecta
(a) ResNet18
(b) VGGI6
20-1
Tradkn Lone
Figure 17: The learning curves of ResNet18 and VGG16 on CIFAR-10 with cosine annealing and warm restart schedulers. The weight decay hyperparameter: \ = Às = 0.0005 and Aw = 0.5. Top Row: Test curves. Bottom Row: Training curves. AdamS yields significantly lower test errors and training losses than AdamW W and Adam.
0.33
0,23
0.24
0.58
10.
30
10.
Whight Ocoo
Wright Dena
(a) ResNet18
(b) VGGI6
Figure 18: The test errors of ResNet18 and VGG16 on CIFAR-10 under various weight decay with cosine annealing and warm restart schedulers. AdamS yields significantly better optimal test performance than Adam W and Adam.
18


Reviews:
[{"review_id": "jbXQPbStVVH", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors study the effect of weight decay across different optimizers. When one uses weight decay, the learning rate which multiplies the weight decay is different from the effective learning rate (the term multiplies the gradient dependent contribution). The authors propose to adjust for this by having the effective learning rate multiplying the weight decay term, ie Delta theta= -eta_{eff} lambda theta-eta_{eff} F(gradient). For SGD, this implies that one can roughly equate weight decay and L2 regularization by rescaling the L2-parameter. For the case of Adam, the effective learning rate is anisotropic, which they fix by taking the mean of the vector 'v'. The authors show that after correcting for the effective learning rate, which they call AdamS, they can get the same performance as what one gets with SGD in CIFAR10 and CIFAR100. \n\nI find the discussion about rho and R confusing (they are not defined precisely and using the small lr/lambda approximation seems unnecessary) and they do not really add much.  Similarly the term unstable seems a little strong given that there is nothing bad going on with training of such models. They also make comments about reinterpreting weight decay as flattening the loss and increasing the learning rate which they don't pursue further nor connect with the main point and it seems a little out of context. The main conceptual point of the paper is simply that the weight decay should have the same effective learning rate as the gradients, it would be nice if the authors could make this more clear and more central.\n\nThe paper main point then is to equate the learning rate in the weight decay coefficient by the effective learning rate and they show that this might be enough to bridge the gap between Adam and SGD. I think this is an interesting problem, but given that this is their unique point they should probably back it up with more experiments, since they only check it for CIFAR datasets (4 experiments total). If the authors could do one extra experiment on Imagenet (for example with a standard Resnet-50), that would raise my score to weak accept.  ", "extra_scores": {}, "extra_text": {}}, {"review_id": "e-UuszURwzN", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper presents a novel weight decay regularization, stable weight decay by using the bias correction to the decoupled weight decay in adaptive gradient descent optimizer. They empirically found that the L2 regularization the decouple weight decay are unstable weight decay. The proposed stable weight decay can fix this issue in terms of dynamical perspective. Experimental results based on benchmark dataset show that the proposed scheme outperform the popular and advanced optimizers in generalization. Weight decay has been a basic technique in most optimizer and indeed there are not many studies for the effect on the performance. Also, how to obtain the optimal weight decay is still missing. \n\nI think this paper has done a decent investigation on this topic. The overall paper is easy to follow and seems technically sound. \n\nHowever, one major issue in my mind is that conclusions are only supported by the empirical results, which though look promising. No formal theoretical claims or results have been reported. This hurts the paper in terms of theoretical novelty. Even though some shallow analysis has been presented in the draft, it is not enough. \n\nMoreover, a couple of statements can be arrived at by directly observing existing algorithms, which are not that technical. For example, Statement 1 that says “Equation-1-based weight decay is unstable weight decay in the presence of learning rate scheduler.” can be quickly summarized even from an intuitive sense without any derivation, which makes it trivial. There is no need to give Definition 1 formally for Stable Weight Decay as that doesn’t sound like a definition. In the draft, the authors can directly say constant or time-varying weight decay. Stable doesn’t just mean constant, though I can completely understand what the authors really meant in the paper. Statement 4, in my opinion, is one of key results in the draft. However, the current theoretical analysis is not enough to support this conclusion. More formal theoretical analysis is required.\n\nA minor point, in Eq 3, how did the authors arrive at $-2t-1$ in the superscript of weight decay rate, not $-2t+1$? Additionally, I am a little confused about the statement that the effect of weight decay can be interpreted as flattening the loss landscape of $\\theta$ by a factor of $(1-\\eta\\lambda)$ per iteration and increase the learning rate by a factor of $(1-\\eta\\lambda)^{-2}$ per iteration. Can the authors put more detail, a bit more of derivation to show this?\n\n***************************************\nAfter reading the rebuttal and considering carefully, I think the authors' response addressed some of issues in my mind so I raised the score. However, in terms of theoretical foundation, the current paper draft is still only marginal, which requires substantial improvement. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "d0Y2DeCSPcV", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper presents a novel framework that alters the weight decay update rule and aims to improve generalization when applied to 1) momentum-based optimizers and 2) adaptive optimizers. The framework includes the concept of “weight decay rate” and “total weight decay”, where the idea is to make the “weight decay rate” constant throughout training. The framework is applied to “fix” or “stabilize” weight decay when applied to SGD with momentum and Adam (which effectively amounts to changing the weight decay parameter). The result is better generalization when applied to a variety of image classification tasks, compared to the appropriate baselines. \n\nStrengths:\nThe core of the idea is novel, relatively simple, and the experiments support the claim.\n\nWeaknesses:\nMy main complaint is that this paper is poorly written. I don’t understand the motivation behind the reparameterization of the update rule with w instead of \\theta. The concept of “weight decay rate” and “total weight decay” seems to be the most critical part of the paper, but the explanation surrounding them was messy and hard to understand. The language used is not precise at some points in the text.\n\nAt the current state of the paper, I recommend a reject because I don’t think it’s polished enough to be published. \n\nComments and questions:\n- I think the concept of weight decay rate and total weight decay should be explained better, and earlier in the paper. Without knowing what “stable” meant, I had a hard time taking the introduction seriously, because I didn’t know what “unstable” meant exactly, and that the decoupled weight decay method was “unstable”. \n- “\\beta_3 = 0” should be “beta_3 = 1”. The Pytorch documentation calls 1-beta_3 dampening and its default value is zero.\n- I’m not sure I understand the sentence “Although <\\theta_{t-1}> is an approximated value of \\theta_{t-1}, replacing \\theta_{t-1} by <\\theta_{t-1}> still introduces undesirable noise into learning dynamics.” What do you mean by undesirable noise? Why is replacing <\\theta_{t-1}> by <\\theta> justified? \n\n===== Update =====\n\nI have read the authors’ response, the updated paper, and the other reviews. I believe that the changes made by the authors address my concerns about the motivation and the lack of clarity; section 2 reads much better now. \n\nIt seems like the main complaints of the other reviewers are in the lack of more difficult workloads, and the lack of theory. I personally don’t find the lack of theory very important. I think the novelty comes from the simple observation, which no one to my knowledge has come to before, and the experiments support the idea empirically (which I think is what actually matters). I also find it a bit uncomfortable penalizing the authors for not running experiments on ImageNet, and I think the variety in architectures that the authors tried, compensates for this. I do agree that a more modern set of workloads (transformers, or even the same setup as AdamW) would have made the paper much stronger. \n\nI increased my score to a 6 because I think the paper in its current form is enough to get accepted, but there are still improvements that could be done to make it much stronger.", "extra_scores": {}, "extra_text": {}}, {"review_id": "GPSY2oklaRz", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Summary\n\nIn this paper the authors introduce the notion of stable weight decay. The stable weight decay property can be defined in dimension 1 as follow: the effective learning rate represents an amount of time ellapsed between two iteration. The weight decay factor normalized (in log space) by the time ellasped should be constant across iterations.\n\nFrom their framework, the authors also propose SGDS, a minor modification to SGD where the amount of L2 regularization is increased with momentum, in order to balance for the larger step sizes that the momentum will yield.\n\nWhen applied to Adam, the authors derive AdamS, supposed to work better than the previous AdamW, which already improved how weight decay and Adam interact. AdamS has the stable weight decay property, unlike Adam or AdamW. AdamS weight decay amount is scaled by the denominator of Adam, taking the average over all dimensions to make it isotropic.\n\nThe authors test their methods on vision tasks, where Adam is known to underperform compared to SGD, and their method AdamS achieves significant gains compared to AdamW.\n\n## Review\n\nThe reformulation introduced in equation (3) is an interesting alterntive view to look at weight decay, but I find it is not really used by the authors. The definition of the weight decay rate can be done from equation (2).\nThe authors introduce the notion of stable weight decay and seem to right away assume it is a desirable property. In particular, there is no theoretical justification that this is the case.\n\nThe benefit from SGDS is limited, except for hyper parameter tuning (as only the first few iterations of SGD are \"unstable\"), but it serves as a nice illustration of the new concept.\n\nThe part on adaptive methods is more interesting but the authors deviate significantly from their theoretical framework. Verifying the stable weight decay property is actually not optimal, because it is not isotropic. The authors trick is to average the value of the moving average of the squared gradients along all dimensions before using it to rescale the weight decay.\n\nThe experimental section of the paper focuses only on vision. It would have been interesting to see the effect of AdamS on other type of tasks.\n\nOverall I think the idea introduced by the author is interesting, although the theory is not completely coherant. The experiments shows significant improvement but could have been on a more diverse set of tasks. Still,  I recommend acceptance.\n\n\n## Remarks\n\n- In the Introduction, talking about adaptive methods: \"are a class of dominated methods to accelerate\", I'm not sure what dominated means here.\n- what is the point of having $\\beta_3$? after equation (4), the authors say \"SGD with momentum is $\\beta_3=0$\", that doesn't seem right, if $\\beta_3=0$, then the gradient is completely ignored.\n\n=============\nUpdate avec rebutal and discussion with AC and reviewers.\n\nAfter discussing with the others reviewers and AC, I have come to share their concerns with the overall fragility of the paper. We agreed that the methods is sound and likely to work better than AdamW, the proofs are not sufficient. In particular, the authors should strive to provide experiments on different training sets (ImageNet) with learning rate cross validation. The authors do not systematically compare across learning rates which make it hard to interpret the results as being conclusive. In fact only CIFAR-10 is evaluated with multiple learning rates.\n\nThe remark by Elya Loshchilov should also be adressed. Note that you cannot use stochastic noise as a justification, because for heavily overparameterized neural network, the amount of stochastic noise at the optimum is zero (i.e. perfect fitting of the training set). But there is another explanation: Intuitively for a fixed $\\beta_2$, $v_t$ goes to zero as the current gradient goes to zero, and the ratio of the gradient by $v_t$ will converge to some constant, which prevents convergence. $v_t$ goes to zero at the same speed as the gradient but with a delay of $1 / (1 - \\beta_2)$. If there is no convergence, then the gradients won't actually go to zero.\nThe only way to prevent $v_t$ from going to zero is to have $\\beta_2 \\rightarrow 1$ (i.e. the previously mentioned delay going to infinity), but in that case $\\bar{v}_t$ won't go to zero neither. This is only an idea of a possible justification and I would encourage the authors to think carefully about this stability issues in the next revisions.\n\nFinally I would encourage the authors the remove from the theoretical analysis parts that are not actually used (which I and other reviewers have noted).", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a novel way to have weight decay-like update rule. Empirically, the authors claim that it improves generalization when applied to momentum-based optimizers and optimizers with coordinate-wise learning rates.

This paper has been thoroughly discussed, both in public and private mode.
The strength of this paper lies in the possible gain in generalization performance due the proposed change.
The weaknesses are:
- the very confusing and not scientific motivation of the proposed change
- the experiments are not fully convincing

More in details, we all found the discussion on "stable" and "unstable" weight decay extremely confusing. The claim of the paper is that "stable" weight decay should be preferred over "unstable" one. However, to validate a scientific claim it is necessary to carry out an empirical or theoretical evaluation. The theoretical one is simply missing: a number of proposition and corollaries are stated with some simple mathematical facts completely disconnected from the optimization or generalization issues. As it is, removing these arguments would actually make the paper better.
On the empirical side, there is no experiment that supports the simple claim that "the unstable weight decay problem may undesirably damage performance". Instead, what we see are experiments in which the modified update rule seem to perform better, but they don't actually show that "stability" or "instability" are the specific issues at play here. Indeed, any other explanation is equally valid and the experiments do not support any specific one, but rather they can only support the claim that the proposed algorithm might be better than some other optimization algorithms. The *specific reason* why this is happening is not clear.

Turning to the empirical evaluation, the discussion elicited the fact that, a part for CIFAR10, the experiments are carried out without tuning of the learning rates. Hence, it is difficult for us to even validate the claim of superiority of the method. I don't subscribe to the idea that a deep learning paper requires experiments on ImageNet to be valid. Yet, given that there is no supporting theory in this paper, the empirical evaluation should be solid and thorough.

For the above reasons, the paper cannot be published at ICLR.
Decision: Reject