On the Overlooked Pitfalls of Weight Decay and How to Mitigate Them: A Gradient-Norm Perspective
Authors: Zeke Xie
zhiqiang xu
Jingzhao Zhang
Issei Sato
Masashi Sugiyama
Venue: NeurIPS 2023 poster (2023)
Keywords: Weight Decay
Regularization
Optimization
Deep Learning
TL;DR: We report the overlooked large-gradient-norm pitfalls of weight decay, which often indicates bad convergence and poor generalization, and propose a gradient-norm-aware scheduler to mitigate the pitfalls.

Abstract: Weight decay is a simple yet powerful regularization technique that has been very widely used in training of deep neural networks (DNNs). While weight decay has attracted much attention, previous studies fail to discover some overlooked pitfalls on large gradient norms resulted by weight decay. In this paper, we discover that, weight decay can unfortunately lead to large gradient norms at the final phase (or the terminated solution) of training, which often indicates bad convergence and poor generalization. To mitigate the gradient-norm-centered pitfalls, we present the first practical scheduler for weight decay, called the Scheduled Weight Decay (SWD) method that can dynamically adjust the weight decay strength according to the gradient norm and significantly penalize large gradient norms during training. Our experiments also support that SWD indeed mitigates large gradient norms and often significantly outperforms the conventional constant weight decay strategy for Adaptive Moment Estimation (Adam).

On the Overlooked Pitfalls of Weight Decay and How to Mitigate Them: A Gradient-Norm Perspective
Zeke Xieli2 Zhiqiang Xu Jingzhao Zhang", Issei Sato', and Masashi Sugiyama?
The University of Tokyo
RIKEN Center for AlP MBZUAI TTsinguua University
Correspondence: zekexie 16 @ gomil.com, Zhiqiang.Xer Xu mbzuai.ac.c.
Abstract
Weight decay is a simple yet powerful regularization technique that has been very widely used in training of deep neural networks (DNNs). While weight decay has attracted much attention, previous studies fail to discover some overlooked pitfalls large gradient norms resulted by weight decay. In this paper, discover that, weight decay can unfortunately lead to large gradient norms at the final phase (or the terminated solution) of training. which often indicates bad convergence and poor generalization. To mitigate the mendientnoomm-eeteer the first practical scheduler for weight decay, called the Scheduled Weight Decay (SWD) method that can dynamically adjust the weight decay strength according to the gradient norm and significantly penalize large gradient norms during training. Our experiments also support that SWD indeed mitigates large gradient norms and often significantly outperforms the conventional constant weight decay strategy for Adaptive Moment Estimation (Adam).
pitfalls, we present
Introduction
Deep learning
25 has achieved great success in numerous fields over the last decade. Weight decay the most popular regularization technique for training deep neural networks that generalize well 24]. In deep learning, there exist two types of "weight decay": L2 regularization and weight decay.
People commonly
the first type of L2 regularization for training of deep neural networks as V (L. Az, [0,--11)),, where V is the gradient operator, || is the I--norm, t
0, = f,- is the index of iterations, A denotes the model weights, 7/ is the learning rate, L(0) is the training loss, and ÅL is the L2 regularization hyperparameter. In principle, the L2 regularization strength is chosen be constant during training of DNNs and people do need schedule La regularization like learning rate decay. The second type of weight decay was first proposed by Hanson and Pratt IK to directly regularize the weight norm:
OL(0;-1)
0. = (1 - X) - 7h 09
where X' is the (vanilla) weight decay hyperparameter.
While the two types of "weight decay" seem similar to Stochastic Gradient Descent (SGD), Loshchilov and Hutter 332 reported that they make huge differences in adaptive gradient methods, such
as Adam 22 Adam often generalizes worse and finds sharper minima than SGD 40 36] 46) for popular convolutional neural networks (CNNs), where the flat minima have been argued to be closely related with good generalization [16] 17
14 51 21]. Understanding and bridging the generalization
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Table 1: Test performance comparison of L2 regularization, Decoupled Weight Decay, and SWD for Adam, which corresponds to Adam, AdamW, and the proposed AdamS, respectively. We report the mean and the standard deviations (as the subscripts) of the optimal test errors. We make the lowest two test errors bold for each model. The SWD method even enables Adam generalize as well as SGD and even outperform complex Adam variants for the five popular models.
DATASET
MODEL
ADAMS
ADAM
ADAXW
SCD
AMSGRAD
ADABOUND
PADAM
YOGs
RADAM
CIFAR-10
RusNer18
6.966..0
oz
6.3Go.s
5.66u.s
5.12..04
5.87o.12
6.03o.co
VGG16
6.09
7.31ass
6.5th 13
8.42nos
7.146.24
6.70p.12
6.13p.00
6.90v2:
8.560cc
CIFAR-100
RESNET34
21. 41.76:44
27. 27.00
22.56hu.4
21 52
25.554.19
22,87o
22.72m.1
23.5Fl
24. 3l.4
12
20. 2
25. 25.11
21.55a.
19. 8llss
24.450.9
22.69k 1a
21 21.10am
22.15
22 27
GooGLENNH
21.00 31.05...
26. 12
21.25 31.29u
21. 21
25.5 25.580.1
28
21.83a
24.24.
22. 25.25
gap between SGD and Adam have been a hot topic recently. Loshchilov and Hutter [32 argued that directly regularizing the weight norm is more helpful for boosting Adam than L2 regularization, and proposed Decoupled Weight Decay as
(1(00-1) 02 = 11---AA>>> )0.- T)
00
where Aw is the decoupled weight decay hyperparameter. We display Adam with L2 regularization and Adam with Decoupled Weight Decay (AdamW) in Algorithm To avoid abusing notations, we strictly distinguish L2 regularization and weight decay in the following discussion.
While fine analysis of weight decay has attracted much attention recently [52] 33 4]. some pitfalls of weight decay have still been largely overlooked. This paper analyzes the currently overlooked pitfalls of (the second-type) weight decay and studies how to mitigate them by gradient- norm---ooo---rreee scheduler. The main contributions be summarized as follows.
First, supported by theoretical and empirical evidence, we discover that, weight decay can lead to large gradient norms especially at the final phase of training. The large gradient norm problem significant in the presence scheduled or adaptive learning rates, such as Adam. From the perspective of optimization, large gradient norm at the final phase of training often leads to poor convergence. From the perspective of generalization, penalizing the gradient norm is regarded as beneficial regularization and may improve generalization [27 3 2 55). However, previous studies fail to discover that the pitfalls of weight decay may sometimes significantly hurt convergence and generalization.
Second, to the best of our knowledge, we are the first to demonstrate the effectiveness of scheduled weight decay for mitigating large gradient norms and boosting performance. Inspired by the gradient- norm generalization measure and the stability analysis of stationary points, we show that weight decay should be scheduled with the gradient norm for adaptive gradient methods. We design gradient-norm-avve scheduler for weight decay, called Scheduled Weight Decay (SWD), which significantly boosts the performance of weight decay and bridges the generalization gap between Adam and SGD. Adam with SWD (AdamS) is displayed in Algorithm 2 The empirical results in Table T using various DNNs support that SWD often significantly surpasses unscheduled weight decay for Adam.
Algorithm 1: Adam/AdamW
Algorithm 2: AdamS
9t L( + A8;-1;
(L((
m4 Bim--1 1 (1 Bi)gei
m4 Bim4-1 + (1 31 )g1
12 B204-1
+ B2)9?;
0t B204-1 +(1 B2)9? m.
m 1-
mi
m 1-29-
6, 21
G, T-
"
T-3
0. 02 Jo+te
M4 - 1120--1;
G. meam Oz (gradddent--nomm-warr))
0, = 0:-1 Vo,te m. Vo,+e A0,-1
Structure of this paper. In Section 2 we analyze the overlooked pitfalls of weight decay, namely large gradient norms. In Section 3 we discuss why large gradient norms are undesired. In Section 4 we propose SWD to mitigate the Targe gradient norm and boosting performance of Adam. In Section 5 we empirically demonstrate the efffcctiveness of SWD. In Section O we conclude our work.

---

2 Overlooked Pitfalls of Weight Decay
In this section, we report the overlooked pitfalls of weight decay, on convergence, stability, and gradient norms.
Weight decay should be coupled with the learning rate scheduler. The vanilla weight decay described by Hanson and Pratt [13] is given by Equation (T). A more popular implementation for vanilla SGD in modern deep learning libraries, such as PyTorch and TensorFlow, is given by Equation (2) where weight decay is coupled with the learning rate scheduler. While existing studies did not formally study why weight decay should be coupled with the learning rate scheduler, this trick has been adopted by most deep learning libraries.
We first take vanilla SGD studied example, where momentum is involved. It is easy to see that vanilla SGD with L2 regularization 2 ||0l| is also given by Equation (2). Suppose the learning rate fixed in the whole training procedure. Then Equation (2) will be identical to Equation (I) if we simply choose X' = 17>. However, learning rate decay is quite important for training of deep neural networks. So Equation 2 is not identical to Equation (T] practice.
Which implementation is better? In Figure] we empir- ically verified that the popular (quation-(2)-ba -based weight decay indeed outperforms the vanilla implementation in Equation (T). We argue that Equation- cay is theoretically better than (arattioonn---aseee - weight de- --bseeed weight decay in terms of the stability of the stationary points. While we use the notations of sstable""uunstute"" tionary points for simplicity, we formally term "sta- ble"/"unstable" stationary points as strongly/weakly sta- tionary points in Definition [ This clarifies that there can exist gap between true updating and pure gradient-basee updating for stationary points.
0.6
5
5
Tesk Enter
sta-
0.3
0.0
25
3.00
125
200
Epochss
Definition 1 (Strongly/Wcakly Stationary Points). Sup- pose A @ > @ is function that updates model parameters during training where θ is the parameter space. We define A" a strongly stationary point for A O holds during the whole training process. We define 0* weakly stationary point for A if 100- A((0))I = 0 holds only for some iterations.
We compared Equation- --based weight decay and Equation- 2 -basd weight decay by training ResNer18 on CIFAR-10 via vanilla SGD, the presence of a popular learning rate scheduler, Equation (arrioon-------eddddada--a weight de- cay shows better test performance. supports that the form -1 is a bet- ter weight decay implementation than -X80.
Figure
if f110 1 AA(0)]
The training objective at step-t of Equation ① is L(0)= +
X ||0122 while that of Equation 2 is T.(0) + 2l Thus, stationary points of the regularized loss given by Equation are stable during training, while that given by Equation is unstable due to the dynamic training objective.
Weight decay increases gradient norms the final phase of deterministic optimization and stochastic optimization. Following the standard conver- gence analysis of Gradient Descent (GD) ( with the extra modification of weight decay, we prove Theorem
D and show that even GD with vanilla weight decay may not converge to any non-zero stationary point due to missing stability of the stationary points.
Theorem ENon-convergenee due to vanilla weight decay and scheduled learning rates). Suppose learning dynamics is governed by GD with vanilla weight decay (Equation (T) and the learning rate Ne (0, +00) holds. 35 such that satisfies O < 5 S [n 1/+11 for any > O, then the learning dynamics cannot converge to any non-zero stationary point satisfying the condition
X2 16011 > 0,
max(IIFf..0.11) (V
777T+11
where 1c(0) = L(0) + X' ")
|10| is the regularized loss.
We leave the proof in Appendix A. Theorem means that even if the gradient norm zero at the t-th step, the gradient norm at the t 1-th step will not be zero. Thus, the solution does not
3

---

6.40
Adaim Wi-Len
Wi-Se-
035
Akmt WD-deed
Aam Whole
0.055
03-103
Adammi WD-r
0.000
WBnl
0.030
60+6
1.25
625
0.20
0.00
0.00
6.05
0.08
0.05
Adam W
Adam
(c) SGD
Ademri WD+lo>
Acuerii WD-103
Adamiii Worlo2
3.5
Aduenii WD-5044
3.5
Aduenti N0-d04
3.5
Adamti wo-dou
Adamn wo 1o-
Adamni 1--4
2.-
65
6.1
(d) AdamW
(e) Adam
(f) SGD
Figure 2: Weight decay increases the gradient norms and the squared gradient norms ResNet18 for
various optimizers. Top Row: the gradient norms, Bottom Row: the squared gradient norms.
converge to any non-zero stationary point. GD with unstable minima has theoretical convergence guarantee like GD [11] More importantly, due to unstable stationary points, the gradient norms (of the regularized/original loss) naturally also do not converge zero. This may explain why weight decay should be coupled with the learning rate to improve stability of stationary points during training. This non-convergence problem is especially significant when 8 is relatively large in the presence of scheduled or adaptive learning rates, such as Adam.
proved that the gradient norm bound of SGD in convergence analysis depends on the properties of the training objective function and gradient noise. We further prove Theorem 2 and reveal how the gradient norm bound in convergence analysis depends the weight decay strength À when weight decay is involved. We leave the proof in Appendix(A.. (A.2
Existing studies 48 12]
Theorem (Convergence analysis on weight decay). Assume that L(Ø) an C-smooth funnctooo L is lower-hounded as I(()) 2 I.', L(0, X) is the loss over one minibatch X, B T(0,, X) VL(@)] = 0, E [L,
V I 82, and [|VL(0)] S G for any 9. Let SGD optimize
f(0) L(0) + || for t + iterations. If < C
we have
VEFT
min f(xx))
[C+ Czl,
(3)
where
C L(Gu) + 211001 Iz
(4)
C
C₂ = C(C + A)((G + sup()[101)) + +02),
(5)
and sup([ supk=0.... ||0x| || is the maximum Tz norm of 0 the iterations.
Theorem I shows the non-convereence problem with improper weight decay. Theorem further shows that, even with proper convergence guarantees and stable minima. the gradient norm upper bound at convergence still monotonically increases with the weight decay strength A. Obviously, Theorem 2 theoretically supports that weight decay can result large gradient norms. The phenomenon can also be widely observed in the following empirically analysis.
It means that (0aa) VL(0))] S LI) 0b]l holds for any Og and 0s.

---

0.8
0.200
Adami W0-See4
Adamil WD-
0.7
Adami WD-4e-4
Adamw www.eell
0.6
Asamw WD-6
Adamwl WD-40
0.4
0.3
0.2
0.1
23
50
25
50
75
100
125
150
175
200
Epochs
Epachs
(a) VGG16 with BatchNorm
(b) VGGI6 without BatchNorm
2.00
WD-5a-4
Adammi ViD=See
AdamiW WD-4a-4
AdamiN WD
AdaemW WD-3e-4
Adamii WD-1e-4
Adom/N WD=0
Gradent
Squared
0.75
0.50
0.25
0.00
Epochs
(c) VGG16 with BatchNorm
(d) VGG16 without BatchNorm
Figure 3: Weight decay increases the gradient norms and the squared gradient norms of VGG16 with BatchNorm (scale-invariant) and VGG16 without BatchNorm (scale- variant) on CIFAR-10. Top Row: the gradient norms. Bottom Row: the squared gradient norms.
Empirical evidence of large gradient norms resulted by weight decay. Theorem suggests that both unstability and strength of weight decay may increase the gradient norm even in full-batch training. While weight decay is helpful for regularizing the model weights' norm, clearly observed in Figure tha weight decay significantly increases the gradient norms training of DNNs as we expect. Note that the displayed weight decay of AdamW is rescaled by the factor 0.001. The observation generally holds for various optimizers, such as AdamW, Adam and SGD. Figure 3 further shows that the observation holds for both scale-invariant models (VGGI6 with BatchNorm) and scale-variant models (VGG16 without BatchNorm). Moreover, the observation still holds for gradient norms with or without regularization included, because the numerical difference due to regularization is smaller than their gradient norms by multiple orders of magnitude in the experiments. Zhang et al. [52] argued that weight decay increases the "effective learning rate" due smaller weight norms resulted by weight decay. However, the "effective learning rate" interpretation may not explain large gradient norms, because a small weight norm does not necessarily indicate large gradient norm. Interestingly, Figure shows that the gradient curves of VGG16 without BatchNorm and weight decay sometimes exhibit strange spikes, which can be wiped out by weight decay.
Large Gradient Norms Are Undesired
In this section, we demonstrate that the large gradient norm is highly undesirable property in deep learning because large gradient norms correspond to multiple pitfalls.
First, small gradient norms are required for good convergence, which is desired in optimization theory (36]. Popular optimizers also expect clear convergence guarantees (8). The optimizers with no convergence guarantees may cause undesirable training behaviors in practice.
5

---

Second, regularizing the gradient norm has been recognized as an important beneficial regularization effect in recent papers [28
55 [4].
Li et al. [88] and Geiping et al. [T] expressed implicit
regularization of stochastic gradients as
Rgg B
Σ XEB
[V - 90))
(6)
where L(),X) indicate the loss over one data minibatch X E B, [3] is the number of minibatches per epoch, and G is the expected squared gradient norms over each minibatch. Recent studies, Zhao et al. [55] and Zhang et al. [54], also successfully improved generalization by explicitly adding the gradient-norm penalty into stochastic optimization. Obviously, penalizing the gradient norm is considered as an important and beneficial regularizer in deep learning.
Third, large gradient norms may theoretically indicate poor generalization according to the gradient- norm generalization measures [27 2]. For example, Theorem 11 of Li et al. (27 derived a gradient-
norm-based generalization error upper bound that uses the squared gradient norms, written as
Sup Gen = (0
I,*
" of
((0))
where n is the training data size, of is the gradient noise variance the t-th iteration, and G(O:)
[88] ExeB [] X)? 2 is the squared gradient norm with the batch size as 1. An et al. [2] presented
more empirical evidences for supporting the gradient- ceett-nomm-ooee noom-based generalization bound (7) Moreover,
large gradient norms are also believed to be closely related to minima sharpness (20 57
42 46
77. while sharp minima often indicate poor generalization [66] 17 5T 21]. As our analysis focuses
on the large gradient norm at the final phase (or the terminated solution) training, our conclusion is
actually not contradicted to the viewpoint [ 29 I0] that the gradient-noise regularization effect of large initial learning rates at the early phase of training is beneficial.
We also report that the gradient norm and the squared gradient norm show very similar tendencies in our experiments. For simplicity, we mainly use as the default gradient norm measure this paper, unless we specify it otherwise. As we discussed above, the squared gradient norm G is more closely related to implicit gradient regularization, generalization bounds, and SWD than other gradient norms. Overall, the theoretical results above supported that the small gradient norm is a desirable property in deep learning. However, our theoretical analysis suggests that weight decay can lead to large gradient norms and hence hurt generalization. We suggest that generalization can lead to the serious but overlooked performance degradation of weight decay.
poor convergence and
4 Gradient-Vorn-Aware Scheduled Weight Decay
In this section, we design the first practical scheduler to boost weight decay by mitigating large gradient norm.
We focus on scheduled weight decay for Adam in this paper for three reasons. First, Adam is the most popular optimization for accelerating training of DNNs. Second, as the adaptivity of Adam updates the learning rate every iteration, Adam suffers more than SGD from the pitfalls of weight decay due to adaptive learning rates. Third, SGD usually employs Le regularization rather than weight decay.
Adam with decoupled weight decay is often called AdamW (32. which can be written as
0. = (1 - 11)0)~101111-- - 770 mi:
(8)
where it and m, are the exponential moving average of the squared gradients and gradients in Algorithm [ respectively, and the power notation of vector means the element-wise power of the vector. We interpret Tpū as the effective learning rate for multiplying the gradients. However, we clearly see that decoupled weight decay couples weight decay with the vanilla learning rate rather than the effective learning rate. Due to the differences of weight decay and L2 regularization, Adam W and Adam optimize different training objectives. The minimum 0" of the regularized loss function optimized by Adam at the 1-th step is not constant during training. Thus, the regularized loss function optimized by AdamW has unstable minima, which may result in large gradient norms

---

Adars
1.5
Ad
AdemW
12"0
1.2%
Trane
2.00
0:0
32-1
20
Spacha
Norm
(b) Scheduler
(c) Training Loss
2.5
3.75
Adumd
Adunti
Adund Admm
3.50
2.30
0.00
1.
0:2
6.00
33.8
Epects
Spacha
(d) Norm
(c) Scheduler
(f) Training Loss
Figure 4: The curves of the weight decay scheduler, G, and training losses of SWD and constant weight decay. Top Row: ResNet18 on CIFAR-10. Bottom Row: ResNet34 on CIFAR-100 SWD mitigates large gradient norms.
suggested by Theorem [ This may be an internal fault of all optimizers that use adaptive learning rates and weight decay at the same time.
To mitigate the pitfall of large gradient norms, we propose mradient-norm-avee scheduler for weight decay
0= -1 (1- 1pu A)0:-1 v/u m.:
where is the mean of all elements of the vector Uh. Moreover, ve is the exponential moving average of squared gradient norms, the factor i. is expected to be equivalent to the squared gradient norms divided by model dimensionality. While the proposed scheduler cannot give perfectly stable minima for Adam, it can penalize large gradient norms by dynamically adjusting weight decay. We call weight decay with this gradient-norm--aver
scheduler Scheduled Weight Decay (SWD). The The (code is publicly available at GitHub. In Section 5 scheduler. Note that AdamS is slightly slower than A but slightly faster than Adam, while the cost difference is nearly ignorable (usually less than 5%).
pseudocode is displayed in Algorithm 2
we empirically demonstrate the advantage of the gradient-norm-nn
We note that ú is not expected to be zero at minima due to stochastic gradient noise, because the variance of the stochastic gradient is directly observed to be much larger than the expectation of the stochastic gradient at/near minima and depends on the Hessian 43 In some cases, such as full-batch training, it is fine add small value (e.g.. 10-5) to avoid being zero as a divisor.
Lewkowycz and Gur-Ari [26] recently proposed a simple weight decay scheduling method which has limited practical value because it only works well with constant learning rate, the original paper claimed. Bjorck al. 4 empirically studied early effects of weight decay but did not touch mitigating the pitfalls of weight decay. In contrast, effectively penalizes the large gradient norm and hence improve convergence and generalization.
experiments suggest that SWD indeed
Figure 4 verifies multiple advantages of SWD. First, SWD dynamically adjusts the weight decay strength inverse to G as we expect. Second, obviously, SWD can effectively penalize large gradient norms compared with constant weight decay during not only the final training phase but also the nearly whole training procedure. Third, SWD can often lead to lower training losses than constant weight decay. Figure reveals that, to choose proper weight decay strength, SWD dynamically

---

Adams WD=1e-
3.5
AdamS WDw5e4
Weight Decay Schedueer
Adam5 WD=1e4
2.5
Test Error
0.28
Adams
AdamW
2.5
0.26
Adam
AMSGred
0.24
1.0
Padam
0.22
Yogi
0.0
0.20
25
75
100
125
150
175
50
100
125
175
200
Epochs
Epochs
Figure 5: SWD adjusts the scheduler strength according to the initial weight decay hyperpa- rameter. The optimal choice is À 5 x 10t. 100. AdamS outperforms other Adam variants.
Figure 6: The learning curves of all adaptive gra- dient methods by training ResNet34 on CIFAR-
Model: ResNet18. Dataset: CIFAR-10.
6.12
8.54
0.11
A.mm Ain
0:20
Soee الll
620
0.00
0:00
622
622
8.00
830
25
30
100
123
150
175
200
25
30
100
125
150
175
200
25
30
75
100
125
120
175
200
(a) VGGI6 on CIFAR-10
(b) ResNet34 on CIFAR-
(c) DenseNet121 on CIFAR-100
Aamss
Adums
Adurs
23-1
22-4
22-
33-1
33-1
33-1
23
1.50
200
Op0cka
C2203
Spocha
Figure 7: The learning curves of AdamS, AdamW, and Adam on CIFAR-10 and CIFAR-100. AdamS shows significantly better generalization than AdamW and Adam. Top Row: Test curves. Bottom Row: Training curves.
employs a stronger scheduler when the initially selected weight decay hyperparameter is smaller and weaker scheduler when À is larger compared to the optimal weight decay choice, respectively.
5 Empirical Analysis
In this section, we first empirically verify that weight decay indeed affects gradient norms and further conducted comprehensive experiments to demonstrate the effectiveness of the SWD scheduler in mitigating large gradient norms and boosting performance.
Models and Datasets. We train various neural networks, including ResNet BesNet18/344/ 8//3/550 [5], VGG16 37). DenseNet121 [9, GoogLeNee (38), and Long Short-Term Memory (LSTM) 18, on CIFAR-
10/CIFAR-100 23 ImageNet [ and Penn TreeBank (34). The optimizers include popular Adam
variants, including AMSGrad (35), Yogi [49], AdaBound (33, Padam (5) and RAdam
30) See
Appendix B and C for more details and results.

---

WD=5e-4
ad AdamW
10a
AdamW WD=1e-4
AdamW WD-0
AdamS
Eigenvalue
10
Eigenvalue
102
10
10
10a
10
20
40
40 Eigenvalue Rank
20
Eigenvalue Rank
Figure 8: Weight decay significantly increase the minima sharpness in terms of top Hessian eigenvalues. Model: ResNet18. Dataset: CIFAR- 10.
Figure 9: Top 50 Hessian cigenvalues of ResNet18 at the minima learned by Adam, AdamW, and AdamS. AdamS learns flatter min- ima than AdamW.
Generalization. Figure shows the learning curves of AdamS, AdamW, and Adam on several benchmarks. our experiments, AdamS always leads lower test errors. Figure 19 Appendix shows that, even if with similar or higher training losses, AdamS still generalizes significantly better than AdamW, Adam, and recent Adam variants. Figure displays the learning curves of all adaptive gradient methods. The test performance of other models can be found in Table The results on ImageNet (See Tigure[8jin Appendix) shows that, AdamS can also make improvements over AdamW. Simply scheduling weight decay for Adam by SWD even outperforms complex Adam variants. We report that most Adam variants surprisingly generalize worse than SGD (See Appendix CC..
Minima sharpness. Previous studies 20
reported that the minima sharpness measured by the Hessian positively correlate the gradient norm. Note that number of popular minima sharpness depends on the Hessian at minima [21]. In Figures g annd 9
57 43 45|
we visualize the top Hessian cigenvalues to measure the minima sharpness. Figure & shows that stronger weight decay leads to sharper minima. Figure 9 suggests that AdamS learns significantly flatter minima than AdamW. Both large gradient norms and sharp minima correspond to high-complexity solutions and often hurts generalization [16]. This is surprising finding for weight decay given the conventional belief that weight decay as regularization technique should encourage low-complexity solutions.
The Robustness of SWD. Figure I| shows that AdamS is more robust the learning rate and weight decay than AdamW and Adam. AdamS has a much deeper and wider basin than AdamW and Adam. Figure [0] further demonstrates that AdamS consistently outper- forms Adam and AdamW under various weight decay hyperparameters. According to Figure 10 we also notice that the optimal decoupled weight decay hyperparameter Adam W can be very different from La regularization and SWD. Thus, AdamW requires re-tuning the weight de- cay hyperparameter in practice, which is time- consuming. The robustness of selecting hyper- parameters can be supported by Figure where the proposed scheduler also depends on the ini- tial weight decay strength. It means that the pro- posed gradient- aware scheduler exhibit the behavior of negative feedback to stabilize the strength of weight decay during training.
Adam5
0.11
Asamw
0.09
0.07
10-
30
10
10-
10a
10
We iight Decay
Figure 10: The test errors of VGG16 on CIFAR-10 with various weight decay rates. The displayed weight decay value of AdamW has been rescaled by the factor 0.001. A similar experimental result for ResNer34 presented in Appendix
Cosine learning rate schedulers and warm restarts. We conducted comparative experiments on AdamS, AdamW, and Adam the presence of cosine annealing schedulers and warm restarts proposed by Loshchilov and Hutter [31]. We set the learning rate scheduler with recommended

---

Adami
6.25
6.26
10-3
6.26
6.00
6.04
5.04
5.00
Leariing Rite
5.02
Leansing Rate
Leaning
10-4
3.50
20-4
3.80
4.94
4.94
4.72
4.72
4.72
10
4.50
10-1
4.50
10.5
4.50
10-h
10-4
10-1
10-4
10th
10-4
Decay
Weightt
Weight Docay
Figure 11: The test errors of ResNet18 on CIFAR-10, AdamS has much deeper and wider basin near dark points (5 4.9%). The optimal test errors of AdamS, AdamW, and Adam are 4.52%, 4.90%, and 5.49%, respectively. The displayed weight decay of AdamW has been rescaled by the factor 0.001.
setting Loshchilov and Hutter [3 Our experimental results in Figure 12 suggest that AdamS consistently outperforms AdamW and Adam in terms of both training losses and test errors. It demonstrates that, with complex learning rate schedulers, the advantage of SWD still holds.
035
15 15 16 (1
10
250
300
(a) ResNet18
(b) VGG16
(c) ResNet18
(d) VGG16
Figure 12: The test curves and training curves of ResNet18 and VGGI6 CIFAR-10 with cosine annealing warm restart schedulers. The weight decay hyperparameter: Å As = 0.0005 and Aw = 0.5. Left Two Figures: Test curves. Right Two Figures: Training curves. AdamS yields significantly lower test errors and training losses than Adam and Adam.
Limitations. While the proposed scheduler boosts weight decay in most experiments, the im- provement of SWD may be limited sometimes. SWD does not work well particularly when L2 regularization even yields better test results than weight decay (See the experiments on Language Modeling Figure [3 in Appendix C.. Our paper does touch the pitfall of La regularization and how to schedule its strength. We leave scheduling L2 regularization as future work.
Conclusion
While weight decay has attracted much attention, previous studies failed to recognize the problems of unstable stationary points and large gradient norm caused by weight decay. In this paper, theoretically and empirically studied the overlooked pitfalls of weight decay from gradient-norm perspective. To mitigate the pitfalls, we propose simple but effective gradient-norm-awan scheduler for weight decay. Our empirical results demonstrate that SWD can effectively penalize large gradient norms and improve convergence. Morcover, SWD often significantly improve generalization over constant schedulers. The generalization gap between SGD and Adam can be almost closed by such a simple weight decay scheduler for CNNs.
We emphasize that the proposed method is the first scheduled weight decay method rather than novel optimizer. Although our analysis mainly focused on Adam, SWD can be easily combined with other optimizers, including SGD and Adam variants. To best of our knowledge, this formal touch on the art of scheduling weight decay to boost weight decay. We believe that scheduled weight decay as a new line of research may inspire more theories and algorithms that will help further understanding and boosting weight decay in future. Our work just made the first step along this line.
first
10

---

Acknowledgement
MS was supported by JST CREST Grant Number JPMJCRI8A2.
References
[1] Achille, A., Rovere, M., and Soatto, S. (2017). Critical learning periods in deep networks. In International Conference on Learning Representations.
[2] An, H., Xiong, H., Li. X., Li, X., Dou, D., and Zhu, Z. (2020). Can we use gradient norm as measure of generalization error for model selection in practice?
[3] Barrett, D. and Dherin, B. (2020). Implicit gradient regularization. In International Conference on Learning Representations.
[4] Bjorck, J., Weinberger, K. Q., and Gomes, C. (2021). Understanding decoupled and early weight decay. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 6777-6785.
[5] Chen, and Gu, Q. (2018). Closing the generalization gap of adaptive gradient methods in training deep neural networks. arXiv preprint 1rXiv:1806.06663
[6] Chong, E. K., Lu, W.-S., and Zak, S. H. (2023). An introduction to optimization. John Wiley & Sons.
[7] Daneshmand, H., Kohler, J., Lucchi, A., and Hofmann, T. (2018). Escaping saddles with stochastic gradients. In International Conference on Machine Learning, pages 1155-1164.
[8] Défossez, A., Bottou, L., Bach, F., and Usumier, N. (2020). On the convergence of adam and adagrad. arXiv preprint 2003:200..
[9] Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. (2009). Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. leee.
[10] Frankle, J., Schwab, J., and Morcos, A. S. (2020). The early phase of neural network training. International Conference on Learning Representations.
[11] Geiping, J., Goldblum, M., Pope, P., Moeller, M., and Goldstein, T. (2021). Stochastic training is not necessary for generalization. In International Conference Learning Representations.
[12] Ghadimi, S. and Lan, G. (2013). Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization, 23(4):2341-2366
[13] Hanson, S. J. and Pratt, L. Y. (1989). Comparing biases for minimal network construction with propaaaopp.,toon. In Advances neural information processing systems, pages 177-185.
[14] Hardt, M., Recht, B., and Singer, Y. (2016). Train faster, generalize better: Stability of stochastic gradient descent. In International Conference on Machine Learning, pages 1225-1234.
[15] He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778.
[16] Hochreiter, S. and Schmidhuber, J. (1995). Simplifying neural nets by discovering flat minima. In Advances in neural information processing systems, pages 529-536
[17] Hochreiter, S. and Schmidhuber. J. (1997a). Flat minima. Neural Computation, 9(1::1-42.
[18] Hochreiter, S. and Schmidhuber, J. (1997b). Long short-term memory. Neural computation, 98):1735-1780.
[19] Huang, G., Liu, Z., Van Der Maaten, L., and Weinberger, K. Q. (2017). Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700-4708.

---

[20] Jastrzkebski, S., Kenton, Z., Arpit, D., Ballas, N.. Fischer, A., Bengio, Y., and Storkey, A. (2017). Three factors influencing minima sgd. arXiv preprint
[21] Jiang, Y., Neyshabur, B., Mobahi, H., Krishnan, D., and Bengio, S. (2019). Fantastic generaliza- tion measures and where to find them. In International Conference Learning Representations.
[22] Kingma, D. P. and Ba, J. (2015). Adam: A method for stochastic optimization. 3rd International Conference on Learning Representations, ICLR 2015.
[23] Krizhevsky, A. and Hinton, G. (2009). Learning multiple layers of features from tiny images.
[24] Krogh, A. and Hertz, J. A. (1992). A simple weight decay can improve generalization. In Advances in neural information processing systems, pages 950-957.
[25] LeCun, Y., Bengio, Y., and Hinton, G. (2015). Deep learning. nature, 521(7533):436.
[26] Lewkowyez, A. and Gur-Ari, G. (2020). On the training dynamics of deep networks with 1_2 regularization. Conference on Neural Information Processing Systems.
[27] Li, J., Luo, X., and Qiao, M. (2019a). On generalization error bounds of noisy gradient methods for non-convex learning. In International Conference on Learning Representations.
[28] Li, Q., Tai, C., et al. (2017). Stochastic modified equations and adaptive stochastic gradient algorithms. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pages 2101-2110. JMLR. org.
[29] Li, Y., Wei, C., and Ma, T. (2019b). Towards explaining the regularization effect of initial large learning rate in training neural networks. In Advances in Neural Information Processing Systems, pages 11669-11689 1680.
[30] Liu, L.,, Jiang, H., He, P., Chen, W., Liu, X., Gao, J., and Han, J. (2019). On the variance of the adaptive learning rate and beyond. In International Conference on Learning Representations.
[31] Loshchilov, and Hutter, (2016). Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608..33900
[32] Loshchilov, I. and Hutter, F. (2018). Decoupled weight decay regularization. In International Conference on Learning Representations.
[33] Luo, L., Xiong, Y., Liu, Y., and Sun, X. (2019). Adaptive gradient methods with dynamic bound of learning rate. 7th International Conference on Learning Representations, ICLR 2019.
[34] Marcus, M., Santorini, B., and Marcinkiewicz, M. A. (1993). Building large annotated corpus of english: The penn treebank.
[35] Reddi, S. J., Kale, S., and Kumar, S. (2019). On the convergence of adam and beyond. 6th International Conference on Learning Representations, ICLR 2018.
[36] Shapiro, A. and Wardi, Y. (1996). Convergence analysis of gradient descent stochastic algo- rithms. Journal of oplimization theory and applications, 91(2):439
[37] Simonyan, K. and Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv preprint 14X.vv1400915..
(38] Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. (2015). Going deeper with convolutions. In Proceedings of the LEEE conference on computer vision and pattern recognition, pages 1-9.
[39] Wan, R.. Zhu, Z., Zhang, X., and Sun, J. (2021). Spherical motion dynamics: Learning dynamics of normalized neural network using sgd and weight decay, Advances in Neural Information Processing Systems, 34.
[40] Wilson, A. C., Roelofs, R., Stem, M., Srebro, N., and Recht, B. (2017). The marginal value of adaptive gradient methods in machine learning. In Advances Neural Information Processing Systems, pages 4151-4161.
12

---

[41] Wolfe, P. (1969). Convergence conditions for ascent methods. SIAM review, 11(2):226-25 235..
[42] Xie, Z., He, F., Fu, S., Sato, I,, Tao, D., and Sugiyama, M. (2021a). Artificial neural variability for deep learning: On overfitting, noise memorization, and catastrophic forgetting. Neural Computation, 33(8).
[43] Xie, Z., Sato, I., and Sugiyama, M. (2021b). A diffusion theory for deep learning dynamics: Stochastic gradient descent exponentially favors flat minima. In International Conference on Learning Representations.
[44] Xie, Z., Tang. Q., Sun, M., and Li, (2023). On the overlooked structure of stochastic gradients. In Thirty-seventh Conference on Neural Information Processing Systems.
[45] Xie, Z., Tang, Q.-Y,, Cai, Y., Sun, M., and Li, P. (2022a). On the power-law spectrum in deep learning: A bridge to protein science. arXiv preprint arXiv:2201. 13011.
[46] Xie, Z., Wang, X., Zhang, H., Sato, I., and Sugiyama, M. (2022b). Adaptive inertia: Disentan- gling the effects of adaptive learning rate and momentum. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 24430-24459
[47] Xie, Z., Yuan, L., Zhu, Z.,, and Sugiyama, M. (2021c). Positive-negative momentum: Manipulat- ing stochastic gradient noise to improve generalization. In International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 1 11448-11458. PMLR.
[48] Yan, Y., Yang, T., Li, Z., Lin, Q., and Yang, Y. (2018). A unified analysis of stochastic momentum methods for deep learning, In IJCAI International Joint Conference on Artificial Intelligence.
[49] Zaheer, M., Reddi, S., Sachan, D., Kale, S., and Kumar, S. (2018). Adaptive methods for nonconvex optimization. In Advances in neural information processing systems, pages 9793-9803.
[50] Zaremba, W., Sutskever, I, and Vinyals, O. (2014). Recurrent neural network regularization. arXiv preprint arXiv: 1409.232.
[51] Zhang, C., Bengio, S., Hardt, M., Recht, B., and Vinyals, O. (2017). Understanding deep learning requires rethinking generalization. In International Conference Machine Learning-
[52] Zhang, G., Wang, C., Xu, B., and Grosse, R. (2019a). Three mechanisms of weight decay regularization. International Conference on Learning Representations.
[53] Zhang. J., He, T., Sra, S., and Jadbabaie, A. (2019b). Why gradient clipping accelerates training:
A theoretical justification for adaptivity. In International Conference on Learning Representations.
[54] Zhang, X., Xu, R., Yu, H., Zou, H., and Cui, P. (2022). Gradient norm regularizer seeks flat minima and improves generalization.
[55] Zhao, Y., Zhang, H., and Hu, X. (2022). Penalizing gradient norm for efficiently improving generalization deep learning. In International Conference on Machine Learning-
[56] Zhou, P., Feng, J., Ma, C., Xiong, C., HOI, S., et al. (2020). Towards theoretically understanding why sgd generalizes better than adam deep learning. arXiv preprint arXiv:2010.05627.
[57] Zhu, Z., Wu, J., Yu, B., Wu, L., and Ma, (2019). The anisotropic noise in stochastic gradient descent: Its behavior of escaping from sharp minima and regularization effects. In /CML, pages 7654-7663.
13

---

A Proofs
A.1 Proof of Theorem
Proof. We first write the regularized loss function corresponding to SGD with vanilla weight decay
X 2rje
[((0) = L(0)
101
(10)
at t-th step.
If the corresponding L2 regularization X is unstable during training, the regularized loss function 2n,
fe(0) will also be time-dependent function and has no non-zero stable stationary points.
Suppose we have non-zero solution 0 which a stationary point of f(0, t) at t-th step and SGD finds 0r 8" t-th step.
Even if the gradient of f(( at t-step is zero, we have the gradient at (t + 1)-th step as
9++1 = Vf++1(0*) X X( Met1 )
(11)
means that
19 1 2 -1 1008110012
(12)
777
To achieve convergence, we must have ||9++1 1| 0. requires (n. n=:112
| || 0.
Theorem 2.2 of Shapiro and Wardi 36 told us that the learning rate should small enough for convergence. Obviously, we have 7) do in practice.
As ?/ = 7 does not hold, SGD cannot converging to any non-zero stationary point.
The proof is now complete.
A.2 Proof of Theorem
Before formally completing the proof, we first introduce useful Lemma case of Theorem in Yan al. [48] with 3 = $ = O.
which is specialized
Lemma (Convergence of SGD). Assume that L(0) is an C-smooth function finctoth as L(O)* 2 L' EO, X) VL(()) 0, B[JI L(0, X) [=) s 0 | S G for
lower bounded
any 0. Let SGD optimize for t + iterations. If S VITT we have
Co
min E EL
(13)
k=
VI+T
where Co = (106) +CC(G++0O))
Proof. Given the conditions of [()) Lemma L(0) + 2 |10112
we may obtain the resulted conditions of f(0)
As L(0) is an L-smooth function, we have
(V.) (f(+a)]l = VL(02) VT(()) + X(0. - 0.)l < (C+ 1) 0oll
(14)
holds for any 0, and O,. It shows that f()) is an (C + A)-smooth function.
As L is lower bounded as L(O)= 2 L", we have
8 2*
(15)
It means that [Va) (L(0))] S L100, 0b] holds for any 0, and 0s.
14

---

As - VL(0)] = O, we have
((, x) - V = -.
(16)
VL(0) A
82, have
[(I - - -V
(17)
As |]T[()]] 2 G. we have
| = ( + X0 - G + |||0|maxx
(18)
where |00| is the maximum Le norm of any 0.
Introducing the derived conditions Eq. (12) (16) for f into Lemma 1 may treat the objective
optimized by SGD. Then we have
min B (
f(0o) f C
+ +CC + + X100 mmxx) =
o4)
(19)
k=
vt
L(06) + 1116.11
+ CC(+ + A|00 max )² 0²)
vt +++
C
(20)
Obviously, the gradient norm upper bound in convergence analysis monotonically increases as the weight decay strength A.
The proof is complete.
B Experimental Details
Computational environment. The experiments are conducted on computing cluster with GPUs of NVIDIA Tesla" P100 16GB and CPUs Intel Xeon @ CPU E5-2644 2.60GHz.
B.1 Image Classification on CIFAR-10 and CIFAR-100
Data Preprocessing For CIFAR-10 and CIFAR-100: We perform the common per-pixel vero-mean unit-variance normalization, horizontal random flip, and 32 x 32 random crops after padding with 4 pixels on each side.
Hyperparameter Settings: We select the optimal learning rate for each experiment from (0.0001, 0.001, 0.01, 0.1, 1, 10) for non-adaptive gradient methods. We use the default learning rate for adaptive gradient methods in the experiments of Table [ while we also compared Adam, AdamW, AdamS under various learning rates and batch sizes in other experiments. In the experiments on CIFAF-10 and CIFAR-100:
0.1 for SGD and SGDS; 7j = 0.001 for Adam, AdamW, AdamS, AMSGrad, Yogi, AdaBound, and RAdam: r7 0.01 for Padam. For the learning rate schedule, the learning rate is divided by 10 at the epoch of (80, 160} for CIFAR-10 and {100, 150} for CIFAR-100 respectively. The batch size is
128 for both CIFAR-I0 and CIFAR-100.
The strength of L2 regularization and SWD is default to 0.0005 as the baseline. Considering the dz2, Thus, the weight decay of AdamW uses Aw 0.5 linear scaling rule, we choose Aw "
for CIFAR-10 and CIFAR-100. The basic principle of choosing weight decay strength is to let all optimizers have similar convergence speed.
We set the momentum hyperparameter 31 = 0.9 for SGD and SGDS. As for other optimizer hyperparameters, we apply the default hyperparameter settings directly.
We repeated each experiment for three times in the presence of the error bars.
We leave the empirical results with the weight decay setting À 0.0001 in Appendix C
15

---

Table 2: Test performance comparison of optimizers with AL, Xg 0.0001 and Aw 0.1, which a common weight decay setting in related papers. AdamS still show better test performance than popular adaptive gradient methods and SGD.
DAIASET
MODEL
SGD
ADAMS
ADAM
AMSGRAD
ADAMW
AUABOUND
PAUAM
YUGI
RADAM
CIFAR-10
RESNET18
6.58
4.69
5.72
5.33
6.87
5.83
5.43
5.81
VGG16
6.92
6.16
7.04
6.68
6.45
7:33
8.74
6.68)
6.73
CIFAR-100
RESNET34
24.92
23.50
25.56
24.74
23.61
25.67
25.39
23. 23.72
25.65
DENSENET121
20.98
21.35
21.39
22.23
24.23
22.26
22
22 22.40
GOUGLENEr
21.89
21.60
24.60
24.05
21.71
25.08
26.09
22.56
22.35
B.2 Image classification on ImageNet
Data Preprocessing For ImageNet: For ImageNet, we perform the per-pixel zero-mean unit- variance normalization, horizontal random flip, and the resized random crops where the random size (of 0.08 to 1.0) of the original size and random aspect ratio (of to ()) of the original aspect ratio is made.
Hyperparameter Settings for ImageNet: We select the optimal learning rate for each experiment from {0.0001, 0.001, 0.01, 0.1, 10} for all tested optimizers. For the learning rate schedule, the learning rate is divided by 10 at the epoch of (30, 60}. We train each model for 90 epochs. The batch size set to 256. The weight decay hyperparameter of AdamS, Adam W, Adam are chosen from {5 x 10-6, x 10 10-5,5 x 10-1,5 1, x 10 10-3,3 5, x 10-23. As for other optimizer hyperparameters, we still apply the default hyperparameter settings directly.
B.3 Language Modeling
We use a classical language model, Long Short-Term Memory (LSTM) [18 with 2 layers, 512 embedding dimensions, and 512 hidden dimensions, which has 14 million model parameters and is similar to the "medium LSTM" in Zaremba et al. [50]. Note that our baseline performance is better than the reported baseline performance in Zaremba al. [50]. The benchmark task is the wevd-level Penn TreeBank [34]. We empirically compared AdamS, A, and Adam under the common and same conditions.
Hyperparameter Settings. Batch Size: B = 20. BPTT Size: bpl/ = 35. Learning Rate: = 0.001. We run the experiments under various weight decay selected from {10-1, 5 10-5 10-5, 5 10-6, 10-6,10-6 5,5 10-7, 10-7).The dropout probability is set to 0.5. We clipped gradient norm to 1.
C
Supplementary Figures and Results of Adaptive Gradient Methods
Popular Adam variants often generalize worse than SGD. A few Adam variants tried to fix the hidden problems in adaptive gradient methods, including AdamW Loshchilov and Hutter [2) AMSGrad [35] and Yogi 49 recent line of research, such as AdaBound [33, Padam [3]. and
RAdam 30 believes controlling the adaptivity of learning rates may improve generalization. This line of research usually introduces extra hyperparameters to control the adaptivity, which requires more efforts in tuning hyperparameters. However, we and Zhang al. (53) found that this argument is contradicted with our comparative experimental results (see Table T In our empirical analysis, most advanced Adam variants may narrow but not completely close the generalization gap between adaptive gradient methods and SGD. SGD with fair weight decay hyperparameter as the baseline performance usually generalizes better than recent adaptive gradient methods. The main problem may lie in weight decay. SGD with weight decay x = 0.0001, a common setting in related papers. is often a good baseline, as 0.0005 often shows better generalization on CIFAR- 10 and CIFAR-100 We also conduct comparative experiments with À 0.0001. Under the setting À = 0.0001, while some existing Adam variants may outperform SGD sometimes due to the lower baseline performance of SGD, AdamS shows superior test performance. For example, for ResNet18 8 on CIFAR-10, the test error of AdamS is lower than SGD by nearly one point and no other Adam variant may compare with AdamS.
16

---

Adam
95
90
85
65
Weight Detay
Figure 13: Language modeling under various weight decay. Note that the lower perplexity better.
Adamw/
0.11
Adam
AMSörbo
AdsBooun
0.09
Yogi
RAo sm
0.07
Acam
AMSGrrd
0.08
0.06
Andiound
0.05
0.04
0.06
(a) ResNet18
(b) VGG16
Adami/
Adam
0.30
AMSGrao
0.30
Adelloure
Fodamm
Tw Www
Yogi
RAdam
0.28
0.26
Adam
AcaBound
Padam
1.00
125
150
1.00
1.25
150
Epochs
(c) DenseNet121
(d) GoogLeNet
Figure 14: The learning curves of adaptive gradient methods.
Language Modeling. It is well-known that, different from computer vision tasks, the standard Adam (with L2 regularization) is the most popular optimizer for language models. Figure 13 in Appendix Cdemonstrates that the conventional belief is true that the standard T.2 regularization yields better test results than both Decoupled Weight Decay and SWD. The weight decay scheduler suitable for language models is an open problem.
We report the learning curves of all adaptive gradient methods in Figure 14 They shows that vanilla Adam with SWD can outperform other complex variants of Adam.
Figure 13 displays the scatter plot of training losses and test errors during final 40 epochs of training DenseNet122 21 CIFAR-100.
(6 the test performance AdamS, AdamW, and Adam under various weight decay hyperparameters of ResNet34 on CIFAR-100.
17

---

10
10
Adamo AdamN
Adars AdamNN Adam
Trasiing
10.7
Traaiing
20
30-2
10-2
Figure 15: Even if with similar or higher training losses, AdamS still generalizes better than Adam W and Adam. The scatter plot of training losses and test errors during final 50 epochs of training VGG16 on CIFAR-10 10 and DenseNet121 on CIFAR-100.
0.375
Adam5
0.350
Adam
Adam
0.325
Ener
0.500
0.275
10-
10:4
10-
10
10
10
Weight Decay
Figure 16: We compare the generalization of Adam, AdamW, and AdamS with various weight decay rates by training ResNet34 on CIFAR-100. The displayed weight decay of Adam W in the figure has been rescaled by the factor = 0.001. The optimal test performance of AdamS is significantly better than Adam W and Adam.
0.14
Adamm
0.12
Adbm
Erer
0.08
0.04
Epochs
Figure 17: We train ResNet18 CIFAR-10 for 900 epochs explore the performance limit of
AdamS, Adam W. and Adam. The learning rate is divided by 10 at the epoch of 300 and 600, AdamS achieves the most optimal test error, 4.70%.
18

---

0.28
0.26
0.24
Epochs
Figure 18: ResNet50 ImageNet. The lowest Top-1 test errors of AdamS, Adam W, and Adam 24.19%, 24.29%, and 30.07%, respectively.
Table 3: In the experiment of ResNetl 8 trained via SGD CIFAR-10, we verified that the optimal weight decay is approximately inverse to the number of epochs. The predicted optimal weight decay approximately 0.1 Epochs because the optimal weight decay is 0.0005 selected from {10-2. x 10 10-3,5 x 10-4, 10 5 x 10 10-50- 5 x 10 10 10-6) with 200 epochs as the base case. The observed optimal weight decay is selected from {Fpochs 0.1 x Epochs 0.01 x Epochs We observed that the optimal test errors are all corresponding to the predicted optimal weight decay À = 0.1 x Epochs At least in the sense of the order of magnitude, the predicted optimal weight decay fully consistent with the observed optimal weight decay. Thus, the empirical results supports that the optimal weight decay is approximately inverse to the number of epochs in the common range the number epochs.
EPOCHS
À Epochs-
A 0.1 x Epochs 1
À = 0.01 x Epochs"
50
74.06
7.12
7.50
100
22.04
5.56
6.01
200
11.81
5.02
5.61
1000
4.67
4.43
6.02
2000
4.59
4.48
5.70
We train ResNet18 on CIFAR- 10 for 900 epochs to explore the performance limit of AdamS, Adam and Adam in Figure 17
Figure 18 in Appendix shows that, AdamS can make marginal improvements over Adam W.
Adamr
10a
Adumlyy W
Adeloundd Padam Yog! RAdom
10
0.20
Figure 19: The scatter plot of training losses and test errors during final 40 epochs of training ResNet34 on CIFAR- Even with similar or higher training losses, AdamS still generalizes better than other Adam variants. We leave the scatter plot on CIFAR-10 10 in Appendix C
19

---

D Additional Algorithms
We that the implementation of AMSGrad in Algorithm 3 is the popular implementation in PyTorch. We use the PyTorch implementation in our paper, as it is widely used in practice.
Algorithm 3: AMSGind/AMMSGrrda
Algorithm 4: AMSGradS
9t. VL 1) A0,-1;
( 1): m4 8im4- + -811
my = B1m4 1 + (1 gb:
R2 8204-1 + (1 f2)
Pt 3204 + +(1 -22)
m, = I-
m, I-
Umar = maz Ut, Umax):
Umax = (er Umax):
6. 1-32
1-
11 mh 1700-11 0, = 0 Votte
6, = mean (0));
P Voete
mc
V0. 20
E Supplementary Experiments with Cosine Annealing Schedulers and Warm Restarts
In this section, we conducted comparative experiments on AdamS, AdamW, and Adam in the presence of cosine annealing schedulers and warm restarts proposed by Loshchilov and Hutter [31]. We set the learning rate scheduler with a recommended setting of Loshchilov and Hutter [I:: To = 14 and mut 2. The number of total epochs is 210. Thus, trained each deep network for four runs warm restarts, where the four runs have 14, 28, 56, and 112 epochs, respectively. Other hyperparameters and details are displayed in Appendix
We conducted comparative experiments on AdamS, AdamW, and Adam in the presence of cosine annealing schedulers and warm restarts proposed by Loshchilov and Hutter (31] We set the learning rate scheduler with a recommended setting of Loshchilov and Hutter (31) Our experimental results in Figures 12 and 20 suggest that AdamS consistently outperforms AdamW and Adam in the presence of cosine annealing schedulers and warm restarts. It demonstrates that, with various learning rate schedulers, the advantage of SWD may generally hold.
Moreover, did not empirically observe that cosine annealing schedulers with warm restarts may consistently outperform the common snceeiss-cconstne learning rate schedulers for adaptive gradient methods. We noticed that Loshchilov and Hutter [3I] empirically compared four- sour-staged piecewise- constant learning rate schedulers with cosine annealing schedulers with warm restarts, and argue that cosine annealing schedulers with warm restarts are better. There may be two possible causes. First, three-staged pnecewise-constan learning rate schedulers, which usually have a longer first stage and decay learning rates by multiplying 0.1, are the recommended settings, while the four-staged piecewise-constant learning rate schedulers in Loshchilov and Hutter [1] are usually not optimal. Second, warm restarts may be helpful, while cosine annealing may be not. The ablation study on piecewise-constant learning rate schedulers with warm restarts is lacked. We argued that how to choose learning rate schedulers may still be an open question, considering the complex choices of schedulers and the complex loss landscapes.
20

---

0.12
0.12
Adam5
Adam5
0.11
Adaml)
0.11
Ademi/
0.10
0.10
0.09
0.09
0.08
0.08
0.07
0.07
0.06
0.06
0.05
0.05
0.04
0.04
Weight Docay
Weight Decay
ResNet18
(b) VGG16
The test emrrr of ResNet18 and VGG16 CIFAR-10 under various weight decay with cosine annealing and warm restart schedulers. AdamS yields significantly better optimal test performance than AdamW and Adam.
Figure 20:
21


Reviews:
[{"review_id": "R392UsvoSF", "reviewer": "Reviewer_e7t4", "summary": "I would divide this paper's contributions into two parts.  The first part is an algorithm (a variant of Adam) which the authors argue generalizes better than Adam/AdamW and is easier to tune.  The second part is the justification for the effectiveness of that algorithm.\n\n**The algorithm itself** The proposed algorithm, AdamS, is similar to AdamW, except that the weight decay strength is divided by the mean of the current squared gradient EMA.  This is effectively a schedule for weight decay.  It has the effect of penalizing the overall weights less strongly during phases of training when the overall gradient is large, and penalizing the overall weights more strongly during phases of training when the overall gradient is small.   For example, Figure 3 shows gradient norm and weight decay strength during the training of ResNet-34 on CIFAR-10; after the first learning rate drop, squared gradient norm grows, which causes the weight decay strength to shrink; after the second learning rate drop, squared learning rate plummets, which causes weight decay to rise.\n\n**Experimental evaluation** Table 1 and section 5 experimentally evaluate AdamS and argue that the algorithm is better generalizing / easier to tune than AdamW.\n\n**Justification**: The authors justify their algorithm along the following lines (which I disagree with; see more below):\n\n1. When the learning rate is potentially changing, gradient descent with weight decay has trouble converging to stationary points - in particular, the authors argue (Theorem 1) that in this setting, the gradient norm will be lower bounded near stationary points.  Therefore, the authors argue that weight decay causes the gradient to be large, especially at the end of training.\n2.  The authors argue that large gradients are bad for generalization.\n3. As a consequence of #1 and #2, AdamW-style weight decay is bad for generalization (since it causes large gradients near stationary points, and large gradients are bad for generalization)\n\nThe authors acknowledge that weight decay on networks with normalization layers has been shown in the literature to control the effective learning rate, but insist that their paper's mechanism is unrelated to this, and propose an entirely different mechanism.\n\n---\n\n**post-rebuttal update**\n\nAfter discussion with the authors, I'm raising my score from 3 (\"reject\") to 6 (\"weak accept\").    Firstly, it's clear that my original review was wrong; I had argued that the proposed algorithm's benefits are probably due to the interaction between weight decay and normalization layers, but the authors have correctly pointed out that the same phenomena occur on networks without normalization layers, and I have verified this myself.   The reason for 6 as opposed to 7 is that I believe that the proposed theoretical explanation for why weight decay would cause large gradients is not the real explanation (reviewer xE1B also didn't like this theoretical portion of the paper).  That said, it's entirely plausible that weight decay causing large gradients is one of the *many* aspects of deep learning that is just theoretically unexplainable at the present time.  So, my issue with the submission is less that the paper fails to include a correct theoretical justification, and more that it includes a faulty theoretical justification; I think it is better to have no justification at all than to have a faulty justification.  \n\nThat being said, on the positive side, it seems there is a good chance that the proposed algorithm is a worthwhile addition to the deep learning toolbox (though I am not an expert at evaluating this type of contribution).", "questions": "How do the authors respond to my critique above?", "limitations": "discussed above", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I'm not an expert in experimentally evaluating the performance of neural network training algorithms, but from the experiments in section 5 it does seem plausible to me that AdamS is indeed better / easier to tune than AdamW.", "weaknesses": "While I can accept that the algorithm may be a good idea, I strongly disagree with the proposed justification given in the paper.  I believe the effectiveness of the algorithm is probably unrelated to the justification given in the paper, and is instead closely linked to the implicit effects of weight decay on effective learning rate when the network has normalization layers.\n\nFirst, I would note that all of the experiments in the paper are on networks with normalization layers.  The authors argue in Figure 2 that VGG-16 is \"not scale invariant\" but the VGG-16 architecture does have many BatchNorm layers, even though it is not fully scale-invariant, so I believe that the literature on scale invariance is still quite related to what is going on with the VGG-16.\n\nFor networks with normalization layers, there is a clear mechanism by which weight decay causes large gradients.  If $L$ is the loss for a scale-invariant network, then $\\nabla L(c \\theta) = \\frac{1}{c} \\nabla L(\\theta)$, i.e. scaling down the weights $(c < 1)$ will automatically scale up the gradients (see Lemma 1 here: http://www.offconvex.org/2020/04/24/ExpLR1/).  The large gradients have been reported to cause the 'effective learning rate' of training to be higher, which is thought to be good for generalization.  My intuition for the mechanism in this paper is: if the gradient norm is too large, our effective learning rate will be too large for the algorithm to converge, so we need to decrease weight decay so the gradient norms come back down; if the gradient norm is too small, our effective learning rate will be too small (which makes convergence slow and is bad for generalization) so we need to increase weight decay so that the gradient norm moves back up.\n\nBy contrast, the proposed justification doesn't really make sense to me.  Figure 2 shows that higher weight decay causes growth in the gradient norm _in the middle of training_, not at convergence.  This is much more easily explained by scale-invariance (for the VGG-16, the scale invariance of certain layers) than by Theorems 1 and 2.\n\nFigure 5 shows that weight decay increases the eigenvalue spectrum at convergence.  I believe this is related to cross-entropy loss, and wouldn't happen with e.g. squared loss.  For cross-entropy loss, the Hessian is small when the margins are large.  Adding weight decay prevents the margins from becoming large and therefore the Hessian from shrinking.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate Reviewer e7t4 for hard work and admitting the value of our contribution. \n\nWe notice that Reviewer e7t4 currently tends to reject our work only because the reviewer believes an alternative scale-invariance mechanism of our experiments. However, we respectfully note that the justification of Reviewer e7t4 made a key factual mistake, which misled Reviewer e7t4’s rating on our work.\n\nStandard VGG-16 (in the original paper and our paper) actually has no BatchNorm layers, while sometimes the so-called VGG-16-bn (with BatchNorm) is used in other papers. Thus, as we mentioned in the paper, scale invariance itself cannot explain the observed pitfalls of weight decay. So the belief that SWD is unrelated to large gradient norms and the associated ``evidence” are both inconvincible.\n\nWe have duly addressed your justifications as follows.\n\nQ1: I believe the effectiveness of the algorithm is probably unrelated to the justification given in the paper, and is instead closely linked to the implicit effects of weight decay on effective learning rate when the network has normalization layers.\n\nA1: The belief that SWD is not related to mitigating large gradient norms has no true evidence. In contrast, we showed that SWD mitigates the overlooked pitfalls of weight decay with empirical and theoretical evidence. We did not observe that SWD or the pitfalls behave qualitatively different on ResNet (with BatchNorm) or VGG(without BatchNorm).\n\nQ2: I would note that all of the experiments in the paper are on networks with normalization layers. The authors argue in Figure 2 that VGG-16 is \"not scale invariant\" but the VGG-16 architecture does have many BatchNorm layers, even though it is not fully scale-invariant, so I believe that the literature on scale invariance is still quite related to what is going on with the VGG-16.\n\nA2: We respectfully point out that the first judgement is wrong. Standard VGG which we used is indeed not scale invariant and has no BatchNorm layer.\n\nQ3: My intuition for the mechanism in this paper is: if the gradient norm is too large, our effective learning rate will be too large for the algorithm to converge, so we need to decrease weight decay so the gradient norms come back down; if the gradient norm is too small, our effective learning rate will be too small (which makes convergence slow and is bad for generalization) so we need to increase weight decay so that the gradient norm moves back up. By contrast, the proposed justification doesn't really make sense to me.\n\nA3: We totally agree with the intuition (except that it only connects to scale invariance). Moreover, this intuition is actually NOT contradicted to the large-gradient norm pitfall. The large-gradient-norm pitfall is exactly a signal of poor convergence and generalization at the final training phase. The proposed SWD indeed automatically adjusts the strength of weight decay and then successfully mitigate the large gradient norms as well as poor convergence/generalization. \n\nQ4: Figure 2 shows that higher weight decay causes growth in the gradient norm in the middle of training, not at convergence. This is much more easily explained by scale-invariance (for the VGG-16, the scale invariance of certain layers) than by Theorems 1 and 2.\n\nA4: Figure 2 shows that weight decay causes growth in the gradient norm in both the middle phase and final phase of training, while the gradient norm at the final phase is relatively lower. This can be explained by Theorems 1 and 2, but cannot by explained scale invariance only (the VGG-16 has no scale-invariant layer). Moreover, the gradient norm growth at the final phase is more interesting, because it closely relates to poor convergence and generalization. \n\n \nQ5: Figure 5 shows that weight decay increases the eigenvalue spectrum at convergence. I believe this is related to cross-entropy loss, and wouldn't happen with e.g. squared loss. For cross-entropy loss, the Hessian is small when the margins are large. Adding weight decay prevents the margins from becoming large and therefore the Hessian from shrinking.\n\nA5: Thanks for the comments. We just reported the interesting observation that the Hessian eigenspectrum and minima sharpness are significantly affected by SWD. Your comment may be right but does not indicate any weakness of our work. \n\nFinally, we sincerely thanks Reviewer e7t4 for the hard work and comments again. \n\nWe strongly encourage the reviewer to re-evaluate our work without the distraction of misunderstanding and factual mistake. \n\nWe appreciate it very much in advance.\n"}, {"review_id": "1wPQ9a6Y7M", "reviewer": "Reviewer_xE1B", "summary": "This paper studies the role of weight decay and its connection with large gradient norms in deep learning settings. In particular, the paper highlights differences in variants of weight decay and also the effect of weight decay on gradient norm in the final phase. Based on the observation that weight decay yields large gradient norms, the authors propose a scheduler for weight decay, called the Scheduled Weight Decay, which dynamically adjusts the weight decay strength.", "questions": "Please refer to the concerns raised in the weakness section.", "limitations": "Not applicable.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "The paper is reasonably well-written and the motivation for studying the effect of weight decay is both important and clear. The approach is easy to implement and the results on some smaller scale benchmarks seem encouraging.", "weaknesses": "(1) The theoretical results are weak and not very interesting. Theorem 2 follows from standard convergence rates for SGD. I could not find any novelty in this result. The authors should highlight the novelty of the result and proof technique in the main paper.\n\n(2) While Theorem 1 seems interesting, it seems somewhat weak. The lower bound on the difference in learning rate is somewhat artificial and unreasonable. For instance, consider decreasing learning rates of SGD typically used in machine learning settings i.e., eta_t >= eta_{t+1}. Then Theorem 1 assumes delta <= eta_t - eta_{t+1}. This implies eta_0 >= delta . t for any t > 0. This makes sense if delta is going to 0 as t -> infinity. Maybe I am missing something but would really appreciate it if the authors can comment about this. \n\n(3) Theorem 1 & 2 seem to analyze gradients of two different functions, which doesn't seem like a proper comparison. More remarks regarding this will be valuable. Furthermore, I would like to get clarification if the y-axis in plots of Figure 2 are gradient norms with regularization included.\n\n(4) Definition 1 is very unclear. The authors should provide more remarks around definition 1 to help readers understand the definition better.\n\n(5) The dependence of C_2 on sup norm of theta is somewhat weird in Theorem 2. Are you assuming ||theta|| is bounded? (otherwise rewrite the theorem statement without this dependence) .\n\n(6) I think the theoretical basis for Gradient-Norm-Aware Scheduled Weight Decay is weak as presented in the paper. While I understand the intuition, it is not clear why this particular proposal is meaningful. Since the theoretical basis for this is lacking, I would expect much more comprehensive experiments to support the method. Unfortunately, the empirical analysis in the paper is somewhat limited.\n\nLine 20: eta -> eta_t is the learning rate & Line 125: E[||∇L(θ, X) − ∇L(θ)||^2] ≤ \\sigma^2?\n\n\nPost-rebuttal\n==========\nI think the theoretical analysis is still not convincing but given that these observations are interesting, I am slightly increasing the score.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate Reviewer xE1B’s hard work and helpful comments.\n\nThe comments shows that the reviewer only has concerns about theoretical evidences in our work. Our theoretical analysis is proposed not as a main contribution but to explain our interesting findings. \n\nOur main contributions are revealing the overlooked pitfalls of weight decay and how to mitigate them. If a simple theoretical mechanism can well describe the pitfalls, it will be an advantage rather than a weakness. Given the importance of weight decay, our contributions are significant. \n\nIf the reviewer indeed accepts the reported overlooked pitfalls and the effectiveness of SWD, we strongly encourage the reviewer to re-evaluate the importance of our contributions.\n\nWe also try to address the mentioned weaknesses about theoretical evidences as follows.\n\nQ1: Theorem 2 follows from standard convergence rates for SGD. I could not find any novelty in this result. The authors should highlight the novelty of the result and proof technique in the main paper.\n\nA1: We frankly admit that Theorem 2 can be easily obtained from standard convergence rates with the modification of weight decay. However, we believe that the simplicity of theoretical analysis is not a weakness. Again, our theoretical analysis is proposed not as a main contribution but to explain our interesting findings, namely the large-gradient-norm pitfall and SWD. We argue with Ockham's Razor that if a simple theory can explain an interesting novel finding, it will be unnecessary to pursue a complex theory with more tricks.\n\nQ2: While Theorem 1 seems interesting, it seems somewhat weak. The lower bound on the difference in learning rate is somewhat artificial and unreasonable. For instance, consider decreasing learning rates of SGD typically used in machine learning settings i.e., eta_t >= eta_{t+1}. Then Theorem 1 assumes delta <= eta_t - eta_{t+1}. This implies eta_0 >= delta . t for any t > 0. This makes sense if delta is going to 0 as t -> infinity. Maybe I am missing something but would really appreciate it if the authors can comment about this.\n\nA2: We would sincerely apologize if we misunderstood your question. In principle, if t approaches infinity and $\\eta_{t}$ approaches, $delta$ and the lower bound indeed will approach to zero. However, in practice with finite iterations and a learning rate schedule, people can only have a finite learning rate. More importantly, Theorem 1 suggests that adaptive optimizaters can lead to a more significant large-gradient-norm pitfall, as $\\delta$ as well as its preconditioned learning rate $\\frac{\\eta_{t}}{\\sqrt{v_{t}}}$ can be unstable and large during training.\n\nQ3: Theorem 1 & 2 seem to analyze gradients of two different functions, which doesn't seem like a proper comparison. More remarks regarding this will be valuable. Furthermore, I would like to get clarification if the y-axis in plots of Figure 2 are gradient norms with regularization included.\n\nA3: Thanks for the constructive comment. Theorem 1 and 2 both analyze the regularized loss function $f(\\theta)$, so the messages carried by them are consistent. We note that the expression of $f(\\theta)$ can be slightly different due to the types of weight decay. We use the form of vanilla weight decay in Theorem 1 and use the form standard $L_{2}$ regularization/decoupled weight decay in Theorem 2, where $L_{2}$ regularization and decoupled weight decay are identical and common for SGD. We will follow your suggestion to make them more clear in the revision. \n\nFigure 2 plots the gradient norms without regularization included. However, the empirical conclusions are same with or without regularization, because the numerical difference due to regularization is multiple orders of magnitude smaller than their gradient norms in our experiments.\n\nQ4: Definition 1 is very unclear. The authors should provide more remarks around definition 1 to help readers understand the definition better.\n\nA4: Thanks for the constructive comment. We will provide more remarks and propose a more formal Definition 1.\n\nQ5: I think the theoretical basis for Gradient-Norm-Aware Scheduled Weight Decay is weak as presented in the paper. While I understand the intuition, it is not clear why this particular proposal is meaningful. Since the theoretical basis for this is lacking, I would expect much more comprehensive experiments to support the method. Unfortunately, the empirical analysis in the paper is somewhat limited.\n\nA5: We respectfully argue that the prior purpose of our theoretical analysis is to demonstrate the existence of the overlooked pitfalls of weight decay, because identifying the overlooked large-gradient-norm pitfall of weight decay is our first and most valuable contribution. After the demonstration of the overlooked pitfalls, the proposed Scheduled Weight Decay (SWD) method is a naturally inspired algorithm for mitigating the large-gradient-norm pitfall. The effectiveness of SWD for mitigating the overlooked pitfalls are extremely significant (e.g., Figures 3 and 4). We admit that we have no rigous generalization bound theory of scheduling weight decay at present. Formal theories of analyzing weight decay and learning rate schedulers are interesting. We will leave developing formal theories of analyzing weight decay schedulers as future explorations. \n\nQ6: Typos.\n\nA6: Thank a lot for pointing out the typos. We will correct them in the revision.\n"}, {"review_id": "8o5FsU9dVb", "reviewer": "Reviewer_3dfT", "summary": "This paper studies the overlooked pitfalls of weight decay, a regularization technique used in deep neural networks (DNNs). The authors discovered that weight decay can lead to large gradient norms, particularly at the final phase of training, often indicating poor convergence and generalization. To address this issue, the authors propose a novel method called Scheduled Weight Decay (SWD), which dynamically adjusts weight decay strength according to the gradient norm and penalizes large gradient norms during training. The paper concludes that the SWD approach outperforms the conventional constant weight decay strategy, especially for the Adaptive Moment Estimation (Adam) optimization algorithm.", "questions": "How does SWD affect the training time, given the additional complexity introduced by dynamically adjusting the weight decay?", "limitations": "Not applicable.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. SWD dynamically adjusts weight decay, which can help to avoid large gradient norms that can lead to poor convergence and generalization. The paper demonstrates that SWD outperforms the traditional constant weight decay, especially in Adam optimization, potentially leading to better model performance.\n2. The concept is simple to understand and can be easily implemented in most deep learning frameworks.", "weaknesses": "1. The proposal is specific to weight decay regularization and might not generalize to other regularization techniques.\n2. SWD's performance might depend heavily on how well the scheduling function is chosen, which can be a complex task.\n3. The paper doesn't discuss how SWD might affect the training time, and adjusting weight decay dynamically could potentially increase computational complexity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We highly appreciate Reviewer 3dfT’s kind support and helpful comments.\n\nWe gratefully hope the reviewer can express and insist on your opinions, which may help our community understand and employ weight decay better via our work.\n\nWe also properly respond to your concerns as follows.\n\nQ1: The proposal is specific to weight decay regularization and might not generalize to other regularization techniques.\n\nA1: Thanks for the comment. It is true, while the value of our method is still significant given the importance and popularity of weight decay. \n\nQ2: SWD's performance might depend heavily on how well the scheduling function is chosen, which can be a complex task.\n\nA2: We respectfully note that SWD itself is an adaptive weight decay scheduler which directly depends on gradient norms and requires no further manual design. We also agree that it will be possible to design other more complex scheduler for weight decay in future.\n\nQ3: How does SWD affect the training time, given the additional complexity introduced by dynamically adjusting the weight decay?\n\nA3: In Line 201, we mentioned that the cost difference of AdamS versus AdamW/Adam is nearly ignorable (usually less than 5%) in practice. This is not surprising because computing mean(v) is very cheap compared with computing gradients per interation. We will emphasize this point more clearly in the revision.\n"}, {"review_id": "eLiNRvQgjM", "reviewer": "Reviewer_tfGK", "summary": "The authors propose Scheduled Weight Decay (SWD), a method that mitigates the large gradient norm issue caused by constant weight decay factors. The authors demonstrate that SWD can improve the generalization performance of Adam and outperform other adaptive optimizers on CIFAR-10/100 datasets.", "questions": "See above.", "limitations": "The authors have discussed the limitations of the proposed method on datasets other than CIFAR-10/100.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The proposed method is simple and easy to implement. Moreover, it addresses the long-standing generalization gap between adaptive optimizers and SGD on certain tasks.\n2. The authors provide a theoretical analysis of the problem of unstable stationary points.\n3. The authors conduct extensive experiments to validate their claims and compare their method with other optimizers.", "weaknesses": "1. The gradient analyzed in Theorem 2 seems to include the gradient of L2 regularization, which is not included in the empirical analysis (e.g. Fig. 2).\n2. Table 1 in appendix and table 1 in the main text show very different results, especially for SGD. The reason for this difference should be made more clear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We highly appreciate Reviewer tfGK’s kind support and helpful comments.\n\nThe reviewer definitely realized the important value of identifying and mitigating the overlooked but serious pitfalls of weight decay. We gratefully hope the reviewer can express and insist on your opinion to avoid a possible loss of our community.\n\nWe also properly addressed your concerns as follows.\n\nQ1: The gradient analyzed in Theorem 2 seems to include the gradient of L2 regularization, which is not included in the empirical analysis (e.g. Fig. 2). \n\nA1: Thanks for the helpful comment. We will discuss this point in the revision. Figure 2 indeed plots the gradient norms without regularization included. However, we argue that the empirical conclusions are same with or without regularization. because the numerical difference due to regularization is multiple orders of magnitude smaller than their gradient norms in our experiments. \n\n\nQ2: Table 1 in appendix and table 1 in the main text show very different results, especially for SGD. The reason for this difference should be made more clear.\n\nA2: Again, thanks for the helpful comment. The difference between two Tables 1 in the main text and the appendix lies in the hyperparameter choice of weight decay. We discuss why some previous papers’ results are more closed to Table 1 in the appendix. Some previous papers designed novel adaptive optimizers and claimed they generalize as well as SGD only because an improper weight decay strength is chosen for SGD. We have discussed this point in Line 85-101 of the appendix. We will follow your suggestion to present the point more clearly. \n"}]

Author rebuttal:

Meta-review: This paper looks at pitfalls of current approaches using weight decay. The theory for this paper is a little dubious and is not rigorous enough (see for instance review of Reviewer xE1B). Furthermore, algorithm is not well motivated. However, I think the observations are interesting and are worth being carefully examined by the community. For this reason, I recommend acceptance contingent on the commitment that the theoretical analysis and motivation will be carefully examined and discussed based on xE1B's review.
Decision: Accept (poster)