Understanding and Scheduling Weight Decay
Authors: Zeke Xie
Issei Sato
Masashi Sugiyama
Venue: ICLR 2022 Submitted (2022)
Keywords: Weight Decay
Regularization
Optimization
Deep Learning

Abstract: Weight decay is a popular and even necessary regularization technique for training deep neural networks that generalize well. Previous work usually interpreted weight decay as a Gaussian prior from the Bayesian perspective. However, weight decay sometimes shows mysterious behaviors beyond the conventional understanding. For example, the optimal weight decay value tends to be zero given long enough training time. Moreover, existing work typically failed to recognize the importance of scheduling weight decay during training. Our work aims at theoretically understanding novel behaviors of weight decay and designing schedulers for weight decay in deep learning. This paper mainly has three contributions. First, we propose a novel theoretical interpretation of weight decay from the perspective of learning dynamics. Second, we propose a novel weight-decay linear scaling rule for large-batch training that proportionally increases weight decay rather than the learning rate as the batch size increases. Third, we provide an effective learning-rate-aware scheduler for weight decay, called the Stable Weight Decay (SWD) method, which, to the best of our knowledge, is the first practical design for weight decay scheduling. In our various experiments, the SWD method often makes improvements over $L_{2}$ Regularization and Decoupled Weight Decay.

Under review as a conference paper at ICLR 2022
UNDERSTANDING AND
A SCHEDULING WEIGHT DECAY
Anonymous authors Paper under double-blind review
ABSTRACT
Weight decay is popular and even necessary regularization technique for train- ing deep neural networks that generalize well. Previous work usually interpreted weight decay Gaussian prior from the Bayesian perspective. However, weight decay sometimes shows mysterious behaviors beyond the conventional understand- ing. For example, the optimal weight decay value tends be zero given long enough training time. Moreover, existing work typically failed to recognize the importance scheduling weight decay during training. Our work aims theoreti- cally understanding novel behaviors of weight decay and designing schedulers for weight decay deep learning. This paper mainly has three contributions. First, we propose novel theoretical interpretation of weight decay from the perspective of learning dynamics. Second, we propose novel weight-decay linear scaling rule for large-batch training that proportionally increases weight decay rather than the learning rate as the batch size increases. Third, we provide an effective learning- rate-aware scheduler for weight decay, called the Stable Weight Decay (SWD) method, which, to the best our knowledge, is the first practical design for weight decay scheduling. In our various experiments, the SWD) method often makes improvements over L2 Regularization and Decoupled Weight Decay.
INTRODUCTION
Weight decay is popular and even necessary regularization technique for training deep neural networks that generalize well (Krogh and Hertz 1992). People commonly use L2 regularization "weight decay" for training of deep neural networks and interpret model weights (David) 1992; Graves 201 This is true for vanilla Stochastic Gradient Descent (SGD). However, Loshchilly and Hutter (2018) revealed that, when the learning rate is adaptive, the commonly used 1.2 regularization is not identical to the vanilla weight decay proposed by Hanson and Pratt (1889)
Gaussian prior over the
0, (1 (1- X X) (L(),-1
00
where X' is the weight decay hyperparameter, 1j is the learning rate, and I(O,-1) is the loss function over one minibatch. We denote the loss function over the whole training dataset as L(0).
conventional belief is that, from the Bayesian perspective, weight decay indicates Gaussian prior over the model parameters, which regularize the complexity of the learned model (Graves 2011. However, we argue that this Bayesian interpretation cannot explain the empirical success of
weight decay in deep learning. On one hand, Wenzel et al. (2020) demonstrated that the conventional Bayesian posterior usually does not work well in deep learning. This challenged the meaning of weight decay as proper prior in the Bayesian interpretation. On the other hand, [ewkowycz and Gur. Ari 2020) and our experimental results (Figure 2) both suggested that weight decay is surprisingly unnecessary for long-time training. This observation critically challenged the Bayesian interpretation of weight decay in deep learning. The Bayesian interpretation believes that the prior knowledge over model parameters truly exist. Thus, the prior knowledge should not depend on the training time of deep neural networks, which contradicts with the experimental results in Figure 2
Recently researchers started explore other explanations for weight decay. Zhang et al. (2019b) revealed three different roles of weight decay. Some papers (Van Laarhoven, [017] Zhang et al.) 2019b) argued that weigh decay increases the effective learning rate but has regularization effect

---

Under review as a conference paper at ICLR 2022
Table I: Test performance comparison of optimizers. We report the mean and the standard deviations (as the subscripts) of the optimal test errors computed over three runs of each experiment. The SWD method enables Adam generalize as well SGD and even outperform complex Adam variants.
DATASET
MODEL
ADAM
AMSGRAD
ADAMW
ADABOUND
PADAM
Youı
RADAM
CIFAR-10
RESNETI8
5.01aas
4.91aa
6.
6. 6.16a
5.08
6.65a.c
6.12a.0
5.87.
6.0
VGG16
6.42a.s
6.0
7.31aas
7.34
6.59v.1
6.76o.1e
6.1 6.15a.0
6.30p
6.56aax
CIFAR-100
RusNer34
21.52.ar
21.76o.42
27.16
25.533.1p
22.96%.
22.876.13
22.72b.10
23
24.4
DENSENETi21
20.520.2
25.1
24.436.0
21.55o.14
69
11
22.1bm.s
27.27
GOoGLENEr
21.2
21.05gas
26.126.31
25. 25.53k,a7
21.29u17
28.18
21.82,a7
24.2
22 23u3
normalization layers, such as batch normalization (loffe and Szegedy 2015). [i et al. (2020) defined a new "intrinsic learning rate" parameter that is the product of the vanilla learning rate 7) and the weight decay factor À and argued that the convergence time depends on this intrinsic learning rate.
when combined with
However, people still expect to further explore the theoretical mechanism of weight decay, because weight decay sometimes shows mysterious behaviors beyond the existing theoretical understanding. As we discussed above, the Bayesian interpretation has been critically challenged by recent empirical findings (Lewkowycz and Gur-Ari 2020). The interpretation of increasing the effective learning rate only holds for scale-invarian normal lization layers, which is narrow scope.
Contributions. In this paper, we try to theoretically understand the novel behaviors of weight decay and explore the potential of weight decay. Our paper mainly has three contributions.
First, we propose a novel theoretical interpretation of weight decay from the perspective of learning dynamics. We study how the stability of the stationary points and the convergence time depend on weight decay, which corresponds to two main effects of weight decay.
Second, we propose novel weight-decay linear scaling rule (See Rule 2 in Section 3) for large-batch training. We discover that the existing linear scaling rule (Krizhevsky) 2014, Goyal et al.] (017) for large-batch training that proportionally increases the learning rate may sometimes be harmful to large-batch training due to bad convergence. Instead, increasing weight decay is a more suitable solution in this case.
Third, to the best of our knowledge, we are the first to design an effective learning-rate-awarre scheduler for weight decay which may significantly improve test performance. Our work fills the gap in this direction. We note that a trivial weight decay scheduling method recently proposed by Lewkowyez and Gur-Ari (2020) has limited practical value because only works well with constant learning rate, as the original paper claimed. Inspired by the stability of the stationary points, we show that weight decay should be coupled with the effective learning rate. We call the proposed method Stable Weight Decay (SWD). Adam (Kingma and Bal 2015) with SWD (AdamS) is displayed in fully supports that S SWD can significantly improve Adam.
Algorithm 2
The results in Table I
Structure of this paper. In Section 2 we analyze the dynamics of weight decay. In Section B study how weight decay can improve large-batch training. In Scction we propose SWD. In Scction 5 we empirically study SWD. In Section we conclude our work.
2 UNDERSTANDING THE DYNAMICS OF WEIGHT DECAY
In this section, we provide way to understand weight decay from viewpoint of learning dynamics.
Weight decay should be coupled with the learning rate scheduler. The vanilla weight decay described by Hanson and Prati 1989) is given by Equation (T) A more popular implementation for vanilla SGD in modern deep learning libraries, such as Py Torch (Paszke et al. 2019) and TensorFlow (Abadi et al. 2016). is given by
@L(O,-1)
0. ===1 (1-1n1A)00 7)-
(2)
00
2

---

Under review as a conference paper at ICLR 2022
where Th is the learning rate at the t-th step and weight decay is coupled with the learning rate scheduler. While previous work did not studied why weight decay should be coupled with the learning rate scheduler, this trick has been adopted by most deep learning libraries.
We first take vanilla SGD as the studied example, where no Momentum is involved. It is easy to see that vanilla SGD with L2 regularization ~ |10 is also given by Equation (2). Suppose the learning rate is fixed in the whole training procedure. Then Equation (2) will be identical to Equation [ if we simply choose X 17,. However, learning rate decay is quite important for training of deep neural networks. So Equation 2 is not identical to Equation
T in practice.
Which implementation is better? In Figure 1 empirically verified that the popular Equation- based weight decay indeed outperforms the vanilla implementation in Equation (T). We argue that Equation- Eqqaaionn(2)-aasse -bbssed weight decay is theoretically better than Equation- -Toon-----baddd weight decay in terms of the stability of the stationary points. We define the stability as Definition] []
Definition (The stability of the stationary point). The stationary point is stable if a stationary point is fixed during training. Otherwise, we define it as an unstable stationary point.
The training objective at step-t of Equation ① is L( + ل 1012 while that of Equation (2) is 72
L()) + \| Thus, the stationary points of the regularized loss given by Equation ( are stable during training, while that given by Equation ① is unstable due to the dynamic training objective.
Weight decay matters to the stability. As the stability of the stationary points critically depends on the weight decay scheduler, we define stable weight decay accordingly in Definition []
Definition 2 (Stable Weight Decay). The weight decay is stable if the stationary points are stable during training. Otherwise, we define it as unstable weight decay.
We propose Theorem D and prove that even deterministic Gradient Descent (GD) with vanilla weight decay may not converge any non-zero stationary point due the instability of the stationary points.
Theorem 1 (Non-convergence of GD with unstable weight decay). Suppose learning dynamics is governed by GD with vanilla weight decay (Equation (T)) and the learning rate qe E (0, +20) holds. If 35 that satisfies 0 < 8 S In 1 for any t > to. then the learning dynamics camnot converge to any non-zero stationary point satisfying the condition
lim
[[ + [IV[((++))11111
+0+20
Remark. We leave the proof in Appendix A.I. Even if the gradient is zero at the t-th step, the gradient at the t + 1-th step will not be zero. Unstable weight decay causes changing the objective and hence the solution does not converge to any non-zero stationary point. Thus, GD with unstable weight decay has no theoretical convergence guarantee like GD /Wolfe 1969). This explains why weight decay should be coupled with the learning rate to keep the stationary points stable during training. We conjecture that the stability of the stationary points should be a desirable property in deep learning.
Dynamics of weight decay. Lewkowyez and Gur-Ari (020)) empirically discovered that, if we let the training time be approximately inverse to weight decay, the test error monotonically decrease as weight decay decreases, which is beyond the existing theoretical understanding. We will show that it is easy to understand this empirical observation by analyzing the dynamics of weight decay.
We first state Assumption [ Assumption which holds near minima are common and useful for analyzing the solution or the convergence behavior in related papers (Mandt et al. [017 Neyshabur
eet al. 2017 Xie et al. 2021a]] Li et al. 2017 [Zhang et al. 2019a| Zhou et al. 2020). It is known
that stochastic optimization can escape multiple bad minima (Xie al. 2021b] Zhu et al. 2019
Kleinberg et al. (2118) and finally converge to good minimum during training. Thus, Assumption can be repeatedly applied near each minimum during training.
Assumption 1. The loss function around a minimum 0* can be written as
L(0) = L( + 2 - H(0 - -0))
where H is the Hessian at 6*.
We propose Theorem 2 and demonstrate how the learned model parameters 0 may depend on weight decay. We leave the proof in Appendix A.2
3

---

Under review as a conference paper at ICLR 2022
Exp 33-30
0.26
4.-
Eaidle
0.22
0.30
0.00
01-
0.64
0.02
300 Epochs
352
175
16
10" 10
Wright Jecry
Figure 1:
We compared Equation- 4 based
Figure 2: We train ResNet18 via SGD CIFAR- 10 for verifying that feonvergence = O (A-1) With the fixed 200 epochs, the optimal weight decay is about 0.0005. With 0.1A epochs, de- creasing weight decay monotonically decreases test performance. Table further supports that the optimal weight decay is approximately inverse to the number of epochs.
weight decay and Equation- 2 based weight de- cay by training ResNet18 vanilla SGD. In the presence of a popular learning rate scheduler, Equation- -aaatttionnn--eseeee -based weight decay shows better test performance. It demonstrates that the form >0 is better weight decay im- plementation than -XO0.
CIFAR-I0 via
Sel--tii epochs,
Theorem (Dynamies of weight decay). Suppose Assumption holds, and learning dynamics is governed by SGD with weight decay (Equation (2). If the learning rate scheduler is constant, the expected Or after iterations satisfies the following:
EE 0*] [T (HH + XT)] [0o H(H A1) X(H -
(3)
where I is the identity matrix. the learning rate scheduler given by the sequence {m)1 T2....... . the expected 0, after t iterations satisfies the following:
m
ElO,- - 0-] II (H + XD) 100 + X(H + A Jmm
(4)
Theorem is useful tool to understand convergence behaviors which may repeatedly happen during searching minima of deep learning. While Theorem dependd on Assumption []] will show that is remarkably consistent with novel empirical behaviors of weight decay and can also reveal novel insights. Theorem indicates that weight decay mainly has two effects: 1) biasing the expected learned solution and 2) accelerating the convergence.
Effect Weight decay biases the stationary points. Theorem provides us two key insights about weight decay. The first insight is Corollary [ derived from Theorem 2 by choosing the long-time limit. Corollary indicates that large weight decay may not learn accurate minima well. This insight is consistent with the regularization-batt based understanding of weight decay (Krogh and Hertz 1992).
Corollary 1. Suppose the conditions of Theorem 2 hold and H, + > O, where II the i-th eigenvalue of H. Then, in the limit of long training time that - +oc, the displacement from the true minimum to the expected learned solution along the i-th direction is given by
#4 -(H; 10*
where the bias μ lo E
8*) -*).
Figure 2 suggests that, given long enough training time inverse weight decay, decreasing the weight decay hyperparameter may consistently improve test performance, which is also verified by Lewkowyez and Gur-Ari (2020). This observation is contradicted with the conventional regularization- based understanding (Krogh and Hertz 1992), but fully supports Corollary [ that weight decay is bad, if we already have good convergence.
Weight decay accelerates convergence. However, we discover that the long-time posterior cannot explain the convergence behaviors. Figure suggests that, we need to increase the number of training iterations by 1 for maintaining good convergence, which is also discovered by Lewkowycz and Gur-Ari 2020) and Li et al. (2020). To explain why the convergence time depends on 1 need to explore the dynamics of weight decay before approaching the convergence.
Effect 2.
The second insight is Corollary that estimates the number of iterations for good convergence.
4

---

Under review as a conference paper at ICLR 2022
Corollary 2. Suppose the conditions of Theorem 2 hold and 6; = H; À O, where H2 is the i-th eigenvalue of H. Then, the convergence time in the i-th direction is
feonvergence, O ((H+ A) 1)
It is well-known that the Hessian in deep learning has a very small number of large eigenvalues and a large number of nearly zero eigenvalues (Sagun al. 2016 2017). Thus, have H; + À R À for most directions. It means that the convergence time critically depend on the dynamics in the subspace corresponding to those nearly zero eigenvalues:
Fronvergence = 0 11-
Equation reveals two facts. First, Equation supports the statement that the initial learning rate and the weight decay hyperparameter À are linearly coupled for maintaining performance (Loshchilov) and Hutter 2018; Van Laarhoven| [017]] Zhang et al.] [2019b] Hoffer et al. 2018). Second, Equation (7) demonstrates that the number of iterations (not the number of epochs training time) for good convergence depends on O (A-1)) ") This not only explains the relation in Figure [ but also reveals
that we should increase the number of iterations instead of the number of epochs by 1. We note that the second fact also reported by Lewkowyez and Gur-Ari (2020), while Leiie/Lewkowycz and Gur-Ari (2020) did not present formal theoretical analysis like Theorem
In our theoretical analysis, weight decay mainly has two effects. The first effect is to bias the expected solution in the long-time limit. The first effect is negative for learning accurate minima. The second effect is to accelerate the convergence in the subspace corresponding to nearly zero eigenvalues of the Hessian. The second effect is positive for accelerating training. Tuning weight decay needs to balance the tradeoff between the two effects.
WEIGHT DECAY IMPROVES LARGE-BATCH TRAINING
In this section, motivated by the theoretical insights in Section 2 we study how weight decay may surprisingly improve large-batch training.
Large-batch training can efficiently utilize the parallel computation to speedup training of deep networks (Goyal et al., 2017). However, large-batch training suffers from the difficulty of minimizing the training loss and usually find sharp minima that do not generalize well (Hoffer et al., 2017 Keskar
et al.
2017). How to perform large-batch training with good generalization is an important problem
which has attracted much attention from both academia (Zhang et al. 2019a Wu et al. 2020 Wen
al.
2020) and industry You al. 2017 Goyal al. 2017).
Rule 1 mitigates the small noise prohlem in large-batch training. The famous linear scaling rule, namely [ for large-batch training is well-known (Krizhevsky/ 2014 Goyal et al. (017). Large-batch training can easily get stuck in saddle points and sharp minima due to small gradient noise, because the noise magnitude in SGD dynamics is approximately proportional to the ratio of the learning rate 1) to batch size B. namely 3 (Mandt et al. 2017; Keskar et al. 2017). Xie et al. (2021b) recently proved that 1/2 exponentially matters to SGD SGD dynamics. Thus, the learning- Inear scaling rule is a natural result of maintaining the similar magnitude of stochastic gradient noise and similar learning dynamics invariant to the batch size. When the noise magnitude the performance bottleneck, Rule I is usually an effective solution.
Rule 1. When the batch size is multiplied by k, multiply the learning rate by k.
For large- training, people usually focused on the noise magnitude problem but overlooked the convergence problem (Jain et al., 2018 Ma et al.) 2018 (018) Yin et al. 2018). Sometimes, the noise magnitude not the performance bottleneck. People usually perform large-batch training with a fixed number of epochs for keeping the total computational costs moderate. Under the constraint of fixing the number of epochs, obviously, the number of iterations will be multiplied by - if the batch size is multiplied by k. Thus, the number of iterations may be too small to achieve good convergence.
Rule 2 mitigates the bad convergence problem in large-batel training. If the bad convergence problem is the performance bottleneck, the learning-ran -aat linear scaling rule will even be harmful to large-batch training due to slower convergence. When Rule Rule[ causes optimization divergence or bad convergence, we instead propose the weight-decay linear scaling rule as Rule 2 based on Equation
5

---

Under review as a conference paper at ICLR 2022
pecord vireted
014
4.3
0444
600
20
(a) Test Curves
(b) Training Curves
(c) Rule
(d) Rule 2
Figure 3: Large-batch training (B = 16384) with various learning rates and weight decay. Note that 17 10-3 and À = 10 is the baseline choice for B = 128. Subfigure (a) and (b) show that, even slightly increasing the learning rate (by multiplying 16) is harmful to optimization convergence. Subfigure (c) shows that Rule is completely invalid in this common large-batch training setting. Subfigure (d) shows that, multipyyigg weight decay by 128 (A = 0.0128) has the lowest test error,
which fully supports the proposed Rule
77. Because large weight decay may accelerate the convergence. Note that, increasing the learning rate only works in a narrow range, i.e., B S 8192 on ImageNet (Keskar et al. 2017). since too large learning rates may lead to optimization divergence or bad convergence (Keskar et al., 2017] Masters and Luschi 2018). T. To the best of our knowledge, we are the first to propose the weight-decay -eca-d--eccbbbee rule for large-batch training. Previous papers did not design a rule for the second-type problem.
Rule 2. When the batch size is multiplied by k, multiply the weight decay by k.
Empirical Analysis of Large-Batch Batch Training, We conducted experiments to show that, when the learning-rad rate linear scaling rule is harmful due to the convergence problem, the weight-decay linear scaling rule significantly improves targe-batch training. In Figure we trained ResNet18 via SGD on CIFAR-10 with the very large batch size B 16384, which is 128 times the common batch size 128.
We note that, in this common setting on CIFAR-10, even slightly increasing the learning rate is very harmful to large-batch training. Figure demonstrates Rule|Imaa seriously cause bad convergence. This observation means that, in the setting of training ResNet18 on CIFAR-10, the convergence problem is the main performance bottleneck instead of the noise magnitude. As Rule we observed that multiplying the weight decay hyperparameter by 128 works best. Thus, in this second-type problem of large-batch training, Rule Rule[ significanll outperforms Rule
4 STABLE WEIGHT DECAY
In this section, we design weight decay scheduler for adaptive gradient methods, such as Adam.
Algorithm 1: Adam/Adam
Algorithm 2: AdamS (p- = 0.5)
gr VI( + A0,-1;
9t (I(O:-1);
m = 31m4-1
1 + (1 9()
m4 = f-m--1 1 + (1 - -B1)g
Vt 320r-1
+ +(1 - -32)
Vt 32U1-1 + (1 - 32)
vi I-
me
m, T-
& 1-04
é = 1-91
0, 0,-1 Vorte m. 7//0-11
0. = mean(0:);
0, 0,1 Voete M4
Decoupled weight decay is unstable weight decay in adaptive gradient methods. Loshchilov and Hutter (2018) first pointed that, when the learning rate is adaptive, the commonly used L2 regularization is not identical of Momentum and focus on the effect of Adaptive Learning Rate for simplicity. Thus, AdamW (oshhhilov and Hutter, 2018) can be written as
weight decay. In the following analysis, we ignore the effect
(0L(0(-)
0, = (1- m1)0011 1
(8)
00

---

Under review as a conference paper at ICLR 2022
0.12
035
635
0.11
MkkM Ain
0:0
0.00
F20
0.00
0:00
6.07
622
622
100
125
150
175
200
25
150
175
200
25
100
125
(a) VGGI6 on CIFAR-10
(b) ResNet34 on CIFAR- 1000
(c) DenseNet121 on CIFAR-100
Figure 4: The learning curves of AdamS, AdamW, and Adam on CIFAR-10 and CIFAR-I00. AdamS shows significantly better generalization than Adam W and Adam.
Anleet Ackentii
167
0.32
Adbm
0.38
0.33
19
Fezam
Adu Dound
2.24
Adam
0.09
0.2
0.07
180
0.2
0.28
2.55
3.7%
0.92
0.22
0.08
12.4
10-
30-
Text D100
Epects
(eight Decury
Figure 5: The scatter plot of training losses and test errors during final 40 epochs of train- ing ResNet34 on CIFAR-100.
Figure 6: The learning curves
Figure 7: The test errors of VGG16 on CIFAR-10 with var- ious weight decay rates. The displayed weight decay value of AdamW has been rescaled by the factor = 0.001. A similar ex- perimental result for ResNet34 is presented in Appendix C
of all adaptive gradient methods by training ResNet34 on CIFAR- 100. AdamS outperforms other Adam variants. The test perfor-
Even with similar higher train-
ing losses, AdamS still general- izes better than other Adam vari- ants. We leave the scatter plot on CIFAR-10 in Appendix C
mance of other models can be
found Table
where 7 is the exponential moving average of the squared gradients in Algorithm | and the power notation of a vector means the element-wise power of the vector. We interpret rje; the effective learning rate for multiplying the gradients. However, we clearly see that decoupled weight decay uses the vanilla learning rate perform weight decay rather than the effective learning rate. The minimum 9* of the regularized loss function optimized by Adam at the t-th step
0- = - - (H There is no guarantee that ei will be constant for Thus, the regularized loss function optimized by AdamW has unstable minima.
(9)
Scheduling weight decay in adaptive gradient methods. An easy solution to fix the unstable weight decay problem is using the following scheduled weight decay:
(0,(0.-)
0, (1 170, A)01-1 1742
(10)
00
This gives 0- 00-00 0- -(H + -)-100 Equation (10) can indeed make stationary points of the regularized loss function stable for different
However, if we still use the element-wise scheduler ve for weight decay, the anisotropic convergence problem will appear in the subspace corresponding to small eigenvalues of the Hessian. By Theorem
2. the convergence can be measured bylT - (I 2x 1))). Due the element-wing learning rate adaptivity, a common number of iterations may be not enough for achieving the convergence along some dimensions.
7

---

Under review as a conference paper at ICLR 2022
10
10
6.26
10-9
6.26
6.04
6.04
6.04
5.82
5.82
5.82
Leening
10-1
5.60
10-1
5.60
10
5.80
5.30
4.54
4.94
4.72
4.72
10.5
4.50
10-1
4.50
10-9
4.50
10
10"
Weight Decay
Weight Decay
Weight Decay
Figure 8: The test errors of R CIFAR-10. AdamS has much deeper and wider basin near dark points (s 4.9%). The optimal test еπто of AdamS, AdamW, and Adam are 4.52%, 4.90%, and 5.49%, respectively. The displayed weight decay value of AdamW has been rescaled by the factor > 0.001.
Unfortunately, there is no ideal solution to the stability problem and the convergence problem at the same time for adaptive gradient methods. We conjecture that this may be an internal fault of all optimizers that use Adaptive Learning Rate. To balance the tradeoff between the stability and the convergence in presence of Adaptive Learning Rate, we propose non-element-wiie learning- rate- aware scheduler for weight decay as
(T,(0)
0. = - (1-70, --)0 1746"
(11)
00
where ūy is the mean of all elements of the vector e and p is a hyperparameter to control the adaptivity of the scheduler. Note that 17; always isotropic. We call weight decay with this heuristic scheduler Stable Weight Decay. The pseudocode of AdamS is displayed in Algorithm 2
We recommend p 0.5 as the default value in AdamS unless we specify it, because 176 P with p = 0.5 may approximate the magnitude of the adaptive learning rate, Note that Decoupled Weight Decay (Loshchilov and Hutter 2018; Bjorck et al. 2021) is special case of Stable Weight Decay with = 0.
We note that e is not expected to be zero minima due stochastic gradient noise, because the variance of the stochastic gradient is directly observed to be much larger than the expectation of the stochastic gradient at/near minima (Zhu al. (2019) and depends the Hessian (Xie et al. 20216). In some special cases, such as full-batch training, it is fine to add a small value (I.e., 10 to 8 to avoid being zero as divisor.
EMPIRICAL ANALYSIS OF STABLE WEIGHT DECAY
In this section, conducted comprehensive experiments verify the advantage of the SWD scheduler over Decoupled Weight Decay and L2 for Adam. Our experiments also include popular Adam variants, including AMSGrad (Reddi et al. 2019). Yogi (Zaheer et al., 22118. AdaBound (Luo
et al. 2019), Padam (Chen and Gu 2018). and RAdam (Liu et al. 2019)..
Models and Datasets. We choose Adam as the base optimizer, and train popular deep models, including MosNet11//3/5/ (He et al. 2016), VGG16 (Simonyan and Zisserman, 2014), DenseNet121 (Huang et al.] 2017). GoogLeNet (Szegedy al. 2015). and Long Short- Term Memory (LSTM) (Hochreiter and Schmidhuber 1997). on CIFAR- 100 (Krizhevsky and Hinton 2009). and
Penn TreeBank (Mareus et al. 1993). The implementation details are Appendix B
Figure shows the learning curves of AdamS, AdamW, and Adam on several benchmarks. In our experiments, AdamS always leads to lower test errors. Figure 5 shows that, even if with similar higher training losses, AdamS still generalizes significantly better than AdamW, Adam, and recent Adam variants. Figure 6 displays the learning curves of all adaptive gradient methods. The test performance of other models can be found in Table I.. Simply scheduling weight decay for Adam by SWD even outperforms complex Adam variants. We also note that most Adam variants surprisingly generalize worse than SGD (See Appendix C
Image Classification on
8

---

Under review as a conference paper at ICLR 2022
Adand Lt=led
Adamd
Advmmi (R-1e3 Adumi
Adami!
0.34
0.30
0.32
0.20
0.58
0.28
10
10
20
32
60
90
WagghCoccy
Epoddo
Figure 9: Rule 2 holds well for all Adam, AdamW, and AdamS on ResNet18. Figure T6 shows similar results for VGG16, which has no scale-invarian loss landscape.
Figure 10: ResNet50 on ImageNet. The lowest Top-1 test errors of AdamS, AdamW, and Adam are 24.19%, 24.29%, and 30.07%, respectively,
demonstrates that AdamS consistently outper- Robustness to the hyperparameters. Figure further further forms Adam and Adam W under various weight decay hyperparameters. According to Figure 7 we also notice that the optimal decoupled weight decay hyperparameter in AdamW can be very different from L2 regularization and SWD. Thus, AdamW requires re-tuning the weight decay hyperparameter in practice, which is time-consuming. Figure X shows that AdamS is more robust to the learning rate and weight decay than Adam and Adam. AdamS has a much deeper and wider basin than AdamW and Adam. We leave the experiments with cosine annealing schedulers and warm restarts (Loshchilov and Hutter 2016) in Appendix E. which also support that AdamS yields superior test performance and converges lower training losses.
Large-Batch Training. In Figure
9 we again verified that Rule 2 holds well on all Adam, AdamW, and AdamS. We also note that, in the case that applies Rule AdamS generalizes better than AdamW and Adam on large-batch training (B 16384).
Limitations. Figure TO shows that, ImageNet, the improvement of AdamS AdamW becomes marginal. Figure I1 in yields better test results than both Decoupled Weight Decay and SWD,
that, for LSTM on Penn TreeBank, Le regularization
6 CONCLUSION AND DISCUSSION
Our experiments large-batch training shows that the learning-rate linear scaling rule sometimes is harmful to large-batch training due to bad convergence. We propose the weight-decay linear scaling rule for large-batch training as alternative solution. We note that the results may be explained by previous Aapers(Aror) et al., 2019 Li et al.) 2020), which studied scale-invariant deep loss landscape due batch normalization loffe and Szegedy (0155). Instead, the empirical results verified our theoretical analysis with or without scale-invariant loss functions.
Inspired by the theoretical analysis of weight decay on stability and convergence, propose scheduler for weight decay. Previous work did not studied how to design schedulers for weight decay. Our empirical results demonstrate that SWD often makes improvements over L2 regularization and Decoupled Weight Decay. The generalization gap between SGD and Adam can be almost closed by simple weight decay scheduler on CIFAR datasets. Although our analysis mainly focused on Adam, SWD can be easily combined with other Adam variants, too.
Finally, we discuss the limitations of SWD. While SWD makes significant improvements in first three experiments of Section 5 the experiments ImageNet and Language Modeling suggest that the improvement SWD may be marginal for complex loss landscapes. SWD as the first practical design for weight decay scheduling is not perfect. In future, will be interesting to design weight decay schedulers that can accelerate convergence in the early training phase while learning stable and accurate minima in the late training phase.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Abadi, M.. Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., et al. (2016). Tensorflow: system for large-scale machine learning. In 12th [USENIX] symposium on operating systems design and implementation ((OSDI) 16), pages 265-283.
Arora, S., Lyu, K., and Li, Z. (2019). Theoretical analysis of auto rate-tuning by batch normalization. In 7th International Conference on Learning Representations, ICLR 2019.
Bjorek, J., Weinberger, K. Q., and Gomes, C. (2021). Understanding decoupled and early weight decay. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 6777-6785.
Chen, J. and Gu, Q (2018). Closing the generalization gap of adaptive gradient methods in training deep neural networks. arXiv preprint arXiv: 1806.06633.
David, J. M. (1992). A practical bayesian framework for backprop networks. Neural computation.
Goyal, P. Dollár, P., Girshick, R., Noordhuis, P., Wesolowski, L.,, Kyrola, A., Tulloch, A., Jia, Y., and He, (2017). Accurate, large minibatch sgd: training imagenet in hour. arXiv preprint arXiv:1706.02677
Graves, A. (2011). Practical variational inference for neural networks. In Advances neural information processing systems, pages 2348-2356..
Hanson, J. and Pratt, L. Y. (1989). Comparing biases for minimal network construction with back- procpapoppag,ion In Advances in neural information processing systems, pages 177-185.
He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778.
Hochreiter, S. and Schmidhuber, J. (1997). Long short-term memory. Neural computation, 988):1335- 1780.
Hoffer, E., Banner, R., Golan, I., and Soudry, D. (2018). Norm matters: efficient and accurate normalization schemes in deep networks. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pages 2164-2174.
Hoffer, E., Hubara, I., and Soudry, D. (2017). Train longer, generalize better: closing the generaliza- tion gap in large batch training of neural networks. In Advances in Neural Information Processing Systems, pages 1739-1739.
Huang, G., Liu, Z., Van Der Maaten, L., and Weinberger, K. Q. (2017). Densely connected con- volutional networks. In Proceedings of the IEEE conference on computer vision and puttern recognition, pages 4700-4708.
loffe. S. and Szegedy, C. (2015). Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pages 448-45c PMLR.
Jain, P., Kakade, S., Kidambi, R., Netrapalli, P., and Sidford, A. (2018). Parallelizing stochastic averaging, and model misspecification.
gradient descent for least squares regression: Journal of Machine Learning Research, 18.
Keskar, N. S., Mudigere, D., Nocedal, J.. Smelyanskiy, M., and Tang, P. T. P. (2017). On large-batch batch training for deep learning: Generalization gap and sharp minima. In International Conference on Learning Representations.
Kingma, D. P. and Ba, J. (2015). Adam: A method for stochastic optimization. 3rd International Conference on Learning Representations, ICLR 2015.
Kleinberg, B., Li, Y., and Yuan, Y. (2018). An alternative view: When does sgd escape local minima? In International Conference on Machine Learning, pages 2698-2707
10

---

Under review as a conference paper at ICLR 2022
Krizhevsky, A. (2014). One weird trick for parallelizing convolutional neural networks. arXiv preprint arXiv:1404.500
Krizhevsky, A. and Hinton, G. (2009). Learning multiple layers of features from tiny images.
Krogh, A. and Hertz, J. A. (1992). A simple weight decay can improve generalization. In Advances in neural information processing systems, pages 950-957.
Lewkowycz, A. and Gur-Ari, G. (2020). On the training dynamics of deep networks with 1_2 regularization. Conference on Neural Information Processing Systems.
Li, Q. Tai, C., et al. (2017). Stochastic modified equations and adaptive stochastic gradient algorithms. In Proceedings of the 34th International Conference Machine Learning- -oonne 70, pages 2101- 21 10. JMLR. org.
Li, Z., Lyu, K., and Arora, S. (2020). Reconciling modern deep learning with traditional optimization analyses: The intrinsic learning rate. Conference on Neural Information Processing Systems, 33.
Liu, L., Jiang, H., He, P., Chen, W., Liu, X., Gao, J., and Han, J. (2019). On the variance of the adaptive learning rate and beyond. In International Conference Learning Representations.
Loshchilov, and Hutter, F. (2016). Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint 1608 03993.
Loshchilov, I. and Hutter, F. (2018). Decoupled weight decay regularization. In International Conference on Learning Representations.
Luo, L., Xiong, Y., Liu, Y., and Sun, X. (2019). Adaptive gradient methods with dynamic bound of learning rate. 7th International Conference on Learning Representations, ICLR 2019.
Ma, S., Bassily, R., and Belkin, M. (2018). The power of interpolation: Understanding the effec- tiveness of sgd in modern over-parametrized learning. In International Conference on Machine PMLR.
Learning, pages
Mandt, S., Hoffman, M. D., and Blei, D. M. (2017). Stochastic gradient descent approximate bayesian inference. The Journal of Machine Learning Research, 18(1):4873-4907,
Marcus, M., Santorini, B., and Marcinkiewicz, M. A. (1993). Building a large annotated corpus of english: The penn treebank.
Masters, D. and Luschi, C. (2018). Revisiting small batch training for deep neural networks. arXiv preprint arXiv: 1e04.08044.1111
Neyshabur, B., Bhojanapalli, S., McAllester, D., and Srebro, N. (2017). Exploring generalization in deep learning. In Advances in Neural Information Processing Systems, pages 5947-5956.
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019). Pytorch: An imperative style, high-performance deep learning library. In Advances neural information processing systems, pages 8026-8037.
Reddi, S. J., Kale, S., and Kumar, S. (2019). On the convergence of adam and beyond. 6th International Conference Learning Representations, ICLR 2018.
Sagun, L., Bottou, L., and LeCun, (2016). Eigenvalues of the hessian deep learning: Singularity and beyond. arXiv preprint 1xXv::661.00770
Sagun, L., Evci, U., Guney, V. U., Dauphin, Y.. and Bottou, L. (2017). Empirical analysis of the hessian of over-parametrizze neural networks. arXiv preprint arXiv: 044544.
Shapiro, A. and Wardi, Y. (1996). Convergence analysis of gradient descent stochastic algorithms. Journal of optimization theory and applications, 91(2):439
Simonyan, K. and Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv preprint moXiv=1409 1556.
11

---

Under review as a conference paper at ICLR 2022
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. (2015). Going deeper with convolutions. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 1-9.
Van Laarhoven, T. (2017). L2 regularization versus batch and weight normalization. arXiv preprint 07Xiv:7706.05350.
Welling, M. and Teh, Y. W. (2011). Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML-I1), pages 681 1-68...
Wen, Y., Luk, K., Gazeau, M., Zhang, G., Chan, H., and Ba, J. (2020). An empirical study of stochastic gradient descent with structured covariance noise. In Chiappa, S. and Calandra, R., editors, Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pages 3621-3631. PMLR.
Wenzel, F., Roth, K., Veeling, B., Swiatkowski, J., Tran, L.,, Mandt, S., Snoek, J., Salimans, T., Jenatton, R., and Nowozin, S. (2020). How good is the bayes posterior in deep neural networks really? In International Conference on Machine Learning, pages 10248-10259. PMLR.
Wolfe, P. (1969). Convergence conditions for ascent methods. SIAM review, 11(2):226-335.
Wu, J., Hu, W., Xiong, H., Huan, J., Braverman, V., and Zhu, Z. (2020). On the noisy gradient descent that generalizes as sgd. In International Conference on Machine Learning, pages 10367-10376. PMLR.
Xie, Z., He, F., Fu, S., Sato, I., Tao, D., and Sugiyama, M. (2021a). Artificial neural variability for deep learning: On overfitting, noise memorization, and catastrophic forgetting. Neural Computation.
Xie, Z., Sato, I., and Sugiyama, M. (2021b). A diffusion theory for deep learning dynamics: Stochastic gradient descent exponentially favors flat minima. In International Conference on Learning Representations.
Xie, Z., Wang, X., Zhang, H., Sato, I., and Sugiyama, M. (2020). Adai: Separating the effects of adaptive learning rate and momentum inertia. arXiv preprint 2tt.::20000..11155
Xie, Z., Yuan, L., Zhu, Z., and Sugiyama, M. (2021c). Positive-negative momentum: Manipulating stochastic gradient noise to improve generalization. In International Conference Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages
PMLR.
Yin, D., Pananjady, A., Lam, M.. Papailiopoulos. D., Ramchandran, K., and Bartlett, P. (2018). Gradient diversity: key ingredient for scalable distributed learning. In International Conference on Artificial Intelligence and Statistics, pages 1998-2007. PMLR.
You, Y., Gitman, L., and Ginsburg. B. (2017). Scaling sgd batch size 32k for imagenet training. 1eo::0:08000000
arXiv preprint
Zaheer, M., Reddi, S., Sachan, D., Kale, S., and Kumar, S. (2018). Adaptive methods for nonconvex optimization. In Advances in neural information processing systems, pages 9793-9803.
Zaremba, W., Sutskever, I., and Vinyals, O. (2014). Recurrent neural network regularization. arXiv preprint arXiv:1409.23299
Zavriev, S. and Kostyuk, F. (1993). Heavy-ball method in nonconvex optimization problems. Compu- tational Mathematics and Modeling, 4(4):336 341.
Zhang, G., Li, L., Nado, Z., Martens, J., Sachdeva, S., Dahl, G., Shallue, C., and Grosse, R. B. (2019a). Which algorithmic choices matter at which batch sizes? insights from a noisy quadratic model. In Advances in Neural Information Processing Systems, pages 8 8196-8207.
Zhang, G., Wang, C., Xu, B., and Grosse, R. (2019b). Three mechanisms of weight decay regulariza- tion. International Conference on Learning Representations.
Zhang. J., He, T., Sra, S., and Jadbabaie, A. (2019c). Why gradient clipping accelerates training: A theoretical justification for adaptivity. In International Conference Learning Representations.
12

---

Under review as a conference paper at ICLR 2022
Zhou, P., Feng, J., Ma, C., Xiong, C., Hoi, S. C. H., et al. (2020). Towards theoretically understanding why sgd generalizes better than adam deep learning. Advances in Neural Information Processing Systems, 33.
Zhu, Z., Wu, J., Yu, B., Wu, L., Ma, J. (2019). The anisotropic noise stochastic gradient descent: Its behavior of escaping from sharp minima and regularization effects. In ICML, pages 7654-7663.
A PROOFS
A.1 PROOF OF THEOREM 1
Proof. We first write the regularized loss function corresponding to SGD with vanilla weight decay
as
X 2Tt
f((0= = L(0)
|
(12)
at f-th step.
X is unstable during training, the regularized loss function If the corresponding La regularization 2n, (0) will also be time-dependent function and has non-zero stable stationary points.
Suppose we have non-zero solution 0* which is a stationary point of f(0, () at I-th step and SGD) finds 0g 0 /-th step.
Even if the gradient of f0) at t-step is zero, we have the gradient at (t + 1)-th step as
9++1 fi+1 X(n Met1)00 )
(13)
means that || ( ni-l) 12100-122
To achieve convergence, we must have |1 || 0. requires (17 -
or | 2 0.
Theorem 2.2 of Shapiro and Wardi (199) told us that the learning rate should be small enough for convergence. Obviously, we have < co in practice.
As 7 77++ does not hold, SGD cannot converging any non-zero stationary point. The proof is now complete.
A.2 PROOF OF THFOREM 2
Proof. Under Assumption 1. we write the dynamics of SGD with weight decay as
0: 0 (H: - 8*) - 7
(14)
Note that the stochastic gradient noise is zero-mean, [[[] 0. where is the difference of the true gradient and the stochastic gradient. The zero-mean property generally holds because SGD randomly selects minibatches from the whole training dataset, which is widely used in related papers (Xie et 2021c Kingma and Ba 2015 Reddi et al. 2019).
The dynamics above is equivalent to minimizing the training loss with Lg regularization as
0:-1 me Lima ThE
(15)
where
Tona (0)
0*) H(O 0*) + T
(H + X1) - H(H + X1) + HA(H X0)
(16)
13

---

Under review as a conference paper at ICLR 2022
Thus, may rewrite the dynamics of SGD with weight decay
0₂ =0 (/(H + (T) H(H + ne
(17)
0. H( 10 =11 - -n nT + + 1I -1) T
(18)
E B HH AT) 10 =E
mH (T) H(H + -T)-
(19)
where the expectation of the stochastic gradient noise term is zero. As we care about the difference between the expected learned solution and the true minimum 0*, may further obtain
+ 1I mh(H + (0)] H(H + +1) A
00: - -
k=1
(20)
In the case of the constant learning rate scheduler, we may simplify the equation above as
Ei0,- - H(H + tr) = [7 7 A0) H( + +1) X(H + -
(21)
The proof is now complete.
B EXPERIMENTAL DETAILS
Computational environment. The experiments are conducted on a computing eluster with GPUs of TM
NVIDIA @ Tesla" P100 16GB and CPUs of Intel Xeon CPU E5-2640 v3 2.60GHz.
B. IMAGE CLASSIFICATION ON CIFAR-10 10 AND CIFAR-100
Data Preprocessing For CIFAR-10 and CIFAR-100: We perform the common per-pixel zero- mean unit-variance normalization, horizontal random flip, and 32 32 random crops after padding with 4 pixels on each side.
Hyperparameter Settings: We select the optimal learning rate for each experiment from (0.0001, 0.001, 0.01, 0,1, 10} for non-adaptive gradient methods. We use the default learning rate for adaptive gradient methods the experiments of Table ??, while we compared Adam, AdamW, AdanS under various learning rates and batch sizes in the experiments of Figure 7 and Figure In the experiments on CIFAF-10 and CIFAR- 100: 1j 0.1 for SGD and SGDS; r 0.001 for Adam, AdamW, Adam W, AMSGrad, AdamW, AdaBound, and RAdam; 1) 0.01 for Padam. For the learning rate schedule, the learning rate is divided by 10 the epoch of {80, 160} for CIFAR-10 and {100, 150} for CIFAR-100 100, respectively. The batch size is set to 128 for both CIFAR-10 10 and CIFAR- 100, unless we specify it on large-batch training.
The strength of L2 regularization and SWD is default to 0.0005 as the baseline. Considering the linear scaling rule, we choose dw Aza Thus, the weight decay of AdamW uses Aw 0.5 for CIFAR-I0 10 and CIFAR-100. The basic principle of choosing weight decay strength to let all optimizers have similar convergence speed.
We set the momentum hyperparameter 3 = 0.9 SGD and SGDS. As for other optimizer hyperparameters, apply the default hyperparameter settings directly.
We repeated each experiment for three times in the presence of the error bars.
We leave the empirical results with the weight decay setting À 0.0001 in Appendix
B.2 LARGE-BATEE TRAINING ON CIFAR- 10
Model: we always used the common ResNet18 for large-batch training.
we always used Adam and its variant as the optimizer, because adaptive gradient methods often outperform SGD in large-batch training.
Optimizer:
14

---

Under review as a conference paper at ICLR 2022
The learning rate scheduler: Suppose the number of epochs is E, the learning rate is divided by 10 at the epoch of (0.4E.0.E)).
The default batch size is 16384 in large-batch training. The default weight decay is 0.0001 or follows Rule The default learning rate is 0.001 or follows Rule
As for other optimizer hyperparameters, we apply the default hyperparameter setting directly.
B.3 IMAGE CLASSIFICATION ON IMAGENET
Data Preprocessing For ImageNet: For ImageNet, we perform the per-pixel zero-mean unit- variance normalization, horizontal random flip, and the resized random crops where the random size (of 0.08 to 1.0) of the original size and random aspect ratio (of 2 to 5) of the original aspect ratio is made.
Hyperparameter Settings for ImageNet: We select the optimal Iearning rate for each experiment from {0.0001, 0.001, 0.01, 0.1, 1, 10} for all tested optimizers. For the learning rate schedule, the learning rate is divided by 10 at the epoch of {30, 60}. We train each model for 90 epochs. The batch size is set to 256. The weight decay hyperparameter of AdamS, AdamW, Adam are chosen from {5 10-6, 6 5 x 10 5 x 10-4, 5 10 5 x 10 10-2) 2). As for other optimizer hyperparameters, still apply the default hyperparameter settings directly.
B.4 LANGUAGE MODELING
We use a classical language model, Long Short-Term Memory (LSTM) (Hochreiter and Schmidhuber 1997) with 2 layers, 512 embedding dimensions, and 512 hidden dimensions, which has 14 million model parameters and is similar to the "medium LSTM" in Zaremba et al. (2014). Note that our baseline performance is better than the reported baseline performance in Zaremba al. (2014). The benchmark task is the word-level Penn TreeBank (Marcus et al. 1993). We empiricall compared AdamS, AdamW, and Adam under the common and same conditions.
Hy Hlpeeparreeeerr Settings. Batch Size: B = 20. BPTT Size: bptt = 35. Learning Rate: 1j = 0.001. We run the experiments under various weight decay selected from {10-4, 5 x 10-5 -10-5,5 x 10 10-6, 1100--. 5 10-7, 10- 1010000"T J.The dropout probability is set to 0.5. We clipped gradient norm to 1.
C SUPPLEMENTARY FIGURES AND RESULTS OF ADAPTIVE GRADIENT METHODS
Popular Adam variants often generalize worse than SGD. A few Adam variants tried to fix the hidden problems in adaptive gradient methods, including AdamW Loshchilov and Hutter (2018). AMSGrad (Reddi et al. 2019) and Yogi (Zaheer et al., 2018). A recent line research, such as AdaBound (Luo al.) 2019). Padam (Chen and Gu 2018), and RAdam (Liu et al. 2019), believes controlling the adaptivity of learning rates may improve generalization. This line of research usually introduces extra hyperparameters
control the adaptivity, which requires more efforts in tuning hyperparameters. However, we and /Zhang et al. (2019c) found that this argument is contradicted with our comparative experimental results (see Table D. In our empirical analysis, most advanced Adam variants may narrow but not completely close the generalization gap between adaptive gradient methods and SGD. SGD with a fair weight decay hyperparameter as the baseline performance usually generalizes better than recent adaptive gradient methods. The main problem may lie in weight decay. SGD with weight decay À 0.0001, common setting in related papers, often not good baseline, as X= = 0.0005 often shows better generalization on CIFAR-10 10 and CIFAR-100 (see in Figures 19 and 21 of Appendix F). We also conduct comparative experiments with 0.0001 (see Table 2 of Appendix C). Under the setting À = 0.0001, while some existing Adam variants may outperform SGD sometimes due to the lower baseline performance of SGD, AdamS shows superior test performance. For example, for ResNet18 CIFAR- 10, the test error of AdamS is lower than SGD by nearly one point and no other Adam variant may compare with AdamS.
Language Modeling. It is well- that, different from computer vision tasks, the standard Adam (with La regularization) is the most popular optimizer for language models. Figure i Appendix ddemnnstrates that the conventional belief is true that the standard 7.2 regularization yields better
15

---

Under review as a conference paper at ICLR 2022
Table 2: Test performance comparison of optimizers with AL Xg 0.0001 and Aw 0.1, which is a common weight decay setting in related papers. AdamS still show better test performance than popular adaptive gradient methods and SGD.
DATASET
MODEL
SGD
ADAMS
AnAM
AMSGRAD
ADAMW
ARABOUND
PADAM
Ynci
RADAM
CIFAR-IO
RESNETIS
VGG16
6.92
6.16
7.04
6.68
6.45
7:33
6.74
6.69
6.73
CLEAR-100
RESNET34
25.67
23.
DUNSENErI21
20.98
21.35
24.39
22.
22.23
24.23
22.26
22.40
22.40
GooGLeNN
21.89
21.60
24.60
24.05
21.71
25.03
26.69
22.56
22.35
Adarns
95
Figure 11: Language modeling under various weight decay, Note that the lower perplexity better.
test results than both Decoupled Weight Decay and SWD. The weight decay scheduler suitable for language models is open problem.
We report the learning curves of all adaptive gradient methods in Figure 12 They shows that vanilla Adam with SWD can outperform other complex variants of Adam.
Figure 13 displays the scatter plot of training losses and test errors during final 40 epochs of training DenseNet122 21 CIFAR-100.
Figure [4] the test performance AdamS, AdamW, and Adam under various weight decay hyperparameters of ResNet34 on CIFAR-100.
We train ResNet18 on CIFAR-10 for 900 epochs to explore the performance limit of AdamS, AdamW, and Adam in Figure 15
Note that ResNets have scale-invariant loss landscape, while VGGs have no scale-invariant loss landscape. Our theoretical analysis generally holds under Assumption 1, which is independent of the scale-invariant property of the loss landscape. Figure 16 further shows that, with or without normalization layers, our theoretical result can empirically observed. This is beyond the intrinsic learning rate proposed by Li et al. (2020) which depends on the scale-invariant property of the loss function.
014
1.30
2.0
0.23
CCMPS
(a) ResNet18
(b) VGG16
(c) DenseNet121
(d) GoogLeNet
Figure 12: The learning curves adaptive gradient methods.
16

---

Under review as a conference paper at ICLR 2022
Adars
Adarts
Than Lute
xp
30-
0.64
0.66 6.00
0.28
0.22
0.24
0.22
Texl 0m35
Text bives
Figure 13: Even if with similar or higher training losses, AdamS still generalizes better than Adam W and Adam. The scatter plot of training losses and test errors during final 50 epochs of training VGG16 on CIFAR-10 and DenseNet121 on CIFAR-100.
0.275
0333
Acemth
Acem
0.20
022
0203
0375
Figure 14: We compare the generalization of Adam, AdamW. and AdamS with various weight decay rates by training ResNet34 on CIFAR-100. The displayed weight decay of Adam in the figure has been rescaled by the factor zs 0.001. The optimal test performance of AdamS is significantly better than AdamW W and Adam.
Adums
Adum
0.04
160
Figure 15; We train ResNet18 CIFAR-10 for 900 epochs to explore the performance limit of
AdamS, AdamW, and Adam. The learning rate is divided by 10 at the epoch of 300 and 600. AdamS achieves the most optimal test error, 4.70%.
0.18
0.36
0.22
0.10
10
30'
10"
10
Weight
Figure 16: Rule 2 holds well for all of Adam, Adam W, and AdamS, VGG16 on CIFAR-10.
17

---

Under review as a conference paper at ICLR 2022
Table 3: In the experiment of ResNetl8 trained via SGD CIFAR-10, we verified that the optimal weight decay is approximately inverse to the number of epochs. The predicted optimal weight decay approximately 0.1 x Epochs because the optimal weight decay is = 0.0005 selected from {10-2,5 5,5x 10-3 10-30,30000 5 10- 100-11100000 10-4,5, 10 10-5, 5 10-6, 10 10-6) with 200 epochs as the base case. The observed optimal weight decay is selected from {Epochs -1 0.1 Epochs 0.01 x Epochs }. We observed that the optimal test errors are all corresponding to the predicted optimal weight decay À 0.1 Epochs At least in the sense of the order of magnitude, the predicted optimal weight decay is fully consistent with the observed optimal weight decay. Thus, the empirical results supports that the optimal weight decay is approximately inverse to the number of epochs in the common range of the number of epochs.
EPOCHS
Epochs
0.1 Epochs
À 0.01 Epochs
50
74.06
7.12
7.50
100
22.04
5.56
6.01
200
11.81
5.02
5.61
1000
4.67
4.43
6.02
2000
4.59
4.48
5.70
D ADDITIONAL ALGORITHMS
Algorithm 3 is the TensorFlow implementation for SGD,
Algorithm 4 is the implementation of Adai, AdaiW, and AdaiS. As 1-
8s is always 1 in Adai, AdaiW
is identical to AdaiS.
We note that the implementation of AMSGrad in Algorithm is the popular implementation in PyTorch. We use the PyTorch implementation in our paper, as it is widely used in practice.
Algorithm 3: SGD in TensorFlow
(I( + A0--1;
m 8,m-1 19:1
0, 0:-1 + me:
Algorithm 4: Adai AdaiW
VL(O,-)) + 10,-1;
e 32% + 32)94
ēt Fi
T-
T ((s);(;);
B1c (1 : )lCipp(0, c);
m 31 1 + Bie gt:
rn T- AIx
my
0, 0,-1 mi 1710-11
Algorithm 5: AMSGrad/AMSGraaa
Algorithm 6: AMSGradS
VI(A,-1) A0,-1;
9r = VI(A,-1);
me 31m--1 + (1 81)gti
m4 31m4-1 + (1 11-1))) -B1)gti
Vt 32v1-1
82)gr
Uy 9201-1 + (1 - 92)
m = ml 1-
mr = T-
Uman max((' (Ue: Umaz);;
U Umar = (u Umar))
64 I-3
Fmug.
0 I-3
8,11 Votte
mr 1720c-1;
T = mean(e)).
0 0-11 Voete 1y
22 20--1;
Vo,
18

---

Under review as a conference paper at ICLR 2022
3.30
AcamW Acam
Acam Acam
223
3.25
320
1.20
2.15
212
3.5
Epecte
(a) ResNet18
(b) VGG16
122
Adams
129
1011
Trainee Loke
10%
Trahiee
10
10
ACamTW Adar
100
125
150
1.5
25
55
103
125
192
175
202
Spada
Figure 17: The learning curves of ResNet18 and VGG16 on CIFAR-10 with cosine annealing and warm restart schedulers. The weight decay hyperparameter: dL₃ Às 0.0005 and Aw 0.5. Top Row: Test curves. Bottom Row: Training curves. AdamS yields significantly lower test errors and training losses than AdamW and Adam.
E SUPPLEMENTARY EXPERIMENTS WITH COSINE ANNEALING SCHEDULERS AND WARM RESTARTS
In this section, we conducted comparative experiments on AdamS, AdamW, and Adam in the presence of cosine annealing schedulers and warm restarts proposed by Loshchilov and Hutter (2016). We set the learning rate scheduler with a recommended setting Loshchilo and Hutter (2016): To = 14 and Timi 2. The number of total epochs is 210. Thus, trained each deep network for four runs warm restarts, where the four runs have 14, 28, 56, and 112 epochs, respectively. Other hyperparameters and details displayed in Appendix
Our experimental results in Figures 17 and 18 suggest that AdamS consistently outperforms AdamW and Adam in the presence of cosine annealing schedulers and warm restarts. It demonstrates that, with various learning rate schedulers, the advantage of SWD may generally hold.
Moreover, did not empirically observe that cosine annealing schedulers with warm restarts may consistently outperform the common piecewise-constant learning rate schedulers for adaptive gradient methods. We noticed that Loshchilov and Hutter (2016) empirically compared four-staged piecewise- constant learning rate schedulers with cosine annealing schedulers with warm restarts, and argue that cosine annealing schedulers with warm restarts are better. There may be two possible causes. First, three-staged precewise-constall learning rate schedulers, which usually have a longer first stage and decay learning rates by multiplying 0.1, 1, are the recommended settings, while the four-staged piccewise-constant learning rate schedulers in Loshchilov and Hutter (2016) usually not optimal. Second, warm restarts may be helpful, while cosine annealing may be not. The ablation study on piecewise-constant learning rate schedulers with warm restarts is lacked. We argued that how to choose learning rate schedulers may still be an open question, considering the complex choices of schedulers and the complex loss landscapes.
F WEIGHT DECAY IN MOMENTUM
In this section, we rethink weight decay in Momentum.
19

---

Under review as a conference paper at ICLR 2022
0.22
0.32
0.21
0.2
AdaTes
Adam
1440 Smee
0.29
0.23
0.03
(a) ResNet18
(b) VGGI6
Figure 18: The test errors of ResNet18 and VGGI6 on CIFAR-I0 under various weight decay with cosine annealing and warm restart schedulers. AdamS yields significantly better optimal test performance than A and Adam.
It is also commonly believed that L2 regularization is optimal weight decay implementation when Adaptive Learning Rate is not involved. Almost all deep learning libraries directly use L2 regularization as the default weight decay implementation. However, we reveal that L2 regularization is not identical to decoupled weight decay Loshchilov and Hutter (2018) and sometimes harms performance slightly when Momentum involved.
We take Stochastic Heavy Ball (SHB) Zavriev and Kostyuk 1993, which only uses fixed momentum inertia and dampening coefficients, as the studied example the presence of Momentum, as SHB- style Momentum methods are widely used by many popular optimizers. SGD implemented in PyTorch (Paszke et al. (2199) is actually SHB with default hyperparameters. We write SHB with L2 regularization and SHB with decoupled weight decay in Algorithm
Algorithm 7: SWD/SGDDDSWWWW (SHB/SHBW)
Algorithm 8: SGDS (SHBS)
Ir VL(0 A0--1;
St (L(0;-1]
m, 31 + Bagt
my = m1m=-1 + Bagti 0, 0,-1 1714 T-3, 110
0, 0,-1 171111 1110,-1;
Inspired by Theorem
we propose SGD with SWD (SGDS) as
m, = f1m,-1 + Ba9t
(s(1-0f)
(22)
nn 0c-1 1774,, 0: = 1-91
where Pa(1-91), 1-81 1) the effective learning rate in SHB, as m, 3s(1-8i) As (1 - -t) converges 1-81
8. soon, we use the simplified I-3, as the bias correction in practice, where relatively large weight decay in the first dozen iterations can work like model parameter initialization strategy. It easy to see that the factor I-3; for weight decay exactly the difference between our stable weight decay and decoupled weight decay suggested by Loshchilov and Hutter (2018). The pseudocode of SGDS is displayed in Algorithm g It may also be called SHB with SWD (SHBS).
Empirical Analysis on SGDS. We empirically verified that the optimal performance of SGDS/SGDW is often slightly better than the widely used vanilla SGD and SGD, as seen in Figures We leave more empirical results of SGDS Appendix (see (see Table
19 and
In Figure 19 the optimal performance of SGDS/SGDW is better than the widely used Vanilla SGD and SGD. For Vanilla SGD, SGD, and SGDS, we may choose \ = As = 0.0005 to maintain the optimal performance. But we have to re-tune Aw 0.005 for SGDW. Hyperparameter Setting: 31 = 0 for Vanilla SGD; 31 = 0.9 for SGD, SGDW, and SGDS. We repeat each simulation for three runs. A similar experimental result for ResNet18 is presented in Appendix G In Figure 20 report the test performance SGDS and SGD for ResNet18 on CIFAR- 10 under various learning rates and weight decay. SGDS has deeper blue basin than SGD.
20

---

Under review as a conference paper at ICLR 2022
0.2
5965
5225 $46A
10
5820
1015
5823
Larning ho0
5.400
Baaainn Rata
5.93
Than
5.325
5323
4.955
4.95
4.830
4839
0.00
10.4
16-1
10-4
10-
4
5.500
10 1
10
10--
10--
10-
4.00
We gnd Decay
Weight Decay
Figure 19: We compare the gen- eralization of Vanilla SGD, SGD, SGDW, and SGDS under various weight decay hyperparameters by training VGG16 on CIFAR-10.
Figure 20: The errors of ResNet18 on CIFAR-10. SGDS has a slightly deeper blue basin near dark points (S 4.83%). The optimal choices of 7/ and are very close for SGDS and SGD.
Table 4: Test performance comparison of Adai, AdaiS, SGD, and SGDS. Stable/Decoupled Weight Decay often outperform Lg regularization for optimizers involving in momentum. We report the mean and the standard deviations (as the subseripts) of the optimal test errors computed over three runs of each experiment.
DATASET
MODEL
ADAIS
ADAI
SGDS
SGD
CIFAR-10
RESNET18
4.59o. 16
4.740. 14
4. 4.69p.09
5.01p.03
VGG16
5.81o.07
6.00g.
6.280.07
6.42o.0
CIFAR-100
DENSENET121
19.440.21
19.59p.38
19.61o.20
19.8
GOOGLENET
20.50p.25
20.550.32
20.680.0x
21.21o.29
is not well-known that SGDW often outperforms SGD slightly. We believe is mainly because 9g is necessary for maintaining good performance when people switch from SGD to SGDW. As the effective learning rate of SGD is T- 77. the weight use different settings of 31 and B3 decoupled weight decay, we will undesirably change weight decay rate unless we re-tune Aw. However, people usually directly let Aw AL2 for SGDW in practice. Figure 19) shows that, the optimal ÅL and Äg in SGD with 31 = 0.9 and 33 = 1) are both 0.0005, while the optimal Aw is 0.005 instead. The optimal Àτ, and Äs are almost same, while the optimal Aw is quite different. Thus, the advantage of SGDS over SGDW can save us from re-tuning the weight decay hyperparameter.
people rarely know re-tuning Aw based on 1-31
decay rate of SGDW is actually R 1-hh Ay rather than Aw
Ag
G SUPPLEMENTARY RESULTS OF SGD WITH MOMENTUM AND ADAI
We compare SGDS, SGDW, vanilla SGD and SGD under various weight decay hyperparameters by training ResNet18 on CIFAR-10. We observe that the optimal performance SGDS/SGDW better than vanilla SGD and SGD in Figure 21 The advantage of SGDS over SGDW is that we do not need to fine-tune the weight decay hyperparameters based
We report the learning curves of SGD and SGDS in Figure
22
SGDS compares favorably with SGD.
Based on a diffusion theoretical framework (Xie et al. 2021b Xie et al. 2020 proposed Adaptive Inertia Estimation (Adai) that adaptive momentum inertia instead of Adaptive Learning Rate to help training. Adaptive inertia can be regarded as an pertia-adaptive variant SHB. The previous analysis on SHB can be easily generalized to Adai. We display Adai with SWD (AdaiS) in Algorithm
21

---

Under review as a conference paper at ICLR 2022
SGDS
SGDW
10-
10-4
10
10-
10
10-0 10
Weight
Figure 21: We compare the generalization of Vanilla SGD, SGD, SGDW, and SGDS with various weight decay hyperparameters by training ResNet18 on CIFAR- 10. The optimal weight decay rates are near 0. 0.0005 for all three weight implementations. The optimal performance of SGDS/SGDW better than Vanilla SGD and SGD. For Vanilla SGD, SGD, and SGDS, we may safely choose dL Äs 0.0005. But have re-tune dw 0.005 for SGDW. Hyperparameter Setting: 31 = for Vanilla SGD; B1 0.9 for SGD, SGDW, and SGDS.
Sois
6.0
560
0.685
Teat E100
000
0.000
265
0.605
624
0.000
1012
325
75
160
(a) ResNet18 on CIFAR-10
(b) VGG16 on CIFAR-10
(c) GoogLeNet on CIFAR-100
Figure 22: Generalization analysis on SGDS and SGD with Le regularization. Hyperparameter
Setting: As >L 0.0005 and B1 0.9.
0.64
Asa
0.000
A24
0.85
626
0.00
4/
0.0
0.000
75
102
325
150
175
0.6
202
155
375
620
75
160
135
Esxiss
Concs
Epeets
(a) ResNet18 CIFAR-10
(b) VGG16 on CIFAR-10
(c) GoogLeNet CIFAR-100
Figure 23: Generalization analysis on AdaiS and Adai with La regularization. Hyperparameter Setting: Äs ^z, 0.0005.
22

---

Under review as a conference paper at ICLR 2022
2
We report the learning curves of Adai and AdaiS in Figure 23
which may verify the generalization
advantage of AdaiS over Adai.
23


Reviews:
[{"review_id": "rA7fDhYd1ax", "reviewer": "Reviewer_6kHA", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper introduced the readers on vanilla weight decay, first proposed by Hansan and\nPratt (1989). A more popular version is: θt = (1 − ηtλ)θ_(t−1) − η_t δL(θ_(t−1))/δθ\nwhere η_t is  the learning rate at the step, and the weight decay is coupled with the learning rate scheduler. It is consistently used in several deep learning regularizations. The paper tries to explain some of the not known reasons why weight decay is coupled with the learning rate iteration scheme. It came up with some theoretical proofs on why the above\nequation is better than the original version Hanson and Pratt came up with. By citing current research, the paper explained some of the effects of weight decay, such as: Biasing stationary points and accelerating convergence of backpropagation.\nAnother section looked at the implications of weight decay in larger batch training, which according to research, can efficiently utilize the parallel computation to speed up the training of deep neural networks. It compared the prior work on linearly increasing the\nlearning rate when batch size increases to linearly increasing the weight decay by the same factor. The latter approach (which the paper novelly proposed) helped in addressing the bad convergence problem that models with high-batch sizes frequently deal with.\nFinally, the paper proposed AdamS, a variation of Adam optimizer, with an additional weight decay term in the update rule. It compared its results with other learning algorithms like Adam, AMSGrad, RAdam, and Padam and found that AdamS performed slightly better than\nothers. In the supplementary sheets, the proofs for stated theorems were given, details on specific models used were provided, and a combination of SWD with other adaptive variants was stated.\n\nNovelty\n1. Previous work hadn’t caught on why the linear scaling of learning rate isn’t entirely helpful for higher-batch training data because of low convergence. The current paper came up with a distinct approach that sorted this issue out.\n2. Previous work did not focus on the reasons why weight decay should be coupled with the learning rate scheduler, which the current paper explored and tried to exploit with its novel approaches.\n3. The paper designed an approach to schedule wait decay that made improvements over L2 Regularization and Decoupled Weight Decay.\n4. SWD, the technique proposed by this paper, can be used with a slight modification to the current learning algorithms. It’s not an entirely new algorithm in itself, but it utilizes benefits from the currently popular methods and helps provide a better regularization\nand thus better performance.\n\nAdditional comments\nAn insight into future work can be provided on whether there is a plan to target the paper’s limitations, i.e., to come up with a weight-decay scheduler for language models in Natural Language Processing that consistently outperforms L Regularization. ", "summary_of_the_paper": "This paper focuses on Weight decay and its importance in Regularization during training.\nThe purpose of the paper is to propose an interpretation of weight decay for learning\ndynamics, coming up with a mechanism to improve weight decay for larger batch sizes and\ndeveloping a schedular called Stable Weight Decay (SWD).\n", "summary_of_the_review": "The paper is interesting, free from technical errors, useful and fairly rigorous.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ciw-w3CoBLd", "reviewer": "Reviewer_saHk", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Strength:\n\nThe organization of this paper is quite clear. The theoretical analysis in this paper is easy to follow. In the empirical study, the proposed AdamS is shown to be superior to Adam and AdamW in various datasets and neural networks. Overall, the quality of this paper is satisfactory in terms of clarity and empirical evaluation. \n\n# Weakness:\n\nMy concern for this paper is that there are some gaps between the theoretical assumptions and experiment settings.  \n\nOne major issue is about Assumption 1 and the scale-invariant property of some neural architectures used in the experiment. The loss function is a scale-variant function in Assumption 1 but ResNet and DenseNet are both scale-invariant networks. The scale property matters here since weight decay has its main effect in controlling the weight magnitude. Theorems in this paper only rely on the quadratic approximation in Assumption 1. But the practically tested networks violate Assumption 1. \n\nIn addition, there are many existing papers (Hoffer et al., 2018, Arora et al., 2019, Li et al., 2020, Li et al., 2020) that investigate the effect of weight decay in deep neural networks with batch norm. Some of the observed phenomena can be explained by existing theories that respect the scale invariant property of ResNets, see the comments below. This puts the the novelty of theoretical analyses in this paper in a disadvantageous position. \n\nHere are some specific comments.\n\n1. $\\eta_t$=$\\eta_2$ in step-wise learning rate decay, except for 2 or 3 steps, which should not be an issue for stability of stationary points. I notice that the experiment in this paper uses the stepwise learning rate decay. How does the stable $\\eta$ in stepwise learning rate decay lead to unstable training?\n\n2. The paper claim that *Fig. 2 suggests given long enough training time, the optimal weight decay hyperparameter tends to be zero.* However, I think Fig.2 does not show the *optimal* learning rate tends to be zero when the training time is long. To support the claim, the figure should show the result of the same long training time using different weight decay.\n\n3. Figure 3 shows that increasing weight decay is more effective than increasing learning rate when a large batch size is used. This phenomenon can be explained by the effective learning rate argument in scale-invariant neural networks, which claims that increasing weight decay has the effect of increasing the learning rate. The experiment in Fig. 3 is done using a ResNet, i.e. a scale-invariant network. So I wonder if we have the same effect when a standard CNN like VGG is used. \n\n4. The test error curves in this paper (Fig.4 and 5) show a severe overfitting effect when using AdamW, which is not consistent with my experience. Could the authors give some comments on this effect? Is there any existing paper that shows the same overfitting effect of AdamW?\n\n5. Equation (20) and (21) seem to have typos about $\\theta^*$ in both right and left hand sides.\n\nIn summary, I would say that I enjoy the paper's clarity and structure but some main technical issues prevent it being published as its current status.Thus, I suggest settling the conflicts in theory and practice in an updated version. \n\n\n[1] Arora, Sanjeev, Kaifeng Lyu, and Zhiyuan Li. \"Theoretical analysis of auto rate-tuning by batch normalization.\" International Conference on Learning Representations. 2019. https://arxiv.org/abs/1812.03981\n\n[2] Li, Zhiyuan, and Sanjeev Arora. \"An Exponential Learning Rate Schedule for Deep Learning.\" International Conference on Learning Representations. 2020. https://arxiv.org/abs/1910.07454\n\n[3] Li, Zhiyuan, Kaifeng Lyu, and Sanjeev Arora. \"Reconciling modern deep learning with traditional optimization analyses: The intrinsic learning rate.\" Neural Information Processing Systems. 2020. https://arxiv.org/pdf/2010.02916.pdf\n", "summary_of_the_paper": "The paper proposes to understand the effect of weight decay and design schedules for weight decay in Adam based on the theory. The first contribution is to understand the dynamics of weight decay in a neural network by assuming the loss function is quadratic. The theoretical analysis shows that weight decay biases stationary points but accelerates convergence. Second, the paper proposes to increase the weight decay when a large batch size is used based on the convergence acceleration property of WD. Finally, AdamS is proposed to improve the stability of Adam and some empirical results show its effectiveness using various neural architectures on small-scale datasets (not good on ImageNet). ", "summary_of_the_review": "Despite the effectiveness of the proposed AdamS in small-scale datasets, the theoretical analysis about weight decay does not respect the scale invariant property of some important neural networks, which are used in the empirical validation of this paper to demonstrate some results of its analysis. This results in a severe conflict between theory and practice in this paper, so I would suggest a further revision for this manuscript.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "bIpl9tkN38y", "reviewer": "Reviewer_1qbp", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think AdamS is interesting and might help close the gap between SGD and Adam, however given that Adam is not widely used for vision tasks, its applications are limited. In text, weight decay is not used as often, so it is not clear if there are applications there, it might be interesting to explore that further with a more realistic training set/architecture than TreeBank and LSTM.\n\nAdamS improvements are not that large for ImageNet, but this baseline seems pretty strong since previously reported Adam behaviour is much lower (for example in https://arxiv.org/pdf/2002.11803.pdf)\n\n- I like that AdamS is more robust to hyperparameters. \n\n- the connection between section 2 and the rest seems a little weak. \n\n-How does eq7 motivate rule 2? rule 1 and rule 2 have the same scaling with respect to eq7. Why does increasing weight decay improve convergence but not increasing the learning rate?", "summary_of_the_paper": "The main contribution of the paper is to propose AdamS a new version of Adam with weight decay which has an implicit decay of the weight decay.\n\nThe contents of section 2 are quite standard: for GD linear regression with L2 regularization, its effect is mainly to shift the hessian: the minimum is not reached and the time to convergence depends on the eigenvalue of the (modified) hessian.\n\nSections 3,4,5 are roughly independent of 2 and they focus on improving Adam with weight decay. In section 3 they compare rescaling the learning rate vs weight decay with the batch size and they conclude that it is better to rescale the weight decay.  In section 4, they propose AdamS  which is more \"stable\" version of Adam.", "summary_of_the_review": "It seems like AdamS is only beneficial to training and I think it might be helpful to add this modification whenever using Adam in combination with weight decay. The improvements are not superstrong but there does not seem to be any drawback.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "PVVgqfLpkW", "reviewer": "Reviewer_AwnS", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well written and easy to follow. But I find the interpretation on the effect of weight decay is not convincing, and the experiment results has marginal improvements.\n\n# Major concerns\n\n-  The definition on \"stability of the stationary point\" is ambiguous. Since there is no re-definition on \"stationary point\" in this paper, I think the stationary point in this paper still indicates the points whose (full) gradient w.r.t. loss function is zero. Then what is the loss function in this paper? Does it involve $l^2$ regularization part? This should be clarified: If $l^2$ regularization is not involved in, the stationary points w.r.t. the loss function must be fixed whatever optimization method is applied, the definition of stability is redundant; If not, then another issue raises, I will show it below. \n\n- Theorem 1 is trivial. The fact that GD with weight decay as Eq.(1) cannot converge to any stationary points has known long before. However, being unable to converge to a fixed point is not always a disadvantage especially in deep learning tasks. A bunch of work [Hoffer, et al. 2017; Kleinberg, et al. 2018; kunin. et al. 2021] have proven, in either theorem or empirical aspects, that modern neural networks can get better performance by escaping from bad local optimum, or continue moving even after obtaining its best performance. So demonstrating the disadvantage of weight decay formed as Eq.(1) by \"stability\" is not convincing.\n\n- Assumption 1 is not reasonable in the topic discussed in this paper. On one hand, this paper refer to the results of Li, et al. (2020), Van Laarhoven, 2017, etc,  so I assume the authors have taken batch normalization into account. But the hessian of normalized network is not semi-definite on any well-defined point (except origin point). Beside, the intrinsic learning rate proposed by Li, et al. (2020) is based on a special phenomenon in learning dynamics of normalized neural network, the convergence rate $1/\\lambda^*$ indicates the convergence to equilibrium state not the convergence of loss. This is totally different from quadratic cases. More quantitative results about effective learning rate in equilibrium can be seen in a missing reference Wan et al, 2020, in which the roles of weight decay, learning rate, and momentum factor have been thoroughly revealed in learning dynamics of normalized neural network.\n\nOn the other hand, approximating the loss landscape as quadratic well is only reasonable when training is close to end [kunin, et al. 2021]. The hessian matrix varies vastly during the whole training procedure, so any insights drawn from the quadratic approximation is not suitable to characterize the whole training procedure, neither does the practical methods.\n\n- Again, theorem 2 is trivial and well known as a result of naive linear regression GD case. I cannot see any significance and novelty from it and its corollaries. \n\n- The reasons why weight decay improves large-batch training are not justified in section 3. First, the results in theorem 1, 2 did not reflect the effect of batch size. They are essentially full batch cases. Some reasons are either not justified, or contrary to existing work:\n\n\"Sometimes, the noise magnitude is not the performance bottleneck\"\n\nPlease justify it. At least in CIFAR and Imagenet tasks, Smith, et al. (2020) and You, et al. (2020) have taken massive computation resource to confirm that with standard training settings, performance degrades due to full batch GD not insufficient training budget. \n\n\"If the bad convergence problem is the performance bottleneck, the learning-rate linear scaling rule will even be harmful to\nlarge-batch training due to slower convergence.\"\n\nWhy? Please justify it.\n\n\"According to Equation (7), we propose the weight decay linear scaling rule as Rule 2. Because large weight decay may accelerate the convergence\"\n\nWhy? What's the difference between multiplying learning rate by $k$ and multiplying WD by $k$? Eq.(7) cannot reflect the difference.\n\n- In Adaptive learning rate part (section 4) The intuition is unreasonable: \n\n\"We interpret $\\eta v_t^{-1/2}$ as the effective learning rate for multiplying the gradients.\" \n\nWhy? Is it reasonable regarding $v_t^{1/2}$ is actually a vector?\n\n\" The expected solution learned by AdamW in the longe-time limit is ...\" , \"In the long-time limit, Equation (10) can indeed fix the stability\nproblem of the expected solution\"\n\nIs it correct? Note the explicit solution of linear regression GD highly relies on the constant value of learning rate and weight decay, it's not reasonable to derive the solution of adaptive optimizer merely by substituting some letters in the formula, rigorous proof is needed.\n\n\"...where $\\bar{v}_t$ is the mean of all elements of the vector $v_t$...\"\n\nDoes this relaxation preserve the advantages of the original form of Adaptive weight decay mentioned before? Please justify it.\n\n- Experiment results seems not persuasive. The proposed method are only applied on CIFAR/Imagenet datasets, comparing with Adam/AdamW. It is less meaningful to design a new training method which can only obtain better test performance than Adam while still worse than SGD (with momentum) in CIFAR/Imagenet experiment. The authors can prove its effectiveness in some cases where Adam/AdamW is SOTA, like some transformer tasks (Dosovitskiy, et al. 2020; Liu, et al. 2021). ", "summary_of_the_paper": "In this paper, the authors propose a new interpretation on the effect of weight decay in learning dynamics of neural network. Based on their interpretation, they design a weight-decay linear scaling rule for large-batch training, and a learning-rate-aware scheduler for weight decay in common settings. They verify the effectiveness of the proposed method on CIFAR, Imagenet experiments.", "summary_of_the_review": "Overall speaking, the assumption used in this paper is too strong to simplify the real cases properly, so the theoretical interpretation established on this assumption is unconvincing. Besides, the large training part lacks justification on some claims, while there seems to be some errors in derivation on the proposed method SWD. The experiment results are not convincing either. The current quality is below the acceptance threshold.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper analyzes the effects of the weight decay hyperparameter, and based on this analysis, proposes methods to schedule the weight decay. Overall, while I'm glad that more work is being done on understanding the effects of weight decay, I don't think this submission is of sufficient quality for ICLR.

Theorem 1 is simply re-expressing the well-known fact that if the regularization version of weight decay is used, then (simply because it's based on a single objective function) the stationary points are invariant to the choice of learning rate. This may not be apparent due to the misused terminology: "invariant" is referred to as "stable", but "stable stationary point" has a technical meaning very different from the one used here.

Corollary 2 essentially shows that the optimum of the regularized loss is different from the optimum of the unregularized loss. The authors conclude from this that the optimal value of lambda is 0 from the perspective of test error, which is unwarranted.

Overall, the paper centers around the interaction between learning rates and the weight decay parameter. However, as various reviewers point out, this interaction has been analyzed in detail for networks with normalization layers, and normalization completely changes the nature of the interaction. So any analysis would either need to take this into account or limit the scope to networks without normalization.

I encourage the authors to take the reviewers' feedback into account and improve the paper for the next submission cycle.
Decision: Reject