Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: In this paper, the authors propose a new interpretation on the effect of weight decay in learning dynamics of neural network. Based on their interpretation, they design a weight-decay linear scaling rule for large-batch training, and a learning-rate-aware scheduler for weight decay in common settings. They verify the effectiveness of the proposed method on CIFAR, Imagenet experiments.
main_review: The paper is well written and easy to follow. But I find the interpretation on the effect of weight decay is not convincing, and the experiment results has marginal improvements.

# Major concerns

-  The definition on "stability of the stationary point" is ambiguous. Since there is no re-definition on "stationary point" in this paper, I think the stationary point in this paper still indicates the points whose (full) gradient w.r.t. loss function is zero. Then what is the loss function in this paper? Does it involve $l^2$ regularization part? This should be clarified: If $l^2$ regularization is not involved in, the stationary points w.r.t. the loss function must be fixed whatever optimization method is applied, the definition of stability is redundant; If not, then another issue raises, I will show it below. 

- Theorem 1 is trivial. The fact that GD with weight decay as Eq.(1) cannot converge to any stationary points has known long before. However, being unable to converge to a fixed point is not always a disadvantage especially in deep learning tasks. A bunch of work [Hoffer, et al. 2017; Kleinberg, et al. 2018; kunin. et al. 2021] have proven, in either theorem or empirical aspects, that modern neural networks can get better performance by escaping from bad local optimum, or continue moving even after obtaining its best performance. So demonstrating the disadvantage of weight decay formed as Eq.(1) by "stability" is not convincing.

- Assumption 1 is not reasonable in the topic discussed in this paper. On one hand, this paper refer to the results of Li, et al. (2020), Van Laarhoven, 2017, etc,  so I assume the authors have taken batch normalization into account. But the hessian of normalized network is not semi-definite on any well-defined point (except origin point). Beside, the intrinsic learning rate proposed by Li, et al. (2020) is based on a special phenomenon in learning dynamics of normalized neural network, the convergence rate $1/\lambda^*$ indicates the convergence to equilibrium state not the convergence of loss. This is totally different from quadratic cases. More quantitative results about effective learning rate in equilibrium can be seen in a missing reference Wan et al, 2020, in which the roles of weight decay, learning rate, and momentum factor have been thoroughly revealed in learning dynamics of normalized neural network.

On the other hand, approximating the loss landscape as quadratic well is only reasonable when training is close to end [kunin, et al. 2021]. The hessian matrix varies vastly during the whole training procedure, so any insights drawn from the quadratic approximation is not suitable to characterize the whole training procedure, neither does the practical methods.

- Again, theorem 2 is trivial and well known as a result of naive linear regression GD case. I cannot see any significance and novelty from it and its corollaries. 

- The reasons why weight decay improves large-batch training are not justified in section 3. First, the results in theorem 1, 2 did not reflect the effect of batch size. They are essentially full batch cases. Some reasons are either not justified, or contrary to existing work:

"Sometimes, the noise magnitude is not the performance bottleneck"

Please justify it. At least in CIFAR and Imagenet tasks, Smith, et al. (2020) and You, et al. (2020) have taken massive computation resource to confirm that with standard training settings, performance degrades due to full batch GD not insufficient training budget. 

"If the bad convergence problem is the performance bottleneck, the learning-rate linear scaling rule will even be harmful to
large-batch training due to slower convergence."

Why? Please justify it.

"According to Equation (7), we propose the weight decay linear scaling rule as Rule 2. Because large weight decay may accelerate the convergence"

Why? What's the difference between multiplying learning rate by $k$ and multiplying WD by $k$? Eq.(7) cannot reflect the difference.

- In Adaptive learning rate part (section 4) The intuition is unreasonable: 

"We interpret $\eta v_t^{-1/2}$ as the effective learning rate for multiplying the gradients." 

Why? Is it reasonable regarding $v_t^{1/2}$ is actually a vector?

" The expected solution learned by AdamW in the longe-time limit is ..." , "In the long-time limit, Equation (10) can indeed fix the stability
problem of the expected solution"

Is it correct? Note the explicit solution of linear regression GD highly relies on the constant value of learning rate and weight decay, it's not reasonable to derive the solution of adaptive optimizer merely by substituting some letters in the formula, rigorous proof is needed.

"...where $\bar{v}_t$ is the mean of all elements of the vector $v_t$..."

Does this relaxation preserve the advantages of the original form of Adaptive weight decay mentioned before? Please justify it.

- Experiment results seems not persuasive. The proposed method are only applied on CIFAR/Imagenet datasets, comparing with Adam/AdamW. It is less meaningful to design a new training method which can only obtain better test performance than Adam while still worse than SGD (with momentum) in CIFAR/Imagenet experiment. The authors can prove its effectiveness in some cases where Adam/AdamW is SOTA, like some transformer tasks (Dosovitskiy, et al. 2020; Liu, et al. 2021). 
summary_of_the_review: Overall speaking, the assumption used in this paper is too strong to simplify the real cases properly, so the theoretical interpretation established on this assumption is unconvincing. Besides, the large training part lacks justification on some claims, while there seems to be some errors in derivation on the proposed method SWD. The experiment results are not convincing either. The current quality is below the acceptance threshold.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper focuses on Weight decay and its importance in Regularization during training.
The purpose of the paper is to propose an interpretation of weight decay for learning
dynamics, coming up with a mechanism to improve weight decay for larger batch sizes and
developing a schedular called Stable Weight Decay (SWD).

main_review: The paper introduced the readers on vanilla weight decay, first proposed by Hansan and
Pratt (1989). A more popular version is: θt = (1 − ηtλ)θ_(t−1) − η_t δL(θ_(t−1))/δθ
where η_t is  the learning rate at the step, and the weight decay is coupled with the learning rate scheduler. It is consistently used in several deep learning regularizations. The paper tries to explain some of the not known reasons why weight decay is coupled with the learning rate iteration scheme. It came up with some theoretical proofs on why the above
equation is better than the original version Hanson and Pratt came up with. By citing current research, the paper explained some of the effects of weight decay, such as: Biasing stationary points and accelerating convergence of backpropagation.
Another section looked at the implications of weight decay in larger batch training, which according to research, can efficiently utilize the parallel computation to speed up the training of deep neural networks. It compared the prior work on linearly increasing the
learning rate when batch size increases to linearly increasing the weight decay by the same factor. The latter approach (which the paper novelly proposed) helped in addressing the bad convergence problem that models with high-batch sizes frequently deal with.
Finally, the paper proposed AdamS, a variation of Adam optimizer, with an additional weight decay term in the update rule. It compared its results with other learning algorithms like Adam, AMSGrad, RAdam, and Padam and found that AdamS performed slightly better than
others. In the supplementary sheets, the proofs for stated theorems were given, details on specific models used were provided, and a combination of SWD with other adaptive variants was stated.

Novelty
1. Previous work hadn’t caught on why the linear scaling of learning rate isn’t entirely helpful for higher-batch training data because of low convergence. The current paper came up with a distinct approach that sorted this issue out.
2. Previous work did not focus on the reasons why weight decay should be coupled with the learning rate scheduler, which the current paper explored and tried to exploit with its novel approaches.
3. The paper designed an approach to schedule wait decay that made improvements over L2 Regularization and Decoupled Weight Decay.
4. SWD, the technique proposed by this paper, can be used with a slight modification to the current learning algorithms. It’s not an entirely new algorithm in itself, but it utilizes benefits from the currently popular methods and helps provide a better regularization
and thus better performance.

Additional comments
An insight into future work can be provided on whether there is a plan to target the paper’s limitations, i.e., to come up with a weight-decay scheduler for language models in Natural Language Processing that consistently outperforms L Regularization. 
summary_of_the_review: The paper is interesting, free from technical errors, useful and fairly rigorous.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The main contribution of the paper is to propose AdamS a new version of Adam with weight decay which has an implicit decay of the weight decay.

The contents of section 2 are quite standard: for GD linear regression with L2 regularization, its effect is mainly to shift the hessian: the minimum is not reached and the time to convergence depends on the eigenvalue of the (modified) hessian.

Sections 3,4,5 are roughly independent of 2 and they focus on improving Adam with weight decay. In section 3 they compare rescaling the learning rate vs weight decay with the batch size and they conclude that it is better to rescale the weight decay.  In section 4, they propose AdamS  which is more "stable" version of Adam.
main_review: I think AdamS is interesting and might help close the gap between SGD and Adam, however given that Adam is not widely used for vision tasks, its applications are limited. In text, weight decay is not used as often, so it is not clear if there are applications there, it might be interesting to explore that further with a more realistic training set/architecture than TreeBank and LSTM.

AdamS improvements are not that large for ImageNet, but this baseline seems pretty strong since previously reported Adam behaviour is much lower (for example in https://arxiv.org/pdf/2002.11803.pdf)

- I like that AdamS is more robust to hyperparameters. 

- the connection between section 2 and the rest seems a little weak. 

-How does eq7 motivate rule 2? rule 1 and rule 2 have the same scaling with respect to eq7. Why does increasing weight decay improve convergence but not increasing the learning rate?
summary_of_the_review: It seems like AdamS is only beneficial to training and I think it might be helpful to add this modification whenever using Adam in combination with weight decay. The improvements are not superstrong but there does not seem to be any drawback.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes to understand the effect of weight decay and design schedules for weight decay in Adam based on the theory. The first contribution is to understand the dynamics of weight decay in a neural network by assuming the loss function is quadratic. The theoretical analysis shows that weight decay biases stationary points but accelerates convergence. Second, the paper proposes to increase the weight decay when a large batch size is used based on the convergence acceleration property of WD. Finally, AdamS is proposed to improve the stability of Adam and some empirical results show its effectiveness using various neural architectures on small-scale datasets (not good on ImageNet). 
main_review: # Strength:

The organization of this paper is quite clear. The theoretical analysis in this paper is easy to follow. In the empirical study, the proposed AdamS is shown to be superior to Adam and AdamW in various datasets and neural networks. Overall, the quality of this paper is satisfactory in terms of clarity and empirical evaluation. 

# Weakness:

My concern for this paper is that there are some gaps between the theoretical assumptions and experiment settings.  

One major issue is about Assumption 1 and the scale-invariant property of some neural architectures used in the experiment. The loss function is a scale-variant function in Assumption 1 but ResNet and DenseNet are both scale-invariant networks. The scale property matters here since weight decay has its main effect in controlling the weight magnitude. Theorems in this paper only rely on the quadratic approximation in Assumption 1. But the practically tested networks violate Assumption 1. 

In addition, there are many existing papers (Hoffer et al., 2018, Arora et al., 2019, Li et al., 2020, Li et al., 2020) that investigate the effect of weight decay in deep neural networks with batch norm. Some of the observed phenomena can be explained by existing theories that respect the scale invariant property of ResNets, see the comments below. This puts the the novelty of theoretical analyses in this paper in a disadvantageous position. 

Here are some specific comments.

1. $\eta_t$=$\eta_2$ in step-wise learning rate decay, except for 2 or 3 steps, which should not be an issue for stability of stationary points. I notice that the experiment in this paper uses the stepwise learning rate decay. How does the stable $\eta$ in stepwise learning rate decay lead to unstable training?

2. The paper claim that *Fig. 2 suggests given long enough training time, the optimal weight decay hyperparameter tends to be zero.* However, I think Fig.2 does not show the *optimal* learning rate tends to be zero when the training time is long. To support the claim, the figure should show the result of the same long training time using different weight decay.

3. Figure 3 shows that increasing weight decay is more effective than increasing learning rate when a large batch size is used. This phenomenon can be explained by the effective learning rate argument in scale-invariant neural networks, which claims that increasing weight decay has the effect of increasing the learning rate. The experiment in Fig. 3 is done using a ResNet, i.e. a scale-invariant network. So I wonder if we have the same effect when a standard CNN like VGG is used. 

4. The test error curves in this paper (Fig.4 and 5) show a severe overfitting effect when using AdamW, which is not consistent with my experience. Could the authors give some comments on this effect? Is there any existing paper that shows the same overfitting effect of AdamW?

5. Equation (20) and (21) seem to have typos about $\theta^*$ in both right and left hand sides.

In summary, I would say that I enjoy the paper's clarity and structure but some main technical issues prevent it being published as its current status.Thus, I suggest settling the conflicts in theory and practice in an updated version. 


[1] Arora, Sanjeev, Kaifeng Lyu, and Zhiyuan Li. "Theoretical analysis of auto rate-tuning by batch normalization." International Conference on Learning Representations. 2019. https://arxiv.org/abs/1812.03981

[2] Li, Zhiyuan, and Sanjeev Arora. "An Exponential Learning Rate Schedule for Deep Learning." International Conference on Learning Representations. 2020. https://arxiv.org/abs/1910.07454

[3] Li, Zhiyuan, Kaifeng Lyu, and Sanjeev Arora. "Reconciling modern deep learning with traditional optimization analyses: The intrinsic learning rate." Neural Information Processing Systems. 2020. https://arxiv.org/pdf/2010.02916.pdf

summary_of_the_review: Despite the effectiveness of the proposed AdamS in small-scale datasets, the theoretical analysis about weight decay does not respect the scale invariant property of some important neural networks, which are used in the empirical validation of this paper to demonstrate some results of its analysis. This results in a severe conflict between theory and practice in this paper, so I would suggest a further revision for this manuscript.
review_title: null

Meta-review: This paper analyzes the effects of the weight decay hyperparameter, and based on this analysis, proposes methods to schedule the weight decay. Overall, while I'm glad that more work is being done on understanding the effects of weight decay, I don't think this submission is of sufficient quality for ICLR.

Theorem 1 is simply re-expressing the well-known fact that if the regularization version of weight decay is used, then (simply because it's based on a single objective function) the stationary points are invariant to the choice of learning rate. This may not be apparent due to the misused terminology: "invariant" is referred to as "stable", but "stable stationary point" has a technical meaning very different from the one used here.

Corollary 2 essentially shows that the optimum of the regularized loss is different from the optimum of the unregularized loss. The authors conclude from this that the optimal value of lambda is 0 from the perspective of test error, which is unwarranted.

Overall, the paper centers around the interaction between learning rates and the weight decay parameter. However, as various reviewers point out, this interaction has been analyzed in detail for networks with normalization layers, and normalization completely changes the nature of the interaction. So any analysis would either need to take this into account or limit the scope to networks without normalization.

I encourage the authors to take the reviewers' feedback into account and improve the paper for the next submission cycle.
Decision: Reject