Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: ## Summary

In this paper the authors introduce the notion of stable weight decay. The stable weight decay property can be defined in dimension 1 as follow: the effective learning rate represents an amount of time ellapsed between two iteration. The weight decay factor normalized (in log space) by the time ellasped should be constant across iterations.

From their framework, the authors also propose SGDS, a minor modification to SGD where the amount of L2 regularization is increased with momentum, in order to balance for the larger step sizes that the momentum will yield.

When applied to Adam, the authors derive AdamS, supposed to work better than the previous AdamW, which already improved how weight decay and Adam interact. AdamS has the stable weight decay property, unlike Adam or AdamW. AdamS weight decay amount is scaled by the denominator of Adam, taking the average over all dimensions to make it isotropic.

The authors test their methods on vision tasks, where Adam is known to underperform compared to SGD, and their method AdamS achieves significant gains compared to AdamW.

## Review

The reformulation introduced in equation (3) is an interesting alterntive view to look at weight decay, but I find it is not really used by the authors. The definition of the weight decay rate can be done from equation (2).
The authors introduce the notion of stable weight decay and seem to right away assume it is a desirable property. In particular, there is no theoretical justification that this is the case.

The benefit from SGDS is limited, except for hyper parameter tuning (as only the first few iterations of SGD are "unstable"), but it serves as a nice illustration of the new concept.

The part on adaptive methods is more interesting but the authors deviate significantly from their theoretical framework. Verifying the stable weight decay property is actually not optimal, because it is not isotropic. The authors trick is to average the value of the moving average of the squared gradients along all dimensions before using it to rescale the weight decay.

The experimental section of the paper focuses only on vision. It would have been interesting to see the effect of AdamS on other type of tasks.

Overall I think the idea introduced by the author is interesting, although the theory is not completely coherant. The experiments shows significant improvement but could have been on a more diverse set of tasks. Still,  I recommend acceptance.


## Remarks

- In the Introduction, talking about adaptive methods: "are a class of dominated methods to accelerate", I'm not sure what dominated means here.
- what is the point of having $\beta_3$? after equation (4), the authors say "SGD with momentum is $\beta_3=0$", that doesn't seem right, if $\beta_3=0$, then the gradient is completely ignored.

=============
Update avec rebutal and discussion with AC and reviewers.

After discussing with the others reviewers and AC, I have come to share their concerns with the overall fragility of the paper. We agreed that the methods is sound and likely to work better than AdamW, the proofs are not sufficient. In particular, the authors should strive to provide experiments on different training sets (ImageNet) with learning rate cross validation. The authors do not systematically compare across learning rates which make it hard to interpret the results as being conclusive. In fact only CIFAR-10 is evaluated with multiple learning rates.

The remark by Elya Loshchilov should also be adressed. Note that you cannot use stochastic noise as a justification, because for heavily overparameterized neural network, the amount of stochastic noise at the optimum is zero (i.e. perfect fitting of the training set). But there is another explanation: Intuitively for a fixed $\beta_2$, $v_t$ goes to zero as the current gradient goes to zero, and the ratio of the gradient by $v_t$ will converge to some constant, which prevents convergence. $v_t$ goes to zero at the same speed as the gradient but with a delay of $1 / (1 - \beta_2)$. If there is no convergence, then the gradients won't actually go to zero.
The only way to prevent $v_t$ from going to zero is to have $\beta_2 \rightarrow 1$ (i.e. the previously mentioned delay going to infinity), but in that case $\bar{v}_t$ won't go to zero neither. This is only an idea of a possible justification and I would encourage the authors to think carefully about this stability issues in the next revisions.

Finally I would encourage the authors the remove from the theoretical analysis parts that are not actually used (which I and other reviewers have noted).
review_title: Review of Stable Weight Decay Regularization

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary:
This paper presents a novel framework that alters the weight decay update rule and aims to improve generalization when applied to 1) momentum-based optimizers and 2) adaptive optimizers. The framework includes the concept of “weight decay rate” and “total weight decay”, where the idea is to make the “weight decay rate” constant throughout training. The framework is applied to “fix” or “stabilize” weight decay when applied to SGD with momentum and Adam (which effectively amounts to changing the weight decay parameter). The result is better generalization when applied to a variety of image classification tasks, compared to the appropriate baselines. 

Strengths:
The core of the idea is novel, relatively simple, and the experiments support the claim.

Weaknesses:
My main complaint is that this paper is poorly written. I don’t understand the motivation behind the reparameterization of the update rule with w instead of \theta. The concept of “weight decay rate” and “total weight decay” seems to be the most critical part of the paper, but the explanation surrounding them was messy and hard to understand. The language used is not precise at some points in the text.

At the current state of the paper, I recommend a reject because I don’t think it’s polished enough to be published. 

Comments and questions:
- I think the concept of weight decay rate and total weight decay should be explained better, and earlier in the paper. Without knowing what “stable” meant, I had a hard time taking the introduction seriously, because I didn’t know what “unstable” meant exactly, and that the decoupled weight decay method was “unstable”. 
- “\beta_3 = 0” should be “beta_3 = 1”. The Pytorch documentation calls 1-beta_3 dampening and its default value is zero.
- I’m not sure I understand the sentence “Although <\theta_{t-1}> is an approximated value of \theta_{t-1}, replacing \theta_{t-1} by <\theta_{t-1}> still introduces undesirable noise into learning dynamics.” What do you mean by undesirable noise? Why is replacing <\theta_{t-1}> by <\theta> justified? 

===== Update =====

I have read the authors’ response, the updated paper, and the other reviews. I believe that the changes made by the authors address my concerns about the motivation and the lack of clarity; section 2 reads much better now. 

It seems like the main complaints of the other reviewers are in the lack of more difficult workloads, and the lack of theory. I personally don’t find the lack of theory very important. I think the novelty comes from the simple observation, which no one to my knowledge has come to before, and the experiments support the idea empirically (which I think is what actually matters). I also find it a bit uncomfortable penalizing the authors for not running experiments on ImageNet, and I think the variety in architectures that the authors tried, compensates for this. I do agree that a more modern set of workloads (transformers, or even the same setup as AdamW) would have made the paper much stronger. 

I increased my score to a 6 because I think the paper in its current form is enough to get accepted, but there are still improvements that could be done to make it much stronger.
review_title: Review of "Stable Weight Decay Regularization"

final_score: rating: 5: Marginally below acceptance threshold
review_content: In this paper, the authors study the effect of weight decay across different optimizers. When one uses weight decay, the learning rate which multiplies the weight decay is different from the effective learning rate (the term multiplies the gradient dependent contribution). The authors propose to adjust for this by having the effective learning rate multiplying the weight decay term, ie Delta theta= -eta_{eff} lambda theta-eta_{eff} F(gradient). For SGD, this implies that one can roughly equate weight decay and L2 regularization by rescaling the L2-parameter. For the case of Adam, the effective learning rate is anisotropic, which they fix by taking the mean of the vector 'v'. The authors show that after correcting for the effective learning rate, which they call AdamS, they can get the same performance as what one gets with SGD in CIFAR10 and CIFAR100. 

I find the discussion about rho and R confusing (they are not defined precisely and using the small lr/lambda approximation seems unnecessary) and they do not really add much.  Similarly the term unstable seems a little strong given that there is nothing bad going on with training of such models. They also make comments about reinterpreting weight decay as flattening the loss and increasing the learning rate which they don't pursue further nor connect with the main point and it seems a little out of context. The main conceptual point of the paper is simply that the weight decay should have the same effective learning rate as the gradients, it would be nice if the authors could make this more clear and more central.

The paper main point then is to equate the learning rate in the weight decay coefficient by the effective learning rate and they show that this might be enough to bridge the gap between Adam and SGD. I think this is an interesting problem, but given that this is their unique point they should probably back it up with more experiments, since they only check it for CIFAR datasets (4 experiments total). If the authors could do one extra experiment on Imagenet (for example with a standard Resnet-50), that would raise my score to weak accept.  
review_title: Interesting proposal to bridge the gap between Adam and SGD but would like to see more experiments.

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary: This paper presents a novel weight decay regularization, stable weight decay by using the bias correction to the decoupled weight decay in adaptive gradient descent optimizer. They empirically found that the L2 regularization the decouple weight decay are unstable weight decay. The proposed stable weight decay can fix this issue in terms of dynamical perspective. Experimental results based on benchmark dataset show that the proposed scheme outperform the popular and advanced optimizers in generalization. Weight decay has been a basic technique in most optimizer and indeed there are not many studies for the effect on the performance. Also, how to obtain the optimal weight decay is still missing. 

I think this paper has done a decent investigation on this topic. The overall paper is easy to follow and seems technically sound. 

However, one major issue in my mind is that conclusions are only supported by the empirical results, which though look promising. No formal theoretical claims or results have been reported. This hurts the paper in terms of theoretical novelty. Even though some shallow analysis has been presented in the draft, it is not enough. 

Moreover, a couple of statements can be arrived at by directly observing existing algorithms, which are not that technical. For example, Statement 1 that says “Equation-1-based weight decay is unstable weight decay in the presence of learning rate scheduler.” can be quickly summarized even from an intuitive sense without any derivation, which makes it trivial. There is no need to give Definition 1 formally for Stable Weight Decay as that doesn’t sound like a definition. In the draft, the authors can directly say constant or time-varying weight decay. Stable doesn’t just mean constant, though I can completely understand what the authors really meant in the paper. Statement 4, in my opinion, is one of key results in the draft. However, the current theoretical analysis is not enough to support this conclusion. More formal theoretical analysis is required.

A minor point, in Eq 3, how did the authors arrive at $-2t-1$ in the superscript of weight decay rate, not $-2t+1$? Additionally, I am a little confused about the statement that the effect of weight decay can be interpreted as flattening the loss landscape of $\theta$ by a factor of $(1-\eta\lambda)$ per iteration and increase the learning rate by a factor of $(1-\eta\lambda)^{-2}$ per iteration. Can the authors put more detail, a bit more of derivation to show this?

***************************************
After reading the rebuttal and considering carefully, I think the authors' response addressed some of issues in my mind so I raised the score. However, in terms of theoretical foundation, the current paper draft is still only marginal, which requires substantial improvement. 

review_title: Review of "Stable Weight Decay Regularization"

Meta-review: The paper proposes a novel way to have weight decay-like update rule. Empirically, the authors claim that it improves generalization when applied to momentum-based optimizers and optimizers with coordinate-wise learning rates.

This paper has been thoroughly discussed, both in public and private mode.
The strength of this paper lies in the possible gain in generalization performance due the proposed change.
The weaknesses are:
- the very confusing and not scientific motivation of the proposed change
- the experiments are not fully convincing

More in details, we all found the discussion on "stable" and "unstable" weight decay extremely confusing. The claim of the paper is that "stable" weight decay should be preferred over "unstable" one. However, to validate a scientific claim it is necessary to carry out an empirical or theoretical evaluation. The theoretical one is simply missing: a number of proposition and corollaries are stated with some simple mathematical facts completely disconnected from the optimization or generalization issues. As it is, removing these arguments would actually make the paper better.
On the empirical side, there is no experiment that supports the simple claim that "the unstable weight decay problem may undesirably damage performance". Instead, what we see are experiments in which the modified update rule seem to perform better, but they don't actually show that "stability" or "instability" are the specific issues at play here. Indeed, any other explanation is equally valid and the experiments do not support any specific one, but rather they can only support the claim that the proposed algorithm might be better than some other optimization algorithms. The *specific reason* why this is happening is not clear.

Turning to the empirical evaluation, the discussion elicited the fact that, a part for CIFAR10, the experiments are carried out without tuning of the learning rates. Hence, it is difficult for us to even validate the claim of superiority of the method. I don't subscribe to the idea that a deep learning paper requires experiments on ImageNet to be valid. Yet, given that there is no supporting theory in this paper, the empirical evaluation should be solid and thorough.

For the above reasons, the paper cannot be published at ICLR.
Decision: Reject