Venue: ICLR 2018 conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper presents an alternative way to implement weight decay in Adam. Empirical results are shown to support this idea.

The idea presented in the paper is interesting, but I have some concerns about it.

First, the authors argue that the weight decay should be implemented in a way different from the minimization of a L2 regularization. This seems a very weird statement to me. In fact, it easy to see that what the authors propose is to minimize two different objective functions in SGDW and AdamW! I am not even sure how I should interpret what they propose. The fact is that SGD and Adam are optimization algorithms, so we cannot just change the update rule in the same way in both algorithms and expect them to behave in the same way just because the added terms have the same shape!

Second, the equation (5) that re-normalize the weight decay parameter as been obtained on one dataset, as the author admit, and tested only on another one. I am not sure this is enough to be considered as a scientific proof.

Also, the empirical experiments seem to use the cosine annealing of the learning rate. This means that the only thing the authors proved is that their proposed change yields better results when used with a particular setting of the cosine annealing. What happens in the other cases?

To summarize, I think the idea is interesting but the paper might not be ready to be presented in a scientific conference.
review_title: Review

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper investigates weight decay issues lied in the SGD variants, especially Adam. Current implementations of adaptive gradient algorithms implicitly contain a crucial flaw, by which weight decay in these methods does not correspond to L2 regularization. To fix this issue, this paper proposes the decoupling method between weight decay and the gradient-based update.

Overall, this paper is well-written and contain sufficient references to note the overview of recent adaptive gradient-based methods for DNN. In addition, this paper investigates the crucial issue in the recent adaptive gradient methods and find the problem in weight decay. This is an interesting finding. And the proposed method to fix this issue is simple and reasonable. Their experimental results to validate the effectiveness of their proposed method are well-organized. In particular, the investigation on hyperparameter spaces shows the strong advantage of the proposed methods.
review_title: Novel investigation and insight about weight decay in SGD variants

final_score: rating: 7: Good paper, accept
review_content: At the heart of the paper, there is a single idea: to decouple the weight decay from the number of steps taken by the optimization process (the paragraph at the end of page 2 is the key to the paper). This is an important and largely overlooked area of implementation and most off-the-shelf optimization algorithms, unfortunately, miss this point, too. I think that the proposed implementation should be taken seriously, especially in conjunction with the discussion that has been carried out with the work of Wilson et al., 2017 (https://arxiv.org/abs/1705.08292).

The introduction does a decent job explaining why it is necessary to pay attention to the norm of the weights as the training progresses within its scope. However, I would like to add a couple more points to the discussion: 
- "Optimal weight decay is a function (among other things) of the total number of epochs / batch passes." in principle, it is a function of weight updates. Clearly, it depends on the way the decay process is scheduled. However, there is a bad habit in DL where time is scaled by the number of epochs rather than the number of weight updates which sometimes lead to misleading plots (for instance, when comparing two algorithms with different batch sizes).
- Another ICLR 2018 submission has an interesting take on the norm of the weights and the algorithm (https://openreview.net/forum?id=HkmaTz-0W&noteId=HkmaTz-0W). Figure 3 shows the histograms of SGD/ADAM with and without WD (the *un-fixed* version), and it clearly shows how the landscape appear misleadingly different when one doesn't pay attention to the weight distribution in visualizations. 
- In figure 2, it appears that the training process has three phases, an initial decay, a steady progress, and a final decay that is more pronounced in AdamW. This final decay also correlates with the better test error of the proposed method. This third part also seems to correspond to the difference between Adam and AdamW through the way they branch out after following similar curves. One wonders what causes this branching and whether the key the desired effects are observed at the bottom of the landscape.
- The paper concludes with "Advani & Saxe (2017) analytically showed that in the limited data regime of deep networks the presence of eigenvalues that are zero forms a frozen subspace in which no learning occurs and thus smaller (e.g., zero) initial weight norms should be used to achieve best generalization results." Related to this there is another ICLR 2018 submission (https://openreview.net/forum?id=rJrTwxbCb), figure 1 shows that the eigenvalues of the Hessian of the loss have zero forms at the bottom of the landscape, not at the beginning. Back to the previous point, maybe that discussion should focus on the second and third phases of the training, not the beginning. 
- Finally, it would also be interesting to discuss the relation of the behavior of the weights at the last parts of the training and its connection to pruning. 

I'm aware that one can easily go beyond the scope of the paper by adding more material. Therefore, it is not completely reasonable to expect all such possible discussions to take place at once. The paper as it stands is reasonably self-contained and to the point. Just a minor last point that is irrelevant to the content of the work: The slash punctuation mark that is used to indicate 'or' should be used without spaces as in 'epochs/batch'.

Edit: Thanks very much for the updates and refinements. I stand by my original score and would like to indicate my support for this style of empirical work in scientific conferences.
review_title: Important work supported with experiments

Meta-review: This paper generated quite a bit of controversy among reviewers. The main claim of the paper is that Adam and related optimizers are broken because their "weight decay" regularization is not actually weight decay. It proposes to modify Adam to decay all weights the same regardless of the gradient variances.

Calling Adam's weight decay mechanism a mistake seems very far-fetched to me. Neural net optimization researchers are well aware of the connection between weight decay and L2 regularization and the fact that they don't correspond in preconditioned methods. L2 regularization is basically the only justification I have heard for weight decay, and despite rejecting this interpretation, the paper does not provide an alternative justification.

Decoupling the optimization from the cost function is a well-established principle. This abstraction barrier is not completely clean (e.g. gradient noise has well-known regularization effects), and the experiments of this paper perhaps provide evidence that the choices may be coupled in this case. This is an interesting finding, and probably worth following up on. However, the paper seems to sweep the "decoupling optimization and cost" issue under the carpet and take for granted that the decay rate is what should be held fixed. All three reviewers found the presentation to be misleading, and I would agree with them. While there may be an interesting contribution here, I cannot endorse the paper as-is.

Decision: Reject