Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: An interesting finding shows that the advantages of AdamW compared to the vanilla Adam-l2.
main_review: Strengths:
1. The main contribution comes from providing the connection between the proximal operator and the AdamW updating, which is interesting.
2. The submission further shows that scale-freeness is an essential property of AdamW compared to Adam-l2. Moreover, scale-freeness can help mitigate the impact of condition number, which significantly affects the convergence speed.

Weaknesses:
1. One problem is that the submission shows that scale-freeness is essential for the optimizer. However, many optimizers share this property, such as adaGrad; this submission does not show why AdamW is generally better than the others 
2. For Line 5 in Alg.1, if we scale the whole gradient $g_t$, we also get the scale-freeness? Do the author mean only scale the gradient of $f$. ie.e,$\nabla f$? It seems that all the adam-type optimizers all have this property. Why is AdamW special?

Minor Problem:
1. 'AdaGrad does not compute Hessians, and there is no reason to believe it approximates them in general.', which confused me. The quasi-Newton method also does not compute Hessians, but it indeed approximates the inv of Hessians. Moreover, scale-freeness certainly is superior in the scenario that the identity matrix can reasonably approximate the hessian. Why deny such a view here?
summary_of_the_review: An interesting paper provides a good finding.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This work first re-interprets AdamW as an approximation of a proximal gradient method, which takes advantage of the closed-form proximal mapping of the regularizer instead of only utilizing its gradient information as in Adam-l2.  Then it further shows that AdamW is “scale-freeness” algorithm which may enjoys faster convergence speed.  In this way, this work may explain why AdamW generalizes better than Adam-L2.
main_review: Strengths:
(1)	It interprets AdamW as an approximation of a proximal gradient method, and shows the difference between AdamW and Adam-L2. 
(2)	The authors also show that AdamW is “scale-freeness” algorithm which may enjoys faster convergence speed

Weaknesses: 
(1)	In the abstract, the authors aim to solve the problem: why AdamW generalizes better than Adam-L2. However, in the paper, they actually did not solve or try to solve this problem. 

(1.1)	They show the difference between AdamW and Adam-L2 from an approximation of a proximal gradient method. But they do not explain why different approximations give different generalizations? Why the approximation of AdamW is better than Adam-L2 in terms of generalization. 

(1.2)	For the second contribution, AdamW is a “scale-freeness” algorithm that may enjoy faster convergence speed. First convergence speed does not have a necessary correlation to generalization, since one can converge to a very sharp minimum and may generalize worse. Secondly, For the “scale-freeness” algorithm, it is only possible to achieve faster convergence rate. But for AdamW, the authors do not show how/why adamw can achieve a faster convergence rate. This is a big gap. 

(1.3) adam is  “scale-freeness”, but its performance is usually worse than adam-l2. So how “scale-freeness” explains this?

(2)	For interpreting AdamW as an approximation of a proximal gradient method and showing the difference between AdamW and Adam-L2, this seems to be very trivial in my view. What is the novelty or contribution of this? 

summary_of_the_review: Overall, this work does not provide very new insights for improving adamw or showing the better generalization of adamw over adam-l2. Moreover, it also does not have new proof techniques/frameworks.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors first interpret AdamW as an approximation of proximal mapping, and then proposed their own algorithm AdamProx. They aim to unravel the connection between AdamW and AdamProx from both theoretical and empirical perspective. They delicately designed some probing experiments to verify their hypothesis. 
main_review: 
Strengths:
- Their work is well-motivated. An insightful interpretation or explanation of AdamW can benefit the whole deep/machine learning community in a way of guiding people on what scenario to use it and how.

Weakness & major concerns:
- The manuscript definitely need robust editing by a native speaker. To me sometimes it is really hard to follow. 
- The math is not self-contained and the logical flow is disconnected. For example, the line above eq(3) "This results in the following update rule ...". But I don't think it is trivial to derive it from previous formulations. TBH, I fail to see how to derive eq(3). I would appreciate if the author could help me to see the connections between eq(3) and previous formulations. 
- I assume eq(5) is the core updating rule of AdamProx, since there is no description of your algorithm. Are you claim that when $\eta_t$ is small enough, then AdamW is an approximation of eq(5)? Is this assumption too strong? When $\eta$ is too small, then the overall actual learning rate is almost zero which results in ineffective update.
- I feel the experiment section just carried out a comparison between AdamW and Adam-$\ell$2. It supposes to prodive more experiments with AdamProx. 

summary_of_the_review: I recommend to reject the paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper give an analysis to the AdamW optimizer. This paper gives a theoretical view of the AdamW optimizing algorithm and connect it to the proximal gradients methods. Authors also explore in depth the advantages of AdamW over AdaM with l2 regularizer, from the perspective of scale-freeness property. By analyzing the scale-freeness property, authors find that the training is more stable in certain scenarios with large conditional numbers. Authors conduct experiments to validate their conclusions. 
main_review: Strengths:
1. Authors show that AdamW can be seen as an approximation of the proximal updates, utilizing the closed form of proximal mapping of the regularizer. It is nice to see the reasonable theoretical explanations to the AdamW optimizer.
2. Authors analyze the scale-freeness property of AdamW, and further show the advantages of the scale-freeness property which is convincing to explain why AdamW gets better performance when the network is deeper or not equipped with batch normalization.

Weakness:
Authors conduct experiments in small-scale datasets with convolutional neural networks. Actually, a more realistic setting is to validate the vision transformer-based architectures, which is more meaningful because (1) transformer is equipped with AdamW as a standard setting (2)  vision transformers are not equipped with the batch normalization. So I thought if the theoretical analysis can be validated in the transformer-based architectures, and with larger dataset such as ImageNet, the results will be more convincing.

Typos: In section 4 the last sentence of the first paragraph, "??" should be modified.
summary_of_the_review: I lean to accept the paper. The paper analyze the AdamW optimizer well, and the experiments on small dataset support their conclusions. It will be more convincing to conduct experiments in more realistic scenarios such as transformer-based architectures trained on ImageNet.
review_title: null

Meta-review: I agree that reviewer Vxer was confrontational and abusive, especially in the response to the author's rebuttal, and believe that some form of sanction or reprimand is appropriate. That said, I do think that "performance" should be evaluated on both convergence rate and generalization.  Figure 3 does suggest some improvement on generalization for deep versions of resnet without batch normalization.

The three less offensive reviewers all indicated weak acceptance.  One reviewer pointed out the weakness of only getting positive results on deep versions of resnet with batch normalization removed.  Results on transformers, where Adam is typically used, would be more compelling.  This is my primary issue with the paper.  It has not demonstrated improvement in the standard practice of resnet (with batch normalization) and has not presented experiments on transformers.  The theoretical analysis is not aimed at explaining why the improvement is only observed on deep resnets with batch normalization removed or why L2 regularization seems to be of no value when batch normalization is present.  I understand that these are very difficult questions.

The paper has no champion and I am personally concerned about the significance of the contribution.
Decision: Reject