Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents a weight decay integrated Nesterov acceleration algorithm for Adam, AdamW, LAMB and SGD by adopting the proximal point method. It shows the convergence analysis, and perform experiments to validate the results. 
strength_and_weaknesses: Strength: The paper is clear and well written. It presents an acceleration algorithm and apply to Adam, AdamW, LAMB and SGD. It has both theretical analysis and experiments to support the claims. The idea introduced is nice.

Weaknesses: 
1. Better to include more comparisons with other methods, such as the line search methods.
2. To consider the s_k norm seems not necessary. What if s_k is partial, like 1/2? 

clarity,_quality,_novelty_and_reproducibility: 1. Clarity and quality: The paper is clear and very well written. It is better to explain the notations and symbols more clear, and make a session for them. The derivation process can be more rigorous. 

2. Novelty: The idea is clear and nice.

3. Reproducibility: Better to include the simulation codes or link to it.
summary_of_the_review: It is a very well written paper.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work focuses on adaptive gradient optimization algorithms and attempts to provide practical insights for speeding up convergence with such methods. In particular, the authors proposed the Win method to increase the speed of convergence for adaptive optimization techniques. In particular, this technique incorporates a dynamic regularizer into the loss that is inspired by proximal point methods, which improves the convexity of the optimization problem. In addition to providing a convergence proof of the Win technique, the authors first implement this technique for Adam/AdamW, then extend the implementation to LAMB (i.e., technique for optimization with larger step sizes) and SGD. The authors provides convergence guarantees for this acceleration technique within the non convex settings, which matches known lower bounds and slightly improves upon the bounds achieved for baseline algorithms like AdaBelief and RMSProp. In experiments, the Win acceleration is shown to provide a very tangible benefit across a suite of different deep learning architectures, including CNNs, LSTMs, and transformers for both vision and language.
strength_and_weaknesses: Strength:
* The incorporation of the PPM regularization term with a recently-proposed formulation of nesterov-type acceleration is novel and seems to be useful. 
* There proposed Win acceleration can be added to several different types of optimizers (e.g., Adam, SGD, AdamW, LAMB).
* The authors provide convergence guarantees in the non convex setting for their acceleration technique using both Adam and AdamW. These guarantees improve upon prior work to the best of my knowledge. 
* Experiments are performed on a wide variety of architectures (CNNs, ViTs, LSTMs, transformers). X-Win improves upon baseline optimization techniques in nearly all cases. 
* The proposed acceleration technique seems to reduce the need for hyperparameter tuning. 

Weakness:
* The discussion of adaptive optimization techniques is lacking. It is widely recognized that adaptive methods suffer poor generalization relative to SGDM in certain settings (see “The marginal value of adaptive gradient methods in machine learning” and “Towards theoretically understanding why sgd generalizes better than adam in deep learning.”). This has led to research in alternative optimizers like AMSGrad, Yogi, M-SVAG, etc. The authors mention AdamW, but do not mention the rest of this work or sufficiently address the generalization ailments of adaptive methods. I think because of these issues, claiming that optimizers like Adam/AdamW are “default optimizers to train DNNs” is somewhat problematic/incorrect. SGDM is still the default choice for most computer vision tasks, where event recent adaptive algorithms like AdamW achieve inferior performance. 
* The description of the proposed method (especially in the introduction) is lacking in clarity. I believe the paper could be greatly improved if the authors spend time to improve the clarity of these initial explanations of their technique. 
* No discussion of how the computational complexity of the proposed method compares to the other methods (maybe I missed it?). You need to state this comparison, do we incur extra cost by adding Win?

Small stuff:
* Small writing errors: beginning of Section 3 (“accelerating and also stabilizing optimization”, “At below”), end of section 3.1 (shouldn’t “bias correlation” be “bias correction”?).
* Putting AdamW, Adam, and LAMB variants all into Algorithm 1 is a bit confusing, though I’m not sure if there is a better way to present this. 
* It is not completely clear to me from your description how Win+LAMB compares to vanilla LAMB. Making this comparison a bit more directly would be helpful in my opinion. 
* It is a bit weird that Table 1 appears after Table 2 in the writing. 
* For the loss plots (Fig. 1), I’m not sure whether your method actually converges faster, or if it simply converges to a better loss. In some cases, the convergence does appear to be faster, but in others it converges at (seemingly) the same to a better loss. It would be helpful to provide a similar plot with some of the other baseline optimizers (e.g., PAdam, RAdam, Adabelief, etc.) if it is does not require too much extra effort. 
clarity,_quality,_novelty_and_reproducibility: My main concern with this work is the clarity of Sections 1 and 3. I believe the clarity of writing and presentation for this work could be improved, which would make the technique much easier to understanding (i.e., try to be less verbose in writing, organize the ideas better with a clear structure, and be more clear in your explanations). If the authors focus upon improving this presentation, I believe the work could be very well received and impactful.

Novelty/quality/reproducibility seem good. It would be nice for authors to provide a reference implementation of Win (e.g., in PyTorch) for the different optimizers they consider.
summary_of_the_review: To begin, I want to emphasize that this review reflects my current opinion of the work. I am completely open to discussion with other reviewers/authors, and my final score will be mostly based upon this discussion.

My initial thoughts are as follows:
* The idea in this paper is unique and (in my opinion) useful to the community.
* The empirical (and seemingly theoretical, though I'm less familiar with Adam-style analysis) support for this technique is very strong.
* The presentation quality of the work is quite poor (especially Sec. 1, 3).

I believe that if the authors improve the clarity of their discussion/presentation of the technique, the impact of this work will significantly increase. I do not necessarily believe poor writing is a reason for rejection, especially given the strong empirical/theoretical support for Win. However, I truly believe that spending time with improving the writing of this paper (and fixing the other details I mention in the body of my review) will increase this work's potential significantly.

I thank the authors for their interesting work, and I look forward to the subsequent discussion!
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper provides a new analysis of combining weight decay and Nesterov momentum and applying it to several standard first order optimizers (SGD, Adam, AdamW, LAMB). The authors prove convergence, and then show that the new acceleration provides significant improvements on a variety of training tasks.
strength_and_weaknesses: The technique presented is well motivated theoretically, and is straightforward to implement. The experiments are quite diverse and show promising improvements across the board.

The organization of the paper could be a bit better --- right now the authors present the fully general technique and then specialize it to various algorithms, it might be better to present the technique for various algorithms one at a time, starting with SGD, which is the simplest algorithm. 

The combination of weight decay and Nesterov momentum has been applied in other optimizers already (LARS, Shampoo, several others).
clarity,_quality,_novelty_and_reproducibility: The paper is quite clear. The novelty is a bit less clear, as the combination is used before, and the derivation from proximal point methods is well known.
summary_of_the_review: The paper presents an analysis of combining weight decay with nesterov momentum. The technique has been applied to several different optimizers, and is easy to implement in practice. The experimental results show significant gains across a variety of ML tasks.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes a new algorithm of combining weight decay and Nesterov acceleration, called WIN (Weight-decay Integrated Nesterov acceleration). It is applied to extend several standard first order optimizers (Adam, AdamW, LAMB and SGD). The authors provide convergence proof, and experimental evidences that the new acceleration provides significant improvements on a variety of training tasks, including CNNs, LSTMs, and transformers for both vision and language. The proposed idea is unique, with strong theoretical and empirical supports. All the reviewers unanimously accept the paper, so is the final decision. 
justification_for_why_not_higher_score: N/A.
justification_for_why_not_lower_score: Adaptive optimizers recently becomes more important in training large scale deep neural networks, particularly for transformers. The Nesterov acceleration combined with weight decay proposed in this paper shows tangible improvement over standard adaptive optimizers with a theoretical convergence support. So this paper can be a spotlight that may help the practice in speeding up training popular neural network architectures.
Decision: Accept: notable-top-5%