Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper derives SDE approximations of RMSProp and Adam when their hyperparameters changes according to a scaling rule. Experiments on VGG and ResNets were performed on CIFAR-10.
strengths_and_weaknesses: 
Pros:
- The scaling rules in Theorem 5.3 is interesting.
- The SDE approximations of RMSProp and Adam are derived, and they appear to be correct.

Cons:

- Experiment results (e.g., Fig 1-3) do not have confidence bands. Furthermore, in Figure 1 (a) and (b) all curves are very close but they are quite far away in (c) and (d). Does this mean the scaling rule is more respected by Adam but not RMSProp? I found it a bit hard to interpret the results. Perhaps the authors can add some interpretations in the figure caption and in the main texts.
- Why in Figure 2(b) the curves do not start at the same point?
- The proof technique seems not new, the theorems look more like extensions to the stochastic modified equation literature. Perhaps I missed something. Can the authors discuss more in the main text how the new analyses differ in technique from those literatures?
- As some literature suggested, the gradient noise might be heavy-tailed, it is thus open to discuss which of the model (Gaussian vs. Lévy) makes more sense in practice — when would Gaussian noises occur in training practical neural nets?
questions: Please see the "Strengths and Weaknesses" section.
limitations: As there are existing work studying similar problems, especially the Lévy process based SDEs, and some existing work suggests the gradient noises are heavy-tailed. It might be instructive to study whether SDEs in this paper makes more sense than existing work, and to compare and contrast their use cases in practice.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper attains the SDE approximations for two popular optimization algorithms RMSprop and Adam, and validates their applicability empirically. The SDE approximations generate square-root scaling rules, which are validated in the vision and language modeling domains.
strengths_and_weaknesses: Strengths:   
**1)** The paper is generally well-written and the presentation is clear.   
**2)** The theoretical results are new and interesting. The empirical results are provided to validate the derived theories.   
**3)** I did not check every detail of the proof but generally it is technically sound. The assumptions are also supported by empirical results.       

Weaknesses    
I did not find any obvious weakness of the paper.
questions: 1.  [1] proposes an exponential learning rate schedule. How is the square root schedule proposed in the paper compared to the exponential schedule?   
2. [2] employs Levy SDE approximation. Can the authors explain why "the=e quality of the Lévy SDE approximation was not formally guaranteed" and the advantage of using Ito SDE in the paper?

References

[1] Zhiyuan Li and Sanjeev Arora. An exponential learning rate schedule for deep learning. arXiv preprint arXiv:1910.07454, 2019.   
[2] Pan Zhou, Jiashi Feng, Chao Ma, Caiming Xiong, Steven Chu Hong Hoi, Weinan E. Towards Theoretically Understanding Why Sgd Generalizes Better Than Adam in Deep Learning. NeurIPS 2020.   
limitations: Yes.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper derives SDEs corresponding to the dynamics of RMSProp and Adam. Based on the SDEs, the paper proposes to scale the learning rate by $\kappa^{-1/2}$ when the batch size is scaled by $\kappa$. Finally, the paper proposes a SVAG-type algorithm to simulate the training dynamics of the adaptive algorithms and 
strengths_and_weaknesses: Strengths:
1. The paper derives the scaling scheme for the learning rate of adaptive algorithms, which is very impressive.
2. In addition to the SDEs, an approximation bound is given.

Weaknesses:
1. Readers, including myself, may be unfamiliar with the SVAG algorithm if they do not have adequate knowledge of stochastic simulation. More introduction might be required in this part.
2. The SDEs and scaling schemes have been proposed in the literature, though these results have never been presented in such a rigorous way as this paper. The results of this paper might be a little incremental.
questions: 1. In addition to the learning rate $eta$, parameteters $beta$, $beta_1$ and $\beta_2$ should also be changed if the batch size is scaled by a positive constant. How can this be interpreted?

2. Does the SVAG algorithm for adaptive algorithms have any practical applications other than validating the training dynamics?

3. It is mentioned in subsection 4.1 and inn the first paragraph of subsection 4.2 that the condition $sigma\gg\|g\|_2$ is crutial in deriving the scaling scheme. How is this condition related to the conditions of Theorem 4.2?
limitations: The authors properly address this issue in appendix A.2.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The presented paper presents the SDE approximations for Adam and RMSProp optimizers and provides theoretical guarantees of their correctness. The derived SDE approximations revealed the theoretical explanations of the square-root scaling rule of learning rate with batch size increasing. To verify numerically the correctness of the proposed SDEs, authors extend the SVAG technique to the considered classes of SDEs. Numerical experiments consider computer vision and natural language processing models and confirm the correctness of the SDEs and square-root scaling rule.
strengths_and_weaknesses: Strengths
1) authors present solid theoretical explanation of the empirical rule to manage learning rate with increasing of batch size
2) the expected approximation quality of SDEs is verified in large scale problems from CV and NLP
3) SVAG technique is modified for the considered SDEs to speed up simulations

Weaknesses:
1) the presented analysis assumes the not heavy-tailed noise in the gradient estimates, though some works demonstrate that noise in deep networks is indeed heavy-tailed, see e.g. https://tel.archives-ouvertes.fr/tel-03206456/document and https://arxiv.org/pdf/2106.05958.pdf -  that is left to future work
2) no new practically important consequences of the presented analysis. Researchers use the square root scaling rule previously, now we know why this rule works for adaptive method and why the difference with SGD exists.   


questions: Please comment the following remarks
1) what is $\mathcal{G}$ in line 70?
2) please explain why the test functions given in line 319 are satisfied definition 2.4.?
3) how does the presented analysis correlate with results in https://arxiv.org/pdf/1706.02677.pdf%5B3%5D%20ImageNet and https://arxiv.org/pdf/1904.00962.pdf where the special warm up schedules were used?

limitations: The authors mention limitations in the assumptions before deriving the result SDEs (section 2.3).
review_title: null

Meta-review: metareview: The paper attains the SDE approximations for two optimization algorithms RMSProp and Adam. The authors have addressed the concerns raised by the reviewers during the rebuttal period. All the reviewers agreed that the paper should be accepted at NeurIPS 2022. Please incorporate the reviewers’ suggestions in their detailed reviews and revise the final version of the paper properly. 
confidence: Certain
Decision: Accept