Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work extends the analysis framework of (Blanc et al., 2020) and (Li et al., 2022) to study the dynamic of heavy ball momentum along a manifold of minimizers. The authors proposed a momentum scaling rule $\beta = 1 - C\eta^\gamma$ and discovered the optimal scaling power as $\gamma=2/3$ under the analysis framework. The authors also provide empirical verifications to the theoretic prediction on 2-layer neural networks, matrix sensing and ResNet-18.
strength_and_weaknesses: Strength:
- Understanding the best momentum setting is critical in neural network optimization
- The discovery of the optimal momentum scaling, and its connection to finding flatter minima are very interesting results
- Solid theoretic results with clear motivations

Weakness:
- In addition to identifying the best momentum setting, could the authors also provide some theoretic insight of the momentum acceleration? Such as discussing the effect of $\beta > 0$ compared with $\beta = 0$.
- This work would benefit from more empirical evaluations to demonstrate the practical implication of the discovered scaling rule
- Is the label noise of SGD only an analytic artifact? Is there any empirical reason to add such label noise?
- The scaling rule $\beta = 1 - C\eta^\gamma$ seems arbitrary. Is there any reason for such a choice?
clarity,_quality,_novelty_and_reproducibility: The quality of presentation seems to be of standard
summary_of_the_review: This work studies an important problem (finding better momentum values), and discovers a new optimal momentum scaling under the analysis framework. The practical implication seems to be weak unless the authors could further elaborate.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper analyzes the SDE for SGD with momentum and label noise and tries to show that adding momentum accelerates the convergence but not hurting the generalization. This is an important challenge while training big models with numerous training datasets such as DNN. They show that there is an interplay between the speedup of momentum and the limiting diffusion generated by SGD noise. Using that they show two characteristic timescales associated with the training dynamics, and the longest timescale essentially governs the training time. Maximum acceleration is thus reached when these two timescales coincide, which leads to identifying an optimal scaling of the hyperparameters.
strength_and_weaknesses: Strength:
 This paper analyzes the SDE for SGD with momentum and label noise and tries to show that adding momentum accelerates the convergence but not hurting the generalization. This is an important challenge while training big models with numerous training datasets such as DNN. Their analysis provides a theoretical basis to set the momentum hyperparameter. 

Weakness: 
My major concern with regard to the paper is its writing. Mainly the concepts and findings are not explained well formally. It could be more readable if they explain more about the notation they use and also not overloading. 

clarity,_quality,_novelty_and_reproducibility: Analysis of SGDM with this SDE while their proposed scaling regime is novel. All the claim in the paper has theoretical justification. 
summary_of_the_review: 1- My major concern with regard to the paper is its writing. Mainly the concepts and findings are not explained well formally. It could be more readable if they explain more about the notation they use and also not overloading. 

2- They show that for specific \gamma= ⅔ it SGD with momentum (SGDM) can reach best performance. However, their empirical results don’t provide strong evidence. It would be helpful to run SGDM with the proposed hyperparameter over several DNN models and compare the test accuracy with SGD and other popular optimization methods. 

3- For the analysis of the phase when SGDM moves toward zero loss valley they assume GDM instead of SGDM. However, there is no enough justification for why this is true.

4- In eq. 2 no justification for why the last approximation is true. 

5- In eq. 3, since we are in zero valley, isn’t \nalba L (w_i) = 0? But the approx in the next line is not zero.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper studies the convergence of momentum SGD at the vicinity of global minima in the over-parameterized setting. With some approximations which only hold in this vicinity, the scaling analysis predicts an exponent $\gamma=2/3$ as the scaling of the momentum parameter that maximally accelerates the training.
strength_and_weaknesses: First, I have to mention that the paper is not well written, and it takes a lot of effort for readers to understand. The authors should consider improving the clarity of the paper. I list a few points below:

> There should be definition or clarification of notations before using them. For example, in the 5th line of Sec 3.1, “the parameter $\epsilon > 0$ controls the size of the noise …”. However, there is no $\epsilon$ mentioned before. Hence, it is hard to understand what this line is talking about. In addition, in Assumption 3.1, $C^3$ appears before the notation definition of $C^n$. Furthermore, in Sec 3.3, without defining $\delta x$, it is hard to understand the equation $\delta x_{t+1} = J(x^*)\delta x_t$. 

> The paper does not set up the problem well. The paper has some settings in the introduction, Sec 1.1, and some other in Sec 3.1. None of the subsections established the problem setting; instead, the two sections are referring to each other, and readers have to go back and forth to get the context.

> There should be a discussion of the connection between the time scaling analysis and the ordinary SGD setting. In machine learning, SGD is an iterative algorithm taking on discrete time steps with predefined learning rates. It is important to clarify what the timescale means to SGD, and why there is a parameter $\epsilon$ controlling the size of noise and can be taken to zero. 

Second, the modeling of the momentum SGD using Eq.(1) is not appropriate. As in practice, the noise of SGD arises from the estimation of gradients via samples, instead of an additive noise as in Eq.(1). The difference in how the noise is generated results in different properties of the noisy gradient. For example, in the over-parameterized case, the stochastic gradient has been found to have a vanishing noise as the algorithm converges to a global minimum, see automatic variance reduction in [1]. However, the additive noise often keeps a certain level of noise even if it is close to convergence. In addition, it has been found that the variance of stochastic gradients cannot be bounded by a constant (see [2]), while the additive noise usually can. 
Hence, it is better to start the analysis from following equation (instead of Eq.(1)): $\pi_{k+1} = \beta \pi_k - \nabla l_i(w_k)$, where $l_i$ is the loss function evaluated on one training sample. 

Third, I could not understand why the $\approx$ holds in Eq.(2). In principle, the momentum $\pi_{k+1}$ should contain gradient noise, and should not be averaged over. Removing the noise is critical to the final conclusion. I hope the authors can explain this.

Fourth, the paper does not explain why Eq.(5) solves Eq.(1). This is a key step, and should not be omitted from the paper. 

Fifth, the analysis is conducted assuming in the vicinity of the global minima, namely very close to the solution. Many of the assumptions do not hold outside of the vicinity, and the analysis does not apply (for example, the dominance of perpendicular displacement over the longitudinal). Hence, it is possible that the theoretical optimal power of $\gamma=2/3$ is not optimal outside of the vicinity. 

Overall, as for the $\gamma=2/3$ rule, the theoretical analysis in Sec 3.3 is not rigorous (as admitted informal by the authors), but it experimentally works well (as in the ResNet). In addition, as mentioned above in the fifth point, the theory applies to the vicinity of the global minima, and does not match the experimental setup. Given these, I would consider the $\gamma=2/3$ rule as a good finding that is worth trying by practitioners, but I don’t think the theory supports it well. 


[1] Accelerating SGD with momentum for over-parameterized learning

[2] Better Theory for SGD in the Nonconvex World

clarity,_quality,_novelty_and_reproducibility: [clarity] the clarity of the paper should be improved
[quality] several key claims are not well justified
summary_of_the_review: The the paper is not written clearly, and it took much effort for reading.

Several key claims are not well supported, with the proof or justification missing. Without them, I am susbicious of the correctness of the results.

The analysis is conducted within the vicinity of global minima, outside of which many assumptions or treatments do not hold. However, in practice most of the optimization happens outside of this vicinity. Hence, I doubt whether the results of the paper is applicable in practice.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: - Summary:
This paper analyzes the SDE for SGD with momentum and label noise and tries to show that adding momentum accelerates the convergence but not hurting the generalization. This is an important challenge while training big models with numerous training datasets such as DNN. They show that there is an interplay between the speedup of momentum and the limiting diffusion generated by SGD noise. In particular, this paper studies the convergence of momentum SGD at the vicinity of global minima in the over-parameterized setting. With some approximations which only hold in this vicinity, the scaling analysis predicts an exponent $\gamma = 2/3$  as the scaling of the momentum parameter that maximally accelerates the training.

- Strengths:

1. Interesting SDE-based analysis of SGD for DNNs;
2. The discovery of the optimal momentum scaling is an interesting topic; the holds for the topic of connecting momentum scaling to finding flatter minima;

- Weaknesses:

After the rebuttal discussion, some concerns remain:

1. Overall, any algorithmic modification should be strongly supported by thorough experimental evaluations. Using shallow NNs on small datasets is not sufficient to support the claim that scaling momentum as suggested should be used for large-scale, large-dataset scenarios. While we acknowledge that access to such hardware is not always available, smaller but complicated architectures (like ResNets, CNNs, etc) and different datasets (computer vision, NLP, etc) could be used to validate the momentum scaling. Currently, the experiments considered might validate some of the claims, but the outcome of the paper might be used for other settings with no guaranteed performance. 

2. After discussions, some comments by reviewers remain unresolved: 

- It is questionable to take the limit of large minibatch size in SGD to control the stochastic noise. 
- Theoretically speaking, the theory assumes momentum be exactly zero, so as long as momentum is non-zero (regardless of how small it is), the analysis does not apply.
- Improved presentation of the paper (the paper went through a major revision, which makes reviewing difficult).
- More explanations/ablation studies why the specific scaling is working.



Recommendation:

First of all, the authors should be stayed assured that there were discussions regarding the paper, and the area chair read carefully i) the paper, ii) the reviews, and iii) the discussions after the rebuttal. We acknowledge that the paper addresses concerns raised, but the reviewers have remaining concerns:

We recommend the authors include a more thorough experimental validation for their methodology: including experiments that are common in various fields should be sufficient to answer any questions about the empirical validation of the method. A cleaner support of the theoretical supports (both with theory and practice) is required. 

We recommend the authors to consider these (common over all reviewers) concerns to revise their paper; we strongly support for a resubmission to a near-future ML conference.
justification_for_why_not_higher_score: Look into Weaknesses + Recommendation in the meta-review
Decision: Reject