Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors claim that in SGD-trained neural networks, during the phases where the loss is constant, the stochastic noise in the optimization serves to allow the model to learn sparse features, which endows it with a benevolent inductive bias in terms of generalization. The authors demonstrate their ideas in experiments with small to medium sized neural networks.
strength_and_weaknesses: The authors' argumentation seems to include the following steps:

1- SGD can be formulated as GD with a specific random noise (Prop 1.).
2- Variance of the norm of the specific random noise they introduced is proportional to the batch loss (Lemma 2.).
3- It is possible for an SGD trained algorithm to maintain an equilibrium where the loss is lower and upper bounded all the while the parameters take different values in the basin of the minimum but at some distance away from it (Prop 3.).
4- In the phase of the learning where the loss is constant, SGD can be modelled by a SDE whose noise term is characterized by the stochastic gradients (Eq 7).
5- This SDE implies that when the loss is constant, the optimization procedure will be inclined to reduce the l_2 norm of the stochastic gradients, leading to a sparse representation.

I find the authors' work valuable in investigating how the training dynamics and sparse representations evolve during training of neural networks. How efficient representations are achieved throughout the training procedure is a topic that is promising for understanding how neural networks generalize. The authors have a more or less self-consistent argumentation (albeit with the caveats below) and provide interesting experimental results to complement their claims.

Although I believe that the conference audience would benefit from their work, I believe answering the questions below would improve my and possibly other conference audience's understanding of their work:

- The authors' comparison of their approach to the rest of the literature is cursory at best. Although they mention a lot of other works, they usually shy away from making comparisons that illuminate why their results differed from those previously published. I would appreciate even less citations in exchange for meaningful comparison. For example, (Pg. 2 Par 5) What makes authors experimental regime different than that of Lewkowycz et al. 2020.? What does flatness definitions being "questionable" mean (Pg. 2)? Why are their findings regarding batch GD is different than those of Nacson et al 2022? (Pg 7.) In which aspects are authors' demonstrations are "closer to practice" (Pg. 1) and compared to whom?
- Pg. 4 Par. 4: How would the authors argue that Proposition 3 supports the importance of large learning rates per se in leading to the loss stabiliziation? Moreover, how does Prop. 1 serves the authors' exposition exactly? What in framing SGD as a GD + noise enable Claim 4 to be made, that would not be possible if it was not framed as GD?
- What is the justification for the choice of Brownian motion in the authors' proposed SDE? Large learning rates are assumed to produce heavy-tailed behavior in neural network parameters, would these results change the interpretation of the authors' results?
- Throughout the introduction and exposition the authors refer to label noise being a key in understanding the generalization behavior of SGD. I think this might be confusing for some audience since the type of label noise the authors are proposing is simply a theoretical device for their derivation, as opposed to researchers working with uncertainty in supervision signal.
clarity,_quality,_novelty_and_reproducibility: Besides the caveats mentioned above, the authors argumentation are not hard to follow, and their findings are of sufficient novelty and interest. The authors seem to prevent enought details with respect to their experimentation to reproduce their results.

Below are some more minor points for the authors to either help with their exposition or writing.

- Pg. 1 end of Par. 1: sentence unclear.
- Pg. 1 Par.3: Which noise are the authors addressing?
- Pg. 2 Par. 1: In what way "noise covariance [is] being rarely well understood"?
- Pg. 2 Par. 2: "Loss stabilization" is an important concept for this paper but is not explicitly defined early on.
- Pg. 2 Par. 2: Definition of fast-slow dynamics, at least a reference to the larger literature would be helfpul.
- Pg. 2 Par. 3: What does saddle-to-saddle vs. side-to-side mean?
- Pg. 2 Par. 3: leads to learn -> leads the model to learn
- Pg. 2 Par. 4: Insights from our research?
- Pg. 2 Par. 4: "Fitting phase" is not introduced. Neither is "exploration" is properly defined.
- Pg. 2 Par. 6: Why are such definitions "questionable"?
- Pg. 3 Par. 2: The authors intended meaning "''hidden' dynamics" should be explained as early as possible.
- Pg. 3 Par. 2: Which parts of the entire training dynamics is not captured by these analyses?
- Pg. 3 Par. 3: What's the "central phase" of a training.
- Pg. 3 Par. 4: Can the authors qualify / demonstrate which subset of their analyses directly translate to other types of losses?
- Pg. 3 Par. 4: train loss -> training loss.
- Pg. 3 Par. 5: "We note that..." can the authors support this assertion?
- Pg. 3: Instead of "specific label noise", I'd recommend the authors name the particular noise they are proposing and use that term.
- Pg. 3 Proposition 1: Let ... follows -> Let ... follow (also in the appendix)
- Pg. 3 Proposition 1: Please fix the numberings of the propositions etc. in the appendix.
- Pg. 4 Par 2: Please provide a reference for the convergence results mentioned.
- Pg. 5 Par 1: Sparse-inducing -> sparsity inducing.
- Pg. 5 Par 2: ... example of simple non-linear networks ... are diagonal linear networks?
- Pg. 6 Par 3: How do the authors interpret the warm up to serve implicit regularization - their exposition before does not seem to refer to any such function of learning rate increase during early training.
- Pg. 8 Par 1: Why do the authors use the teacher-student setting for illustrating their method.
summary_of_the_review: I think the authors' work provides some interesting findings that would be helpful for a more nuanced understanding of the generalization behavior of deep neural networks. At points, more justification for their modeling choices and/or their inferences seem to be needed, as well as more concrete comparions with the existing body of work.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper provides some insights on the potential reason that the initial large step sizes can learn the model that can generalize well in practice. Center to the claim is an SDE with specific noise covariance structure, and the authors argue that, if we use the large step sizes at the beginning, the loss should stabilize at a certain level without further decreasing. During this phase, the unnecessary feature will be optimized towards 0, which will eventually lead to a solution with sparse features.

strength_and_weaknesses: ### Strength:
* The idea is novel and interesting.
* The authors provide some synthetic examples to interpret the intuition.
* The authors conduct lots of experiments to demonstrate the conjecture.

### Weakness:
* It’s a little bit hard to follow as the authors don’t provide clear logic in the presentation of Section 2.
* The examples are a little bit toy.

clarity,_quality,_novelty_and_reproducibility: The paper is overall clearly-written but needs to be polished to improve the readability. The idea is quite novel. The authors don’t provide the code and hence it’s not clear if all of the results are reproducible.

summary_of_the_review: The authors provide a quite good intuition to interpret the effectiveness of large initial step sizes. Although the example is a little bit toy, I believe this paper will make impacts in the related areas. I feel the authors should be happy to include some discussions for [1]. Although it appears after the ICLR submission deadline, I feel this paper provides some theoretical analysis that potentially demonstrates the intuition discussed in this paper.

[1] Mousavi-Hosseini, Alireza, Sejun Park, Manuela Girotti, Ioannis Mitliagkas, and Murat A. Erdogdu. "Neural Networks Efficiently Learn Low-Dimensional Representations with SGD." arXiv preprint arXiv:2209.14863 (2022).
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studies the implicit bias of SGD with large step sizes. The authors present empirical observations and some theoretical understanding. They show that large step sizes SGD enables exploration by loss stabilization, and this exploration leads SGD to learn sparse features. The authors also provide some insights into the deep neural networks' training dynamics and understanding of some existing tricks.
strength_and_weaknesses: The paper is generally well-written. Understanding the implicit bias of SGD with large step sizes is a significant theoretical and practical issue. This paper provides a rich discussion and offers a sparse feature learning perspective.  However, I feel that neither the analysis is well-designed nor the claim is well-supported. Specifically, I have two primary concerns from the main text of this work: the persuasiveness of theories in Section 2 and the sparse feature bias of deep networks in Section 3.
These two main points also limit the significance or rather limit my ability to determine this work's significance. 


- The persuasiveness of theories in Section 2.
    - **The reasonableness of the SDE in (Eqn. 7).**  SDE is good modeling of SGD only in small step size regime, but it is generally unclear how this SDE modeling is relevant for understanding SGD with large step sizes.  As the authors mentioned in Section 2.3.1, this SDE may be reasonable for the diagonal linear model. However, for nonlinear networks such as ReLU nets, the reasonableness of the SDE modeling (eqn 7) seems questionable (especially the noise covariance). This SDE is used to motivate the bias of learning sparse features but is never used in experiments, where vanilla SGD is applied. It is thus unclear why do not use vanilla SGD to motivate the sparse feature learning.   The authors can address this issue using experiments. 
    
    - **The hidden stochastic dynamics orthogonal to the bouncing directions.** In the abstract, the authors mention ''this loss stabilization induces a hidden stochastic dynamics orthogonal to the bouncing directions''. However, it seems that this paper does not point out the bouncing direction as well as explain the reason for orthogonality. The 1-D example (Eqn. 4) gives us insights into loss stabilization, but it cannot help us understand the authors' claim about the bouncing direction and the orthogonal hidden stochastic dynamics. The authors can construct a 3-D toy example to illustrate the loss stabilization, the bouncing direction, and the orthogonal effective dynamics.
    
    - **The claim that large step sizes learn sparse feature solutions.**
    I do not follow the explanation in Section 2.3.2 of why the eqn (7) implies that SGD implicitly minimizes the $\ell_2$ norm of each column of the feature matrix. This claim is so big and concrete but I do not see too much support for this claim. In addition, all of the authors' analysis holds for any step size in the interval of step sizes that satisfy loss stabilization. It means that SGD with different step sizes in this interval has the same sparse-feature-learning bias. Hence, it seems that the authors' viewpoint can not distinguish the different implicit biases of two sizes in this interval. If this is the case I do not view it as grounds for rejection but it is important for a reader to know. 
 
    
- The sparse feature bias of deep networks in Section 3.
    
    - **The clarification of the feature sparsity coefficient.** What is the feature sparsity coefficient for deep neural networks mentioned on Page 6? As an essential metric, there is neither a clear explanation nor a mathematical formulation of this in the main text, which is not very clear.
    

clarity,_quality,_novelty_and_reproducibility: This is generally well-written work on analyzing the implicit bias of SGD
summary_of_the_review: This paper contains very interesting observations and analyses. However, I feel that the results are overclaimed and analysis support is not enough. Consequently,  I am a bit skeptical about the applicability of the findings.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper empirically shows that SGD with large step size learns sparse features and presented a few minimal theoretical examples that corroborate the empirical observation

strength_and_weaknesses: Strength: some pieces of insights are novel. For example, the authors suggest the mechanism of loss stabilization and suggest its connection to sparsity.

the point that GD and SGD have different inductive biases is also an important insight

The following are the weaknesses that the authors need to address.
1. the title is too strong: the title implies that using a large learning rate always leads to sparsity, but this is not what the paper has shown. The paper only showed that *empirically, a large learning rate leads to sparsity in certain restricted scenarios*. The title is a overclaim that needs to be changed

2. the first contribution has already been studied extensively in quite a few previous works. the authors need to discuss these works and modify the claim accordingly: (1) https://openreview.net/forum?id=uorVGbWV5sw; (2) https://proceedings.mlr.press/v162/mori22a.html

3. contribution 3: it is not exploration that leads to sparse features; it is the multiplicative noise. After all, "exploration" here is a very ambiguous word here and needs revision

4. page3: "we showcase our results for the mean square error, but other losses like the cross-entropy carry the same properties (W2021b)." I don't see which theorem in W2021b shows that cross-entropy carries the same properties. The authors should be more specific about which theorem in W2021b and which "property" they are referring to

5. prop 1: the notation $\mathbf{1}_{i=i_t}$ is undefined

6. prop 1 feels incorrect (it is, at least, unclear). The theorem statement seems to suggest that the SGD on $L(y)$ is equivalent to GD on $L(y + \xi)$, but this is not the case: the second equation of the proof does not compute the gradient with respect to $\xi_i$, which is also a function of $\theta$. The proof thus feels incorrect
- Thus, the message that SGD = GD + label noise is correct is also a misstatement.
- The authors needs to either correct the proof or update the proposition statement

7. the word "stabilize" is used many times but I feel is inappropriate. Look at figure 1: at a large learning rate, the loss does not stabilize to a fixed value but fluctuates drastically around a mean. I think the word "stabilize" is confusing and should be changed

8. the fonts in Figures 3, 4, 6, 7 are not visible to human eyes. The authors need to use a font size similar to the font size of the main text in the figures

9. the word "feature sparsity" is undefined throughout in the paper even though it plays an important role. this leads to a lot of potential confusion. For example, see the third figure in figure 7; the figure tells me that a "larger learning rate leads to smaller feature sparsity," but I guess this is not what the authors are trying to convey

10. I cannot agree with a main (empirical) claim of the paper: "longer stabilization leads to sparser features." The empirical results are only sufficient to show that longer stabilization is positively correlated with sparser features (because they happen simultaneously), but I do not think the empirical results allow one to conclude that one leads to the other. I stress the following point: *correlation cannot imply causality*

 


Minor:
1. perhaps the authors should point out that eq 2 is essentially SGD with batch size 1 
2. the authors should give equation numbers to all the equations in the appendix so they are easier to refer to during the review
clarity,_quality,_novelty_and_reproducibility: The novelty feels fine, even though some main contributions have been discovered here and there in the previous works

The main problem is clarity, and the overclaims I point out in the previous section
summary_of_the_review: The paper has great promise but is hindered by the disclarity, missing of discussion of important previous works, overclaim, and potential incorrectness of the theory

I cannot recommend the paper for acceptance in its current state. I might reconsider my position if the authors make extensive updates to improve the manuscript; otherwise, I suggest resubmitting to a different conference
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper examines SGD and argues empirically and theoretically through a connection to an SDE that large steps sizes can induce sparse features.

The reviewers generally thought that the connection to the SDE was a bit tenuous, and even with the addition of further validating experiments there were still split opinions about whether the theoretical contributions were nevertheless useful: some reviewers thought they provided a nice plausibility argument, while others viewed them as a stretch and a bit of a distraction. Ultimately, the theoretical results are not in themselves strong enough to form the backbone of the paper.

The empirical analysis was the subject of extended discussion. A main concern was that the experimentation was narrow, and it was not necessarily demonstrated that the mechanism behind the effect was in fact the same across the different settings. All together, there were some doubts about the main conclusions. I encourage the authors to develop more convincing evidence before resubmitting to another venue.
justification_for_why_not_higher_score: The main claims in the paper may not have been demonstrated in a sufficiently convincing manner. In particular, following the discussion phase, there are still lingering doubts regarding whether a causal link has been established between large learning rates and the post-training sparsity.
justification_for_why_not_lower_score: N/A.
Decision: Reject