Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper presents an empirical study on an optimization anomaly, which is referred to as the Slingshot Mechanism/Effect by the authors, when adaptive optimizers (e.g., Adams) are used for solving supervised learning problems. Specifically, it was found that, when adaptive optimizers were used, cyclic phase transitions between stable and unstable training occurred and were correlated with how the norm of the last-layer weights changed. The authors also reported findings on the correlations between the onset of the Slingshot Mechanism and the glokking phenomenon. 
strengths_and_weaknesses: Strengths:

* The reported findings hint that extra care may be required when adaptive optimizers are used. The results may also promote further theoretical studies on the limitations of using adaptive optimizers in supervised learning.

Weaknesses:

* Additional numerical experiments could have been conducted to help future theorists better understand the source of issues. In particular, some rather simple diagnoses could have been done but were missing from the paper.
* There is much room for improvement in presentation of the paper. See the specific comments in the Questions section.
questions: **Is the reported phenomenon unique to "deep models"?**

I did not see any numerical experiments on "shallow models." From the appendix, I only see experiments with a transformer (12 layers and 10 million parameters), a CNN with VGG-like architecture (which I assume to be a deep model), a deep MLP with 6 layers, and a deep linear model with 6 layers. Without experiments with simple/shallow models, it is premature to draw the conclusion that Slingshot is a general phenomenon as the authors claimed in Lines 154-162. 

As a matter of fact, given that the authors tried to illustrate the phenomenon using simple quadratic functions (Line 174), could the authors conduct a set of numerical experiments using quadratic functions to see whether the phenomenon still persists?

**Some simple numerical diagnoses could have been done.**

As evident from Line 178, one issue with the adaptive optimization schemes considered is that the effective step size $\mu / (|g| + \epsilon)$ can be much larger than the unnormalized step size $\mu$ when the magnitude of the gradient $g$ is small and $\epsilon$ is small. (Here, a small $\epsilon$ is assumed; otherwise, the update rule will not adapt much to the magnitude of $g$.) This implies that, when the magnitude of $g$ is small, i.e., when the current iterate is near a stationary point, the next iterate of the gradient-based update can be very far from the stationary point, potentially leading to a sudden increase in the loss. 

For gradient-based updates, when the objective function is smooth, it is known that a sufficient condition for decrease is to choose the step size to be less than $2/L$, where $L$ is the Lipschitz constant of the gradient (and is related to the Hessian). It would be useful to plot in Figure 3 the effective step size, given by $\mu / (|g| + \epsilon)$, along with the sharpness to see if the step size is still small enough for the sufficient condition for decrease to hold. If the violation of the condition coincides with the increase in the loss, then this would provide a simple explanation of the Slingshot Effect.

**Comments on presentation and typos:**

* Are the Slingshot Mechanism and the Slingshot Effect the same thing? Both terms are used throughout the paper.
* Figure 1 and its first citation are too far apart: Figure 1 appears on the first page (even before the main text), but it citation does not appear until Line 158. 
* A typo in Line 190: "...evidence of **Elingshot** Effects."
* Line 230: What is $3.10e^{-04}4$? Also, it uncommon to write $10^{-04}$ instead of $10^{-4}$. This has occurred several times throughout the paper.
limitations: I do not have any comments on this.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper studies the grokking phenomenon introduced by Power et al [1], which is a curious phenomenon of delayed generalization. This paper describes a characteristic of training termed the "slingshot mechanism", that is found in a number of tasks and optimization settings. The slingshot mechanism refers to periods of instability found in adaptive optimizers in the late stages of training. These periods of instability are accompanied by norm increases in the classification layer, as well as an improvement in generalization performance for the grokking tasks introduced in [1]. The paper studies this phenomenon and finds that it occurs in a number of different tasks more general than those in [1], and find that it is controlled by the epsilon parameter in adaptive optimizers.
strengths_and_weaknesses: ### Strengths

- The paper has a good and comprehensive discussion of related work.
- The paper characterizes an interesting phenomenon that is sometimes found in the optimization trajectory of adaptive optimizers.
- Interesting observations regarding the effect of epsilon on the slingshot behavior.

### Weaknesses

- There is no explanation, empirical or otherwise, for how the slingshot mechanism may relate to generalization. Understanding the delayed generalization is an important aspect of why the grokking phenomenon in [1] is interesting to the community. I believe that in the current state, without a deeper understanding, this work may be more appropriate for a workshop submission.
- The paper claims that the slingshot mechansim is a general phenomenon, however on CIFAR10 the phenomenon was mostly observed when training with extremely small training data (e.g. 200 examples), and no correlation to better generalization was shown. There is one experiment in the Appendix where the full CIFAR10 dataset is used in training a ViT model, but this model only achieves a final test accuracy of <80% and it's not clear how much test accuracy is improved due to the slingshot mechanism due to a single run and increasing test loss. This calls into question how relevant or general the identified behavior is to modern day neural networks.


questions: - In Figure 3(d), why do we not see a corresponding spike in sharpness at the onset of the second plateau? Based on the hypothesis that the weights are flung to a new region in parameter space when the curvature grows too large, I would expect to see the curvature spike at each transition.

- Figure 5, what is the accuracy being plotted? Is it training accuracy or test accuracy? If test accuracy, what is the training accuracy of these models? Do all the runs achieve 100% training accuracy? It is unclear whether correlation with better generalization comes from the appearance of slingshots or from the bad hyperparameter choices required to remove slingshot behavior from the training trajectory. 

- Dataset size is a crucial parameter in determining whether grokking / generalization occurs in [1]. How does the correlation between generalization vs slingshot behavior change when models are trained with differing data size? Does slingshots exist regardless of whether the model generalizes?
limitations: Beyond the weaknesses I listed, the authors were good at addressing several limitations of this work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors describe a "slingshot" effect present in the use of Adam (and cousins). This purportedly cyclic effect can be observed when the norm of the final layer sharply increases, often accompanied by an increase in train loss. The authors offer the slingshot effect as a possible explanation for the ability of Adam to induce generalization even without explicit regularization.
strengths_and_weaknesses: This is overall a nicely executed empirical paper that reveals a moderately interesting phenomenon. A cyclic regime of generalization would be quite remarkable and this paper offers preliminary evidence that supports the authors' hypothesis. It is also refreshing that the authors did not attempt to handwave nonsense poorly-motivated math into the paper, which is unfortunately a severe problem even in 2022.

My main concerns are the following:

- Significance: Given that this effect explains generalization in unregularized optimization, the overall impact to practical deep learning is somewhat muted.
- Strength of evidence: the results in the paper leave me with certain questions (most notably #6 below) as to whether the authors have sufficiently validated their hypothesized effect.
- Implications: the paper enumerates a number of interesting facts (e.g. dependence on $\epsilon$) but doesn't really dive deeper beyond some of those facts, leaving the reader wondering as to why those facts were presented.

Overall, I enjoyed reading this paper. I don't think it's super noteworthy in terms of immediate impact, but it is (as far as I can tell) a novel discovery that may serve as a waypoint on our path to better understanding of oddball generalization phenomena in deep learning.
questions: 1. Is there a reason validation error/loss is not depicted in Fig 1? Given the emphasis in this paper on generalization, it seems odd not to depict in the headline figure.
2. Figure 2(c) looks either incorrect or misgenerated—the validation loss should not be zero at the beginning of training, unless I am missing something.
3. Do the authors have any explanation for why Figure 3(d) doesn't show a high-curvature update on the second plateau?
4. Can the authors elaborate more on the implications of the larger $\epsilon$ result? To me, larger $\epsilon$ ruining the grokking effect sort of implies that non-adaptive methods (e.g. plain GD) don't benefit from slingshotting either (which is exactly what the authors found empirically). But it would still be good to explicitly connect the dots here.
5. I am also left wondering about the result on the paragraph starting on line 243. It's cool that the authors discovered it but what is it supposed to tell me? What about low-dim data makes the slingshot effect less likely to occur?
6. Finally, is it fair to call this effect cyclic when most empirical examples only have two cycles (with a few having 3)? Is there a setup where we can see $n$ distinct plateaus and sharp norm increases for some $n >> 3$?
limitations: The authors do a good job not overclaiming, and they explicitly enumerate a number of other regimes that achieve the results despite lack of "slingshot". Certain items are left unaddressed (see "Questions").

Minor
----

- Line 84: missing author name
- Line 112: what is "Adam without momentum" and how is it different from RMSProp?
- Line 145: "layers" $\rightarrow$ "layer"
- Line 170: "has" $\rightarrow$ "have"
- Line 245: "choices" $\rightarrow$ "choice"
- Line 254: "it's" $\rightarrow$ "its"
review_title: null

Meta-review: metareview: The paper examines a widely known phenomenon when training neural networks with adaptive optimizers, where the training loss cyclically alternates in later stages. Some evidence is given that, in the absence of explicit regularization, this is associated with improved generalization. The more concrete contribution of the paper is to show a strong link between the aforementioned cyclicity and sudden cyclic growth in the weight of the last layer of the network. The main concern is that beyond this empirical observation a specific mechanism behind the phenomenon is not identified, nor a clear connection is made with the apparent benefit to generalization. While the observations have merit, it is hard to determine whether cause-effect relationships exist between them, making the paper feel as a work-in-progress and only weakly significant to the community (judging by the reviewers being underwhelmed). If the authors are convinced of their message that "slingshots" cause "grokking" (in contrast to, e.g., being a by-product while the real mechanism lies elsewhere), then they are advised to show exactly that in the further elaboration of their work.
confidence: Less certain
Decision: Reject