The Slingshot Mechanism: An Empirical Study of Adaptive Optimizers and the \emph{Grokking Phenomenon}
Authors: Vimal Thilak
Etai Littwin
Shuangfei Zhai
Omid Saremi
Roni Paiss
Joshua M. Susskind
Venue: NeurIPS 2022 Submitted (2022)
Keywords: optimization
adam
adamw
adaptive optimizers
grokking
training instability
instability
empirical
empirical science of deep learning
double descent

Abstract: The \emph{grokking phenomenon} as reported by Power et al.~\cite{power2021grokking} refers to a regime where a long period of overfitting is followed by a seemingly sudden transition to perfect generalization. In this paper, we attempt to reveal the underpinnings of Grokking via a series of empirical studies. Specifically, we uncover an optimization anomaly plaguing adaptive optimizers at extremely late stages of training, referred to as the \emph{Slingshot Mechanism}. A prominent artifact of the Slingshot Mechanism can be measured by the cyclic phase transitions between stable and unstable training regimes, and can be easily monitored by the cyclic behavior of the norm of the last layers weights. We empirically observe that without explicit regularization, Grokking as reported in \cite{power2021grokking} almost exclusively happens at the onset of \emph{Slingshots}, and is absent without it. 
    While common and easily reproduced in more general settings, the Slingshot Mechanism does not follow from any known optimization theories that we are aware of, and can be easily overlooked without an in depth examination. Our work points to a surprising and useful inductive bias of adaptive gradient optimizers at late stages of training, calling for a revised theoretical analysis of their origin.

The Slingshot Mechanism: An Empirical Study of Adaptive Optimizers and the Grokking Phenomenon
Anonymous Author(s)
Affiliation
Address email
Abstract
The grokking phenomenon as reported by Power al. [3 refers regime where long period of overfitting is followed by seemingly sudden transition to perfect generalization. this paper, we attempt to reveal the underpinnings of Grokking via series of empirical studies. Specifically, we uncover an optimization anomaly plaguing adaptive optimizers extremely late stages of training, referred to as the Slingshot Mechanism. A prominent artifact of the Slingshot Mechanism measured by the cyclic phase transitions between stable and unstable training regimes, and can be easily monitored by the cyclic behavior of the norm of the last layers weights. We empirically observe that without explicit regularization, Grokking reported in [13] almost exclusively happens at the onset of Slingshots, and is absent without it. While common and easily reproduced in more general settings, the Slingshot Mechanism does not follow from any known optimization theories that we are aware of, and can be easily overlooked without an in depth examination. Our work points surprising and useful inductive bias of adaptive gradient optimizers at late stages training, calling for a revised theoretical analysis of their origin.
11
13
15
16
train loss
last layer norm
Slingshot
10a
10-2
0.0780
1055
10
Fast layer
0.0975
10-6
0.0770
10-4
0.0765
1000
1400
1800
2000
opoch
Figure 1: Slingshot Effects are observed with fully-comnected ReLU network (FCN). The FCN is trained with 200 randomly chosen C1 samples with Adam. Multiple Slingshot Fffects occur in cyclic fashion as indicated by the dotted red boxes. Fach Slingshot Effect is characterized by period of rapid growth of the last layer weights, an ensuing training loss spike, and norm plateau.
Introduction
17
18 Recently, the grokking phenomenon was proposed by [3] the context of studying the optimization and generalization aspects in small, algorithmically generated datasets. Specifically, grokking refers
Submitted to 36th Conference on Neural Information Processing Systems (NeurIPS 2022). Do not distribute.

---

20 to sudden transition from chance level validation accuracy to perfect generalization, long past the point of perfect training accuracy, i.e., Terminal Phase of Training (TPT). This curious behavior 21
22 contradicts the common belief of early stopping in the overfiting regimes, and calls for further understandings of the generalization behavior of deep neural networks.
23
24 In the literature, it has been suggested that in some scenarios, marginal improvements in validation
25 accuracy appears in TPT, which seem to directly support grokking. For example, has been shown
26 in 14] that gradient descent on logistic regression problems converges to the maximum margin
27 solution, a result that has been since extended cover a wider setting 11 17.. key finding in [14]
28 shows that when training on linearly separable data with gradient descent using logistic regression,
29 the classifier's 's margin slowly improves at rate of O logg (1ogr) while the weight norm of the predictor
30 layer grows at rate of (t), where t is the number of training steps. While specified for gradient descent, Wang et al. [17] showed that similar results also hold for adaptive optimizers (such Adam
31
32 and RMSProp). Taking these results into consideration, one could reasonably hypothesise that deep nonlinear networks could benefit from longer training time, even after achieving zero errors on the
33
34
training set.
35
36 In this paper, we provide in depth empirical amalyses to the mechanism behind grokking. We find that the phenomenology of grokking differs from those predicted by [4] in several key aspects. To be concrete, we find that grokking occurs during the onset of another intriguing phenomenon directly related to adaptive gradient methods (see Algorithm T for generic description of adaptive gradient methods). In particular, leveraging the basic setup in 13 we make the following observations:
38
40
41
1. During the TPT, training exhibits cyclic behaviour between stable and unstable regimes. A prominent artifact of this behaviour can be seen the norm model's last layer weights, which exhibits a cyclical behavior with distinct, sharp phase transitions that alternate between rapid growth and platcaus over the course of training.
42
43
44
2. The norm grows rapidly sometime after the model has perfect classification accuracy on training 45
46 data. A sharp phase transition then occurs when the model missclassifies training samples. This
47 phase change is accompanied by sudden spike in training loss, and a plateau in the norm growth of
48 the final classification layer.
49 3. The features (pre-classificatiio layer) show rapid evolution as the weight norm transitions from
50 rapid growth to growth plateau, and change relatively little at the norm growth phase.
51 Phase transitions between norm growth and norm plateau phases are typically accompanied by a sudden bump in generalization as measured by classification accuracy validation set, as observed [13].
53 in dramatic fashion
54 5. It is empirically observed that grokking as reported in 3 almost exclusively happens at the onset of Slingshots, and is absent without it.
55
56 We denote the observations above as the Slingshot Effect, which is defined to be the full cycle starting from the norm growth phase, and ending in the norm plateau phase. And empirically, single training run typically exhibits multiple Slingshot Effects. Moreover, while grokking as described in [13 might be data dependent, reproduced in multiple scenarios, encompassing variety of models (Transformers and MLPs) and datasets (both vision. algorithmic and synthetic datasets). Since we only observe Slingshot Effects when training classification models with adaptive optimizers, our work can be seen as empirically characterizing an implicit bias of such optimizers, Finally, while our observations and conclusions hold for most variants of adaptive gradient methods, we focus on Adam in the main paper, and relegate all experiments with additional optimizers to the appendix.
57
58
59
find that the Slingshot Mechanism is pervasive, and can be easily
80
81
82
63
64
65
Algorithm Generic Adaptive Gradient Method
Input: Xi E F, step size μl, sequence of functions (Φ+, 3/x )Z R Output: Fitted a.
for 1.... d'd. do gi Vf((zt).
m. 0c .... 9t) and ve(g),...,
Z141 Zt amm
2

---

66
1.1
Implications of Our Findings
67
The findings in this paper have both theoretical and practical implications that go beyond characteriz- ing Grokking. A prominent feature of the Slingshot Mechanism is the repeating phase shifts between stable and unstable training regimes. where the unstable phase is characterized by extremely large gradients, and spiking training loss. Furthermore, we find that learning at late stages of training have
68
69
70
71 a cyclic property, where non trivial feature adaptation only takes place at the onset of a phase shift. From a theoretical perspective, this is contradictory to common assumptions made in the literature of convergence of adaptive optimizers, which typically require I, smooth cost functions, and bounded stochastic gradients, either in the L22 or Los norm, decreasing step sizes and stable convergence
72
73
74
75
From the apparent generalization benefits of Slingshot Effects, we cast doubt on the ability of current working theories to explain the Slingshot Mechanism.
[
76
77
Practically, our work presents additional evidence for the growing body of work indicating the
78
importance of the TPT stage of training for optimal performance [6
79 In an era where the sheer size of models are quickly becoming out of reach for most practitioners,
80 our work suggest focusing on improved methods to prevent excessive norm growth either implicitly
81 through Slingshot Effects or through other forms of explicit regularization or normalization.
2
Related Work
83
The Slingshot Mechanism we uncover here is reminiscent of the catapult mechanism described in Lewkowycz al. (9]. Lewkowycz al. (9] show that loss of a model trained via gradient descent with an appropriately large learning rate shows a non-monotonic behavior -the loss initially increases and starts decreasing once the model "catapults" to region of lower curvature early in training. However, the catapult phenomenon differs from Slingshot Effects in several key aspects. The catapult mechanism is observed with vanilla or stochastic gradient descent unlike the Slingshot Mechanism that is seen with adaptive optimizers including Adam (7) and RMSProp [IS). Furthermore, the catapuli phenomenon relates to a large initial learning rate, and does not exhibit a repeating cyclic behavior. More intriguingly, Slingshot Effects only emerge late in training, typically long after the model reaches perfect accuracy on the training data.
84
85
86
87
86
89
90
91
92
93
Cohen et al. 3 describe a "progressive sharpening" phenomenon in which the maximum eigenvalue of the loss Hessian increases and reaches value that is equal to or slightly larger than 2/n where is the learning rate. This "progressive sharpening" phenomenon leads to model to enter a regime Cohen et al. (3 call Edge of Stability where-in the model shows non-monotonic training loss behavior over short time spans. Edge of Stability is similar to the Slingshot Mechanism in that it is shown to
94
95
96
97
96
occur later on in training. However, Edge of Stability is shown for full-batch gradient descent while
99
we observe Slingshot Mechanism with adaptive optimizers, primarily Adam 7 or Adam [10]
As noted above, the Slingshot Mechanism emerges late in training, typically longer after the model reaches perfect accuracy and has low loss on training data. The benefits of continuing to training model in this regime has been theoretically studied in several works including [14 [T]. Soudry 102
100
101
103
et al. [4] show that training linear model on separable data with gradient using the logistic
104
[4] prove that the loss decreases at rate of O()) while the margin increases much slower O( 1 ((ing where i is the number
loss function leads to max- max-margin solution. Furthermore Soudry et al.
105
106
of training steps. Soudry et al. [14 also note that the weight norm of the predictor layer increases at a logarithmic rate, i.e., (log(t)). Lyu and [ generalize the above results to homogeneous neural networks trained with exponential-type loss function and show that loss decreases at rate of This is, where L is defined as the order of the homogenous neural network. Although these results indeed prove the benefits of training models, their analyses are limited
107
108
0 (
109
110
111
to gradient descent. Moreover, the analyses developed by Soudry et al [14] do not predict any
112
phenomenon that resembles the Slingshot Mechanism. Wang et al. [17 show that homogenous neural networks trained with RMSProp [5] or Adam without momentum [17] do converge in direction to the max- mar-inn solution. However, none of these papers can explain the Slingshot Mechanism and
113
114
specifically the cyclical behavior of the norm of the last layer weights.
3

---

+M.031011 RyNTram
landd Iorlux
Vit
10
107
37
(a)
100
0.000
0.003
0:000
Hap
(b)
(d)
Figure 2: Division dataset: Last layer weight norm growth versus loss on training data b) accuracy training data (c) loss on validation data d) accuracy on validation data e) normalized relative change in features of first Transformer layer (f) normalized relative change in features of second Transformer layer. Note that the feature change plots are shown starting at 10K step to emphasize the feature change behavior during norm growth and plateau phases, revealing that the features stop changing during the norm growth phase and resume changing during the plateaus.
3
The Slingshot Mechanism
116
117
3.1
Experimental Setup
118
We use the training setup studied by Power al. 1K in the main paper as working example to illustrate the Slingshot Mechanism. In this setup, we train decoder-only Transformers [16] on modular division dataset [13] of the form + b c, where a, b and e are discrete symbols and + refers to division modulo for some prime number split into training and validation sets. The task consists of calculating given a and b. The algorithmic operations and details of the datasets considered in our experiments are described in Appendix B. The Transformer consists of layers, of width 128 and 4 attention heads with approximately 450K trainable parameters and optimized
119
120
122
124
125
by Adam 7 [0]. For these experiments we set learning rate to 0.001, weight decay to 0, 8 = 0.9, 10-$, linear learning rate warmup for the first 10 steps and minibatch size to 512 which are in line with the hyperparameters considered in [13
126
32 0.98,
Figure 2 shows the metrics of interest that we record on training and validation samples for modular
128
129
division dataset. Specifically, we measure 1) train loss; 2) train accuracy: 3) validation loss; 4) validation accuracy; last layer norm: denoting the norm of the classification layer's weights and 6)
130
feature change: the relative change of features of the I-th layer (h') after the t-th gradient update step
We observe from Figure d that the model is able to reach high training accuracy around
132
[AS] ||
step 300 while validation accuracy starts improving after 10 steps as seen in Figure 2.. Power et as grokking. We observe that while the validation accuracy does not exhibit any change until much later in training, the validation loss shown
139
134
al. [3] originally showed this phenomenon and refer
135
Figure : exhibits a double descent behavior with an initial decrease, then growth before rapidly decreasing to zero.
136
137
138
Seemingly, some of these observations can be explained by the arguments in 14 and their extensions
139
to adaptive optimizers [17] Namely, at the point of reaching perfect classification of the training set, the cross-entropy (CE) loss by design pressures the classification layer to grow in norm relatively fast rate. Simultaneously, the implicit bias of the optimizer coupled with the CE loss, pushes the direction of the classification layer to coincide with that of the maximum margin classifier, albeit at a
140
142
much slower rate.

---

144
These insights motivate us to measure the classifier's last layer norm during training. We observe in Figure Za that once classification reaches perfect accuracy on the training set, the classification layer norm exhibits a distinct cyclic behavior, alternating between rapid growth and plateau, with a sharp phase transition between phases. Simultaneously, the training loss retains a low value in periods of rapid norm growth, and then wildly fluctuating in periods of norm plateau. Figure and Figure 2 shows the evolution of the relative change in features output by each layer in the Transformer. We observe that the feature maps are not updated much during the norm growth phase. However, the phase transition, we observe that the feature maps receive a rapid update, which suggests that the internal representation of the model is updating.
145
147
148
149
151
153
Is Slingshot a general phenomenon? In an attempt to ascertain the generality of Slingshot Effects as an optimization artifact, we run similar experiments with additional architectures, datasets, optimizers, and hyperparameters. We use all algorithmic datasets as proposed in [3], as well as frequently used vision benchmarks such as CIFAR-10 [ and even synthetic Gaussian dataset. For architectures, use Transformers, MLPs and deep linear models (see figure D. We find abundant evidence Slingshot Effects all of our experiments with Adam, Adam W and RMSProp, We are unable to observe Slingshot Effects with Adagrad [5] and also with stochastic gradient descent (SGD) or SGD with momentum, pointing to the generality of the mechanism across architectures and datasets. We refer the reader to Appendix A for the full, detailed description of the experiments.
154
155
156
158
160
162
Why does Slingshot happen? We hypothesize that the norm growth continues until the curvature of the loss surface becomes large, effectively "flinging" the weights to different region in parameter space as small gradient directions get amplified, reminiscent of the mechanics of a slingshot flinging projectile. We attempt quantify how far a model is flung by measuring the cosine distance between checkpoint during optimization and initial parameters. Specifically, we divide the model parameters into representation (pre-classifier) parameters and classifier (last layer) parameters and calculate how far these parameters have moved from initialization. We show that checkpoints collected after model experiences Slingshot have larger representation cosine distance. We defer the reader to the appendix for further details.
163
184
185
166
167
168
169
1.75
40
40
1.75
1.50
1.25
1.00
1.00
20
100
0.30
10
0.50
0.25
2.00
6000
2,00
60/0
0.00
spdiate sharpnees
layer
up op-iete shurpnsss
layor norm
50
2.00
1.75
200
1.5
1.25
Sycr
150
30
20
50-
0.25
0.00
2009
4000
60000
10000
2000
40000
80000
13000
opoch
opoch
(b)
(d)
Figure 3: Curvature metric (denoted "update sharpness") evolution vs norm growth on (a) addition, (b) subtraction, (c) multiplication. and (d) division dataset. Note spike in the sharpness metric near the phase transitions between norm growth and plateau.
171 By design, adaptive optimizers adapt the learning rate on a per parameter basis. In toy, convex scenarios, the parameter provably determines whether the algorithm will converge stably. To
5

---

173
illustrate this, we take inspiration from [3, and consider a quadratic cost function L(A, 1e Ax B x + C, A E Rdxd xx R E R2, C E RR where we assume A is symmetric and positive definite. Note that the global minimum of this cost is given by x² = -A- The gradient of this cost with respect to x is given by g Ax B. Consider optimizing the cost with adaptive Az,+ optimization steps of the simple form 1++1 3t 1 to I2 / +1 where y is learning
174
175
178
177
178
rate, and the division and absolute operations are taken element wise. Starting from some zo, the error E = Zt- x² evolves according to:
179
Et+1 = (udiag [Aez + )A)e def
= Mee
(I)
Note that the condition ||A||s < 2c where || Il. denotes the spectral norm, implies that the mapping Mr is a contraction for all values of t, and hence convergence to the global optimum is guaranteed (This is in contrast to gradient descent, where the requirement is || . < 3)) Note that the choice
180
181
182
183
of crucially controls the requirement on the curvature of the cost, represented by the the spectrum of A in this case. In other words, the smaller e, the more restrictive the requirements on the top eigenvalue of A. In (3, it was observed that full batch gradient descent increases the spectral norm of the Hessian to its maximum allowed value. We therefore hypothesize that for deep networks, a small value for e requires convergence a low curvature local minimum, causing a Slingshot Effect when this does not occur. Moreover, we may reasonably predict that increasing the value of e would lift the restriction on the curvature, and with it evidence of Slingshot Effects.
184
185
186
197
189
190
Figure 3 shows evidence consistent with the hypothesis that Slingshot Effects occur in the vicinity of high loss curvature, by measuring the local loss surface curvature along the optimization trajectory. Let Hi denote the local Hessian matrix of the loss, and de the parameter update at time t given the optimization algorithm of choice. We use the local curvature along the trajectory of the optimizer, given by || ", Heut, as a curvature measure. Across the arithmetic datasets from [3] whenever the last layer weight norm plateaus, the curvature measure momentarily peaks and settles back down.
191
192
193
194
195
196
Varying c We next observe from Figure that the training loss value also spikes up around the time step when the weight norm transitions trom growth to plateau. A low training loss value suggests that the gradients (and their moments) used as inputs to the optimizer are small, which in turn can cause the hyperparameter value play role in calculating updates. Our hypothesis here is that the Slingshot Effect should eventually disappear with a sufficiently large To confirm this hypothesis, run experiment where vary e while retaining the rest of the setup described in the previous section.
198
199
200
202
Figure 4 shows the results for various values of considered in this experiment. We first observe that the number of Slingshot Effect cycles is higher for smaller values of e. Secondly, smaller values of e cause grokking to appear at earlier time step when compared to larger values. More intriguingly, models that show signs of grokking also experience Slingshot Effects while models that do not experience Slingshot Effects do not show any signs of grokking. Lastly, the model trained with the largest = 10-5 shows no sign of generalization even after receiving 500K updates.
204
206
208
209
3.2
Effects on Generalization
In order understand the relationship between Slingshot Effects and neural networks generalization, we experiment with various models and datasets. We observe that models that exhibit Slingshot tend generalize better, which suggests the benefit of training models for a long time with Adam 77 and AdamW [00. More surprisingly, we observe that Slingshots and grokking tend to come in tandem.
211
212
213
214
Transformers with algorithmic datasets We follow the setting in Power et al. and generate several datasets that represent algorithmic operations and consider several training and validation splits. This dataset creation approach is consistent with the methodology used to demonstrate grokking [13.. The Transformer is trained with Adam W [10] with a learning rate of 0.001, weight
215
216
217
218
decay set to O, and with learning rate warmup for 500K steps. We consider e of Adam as a hyperparameter in this experiment. Figure summmrriee the results for this experiment where the x-axis indicates the algorithmic operation followed by the training data split size. As can be seen in Figure 5 Slingshot Effects are seen with lower values of and disappear with higher values of
220

---

10
10
10
6.26
10"
16-1
2003
X048
(a)
(b)
training loss VS epochs
oarten
1033
Ieit Myye nom
2.10
1.0
17
39-4
Fax127
38
5x127
(d)
validation loss vs epochs
S
0.50
1000
23937
2326
3002
1233
0.00
1000
2326
1326
$3300
13227
10097
2533
2004
03133
2205
(h)
training accuracy vs epochs
tutt wo
312
0.10
3.
0.34
6.25
0.2
0.00
10000
Basch
sped
1000
validation accuracy epochs
Figure 4: Varying Adam on Division dataset. Observe that as e increases, there no Slingshot Effect or grokking behavior. Figure (a) corresponds to default suggested in where the model trained with smallest value undergoes multiple Slingshot eycles.
222 which confirms the observations made in Section with modular division dataset. In addition, models that exhibit Slingshot Effects and grokking (shown in green) tend to generalize better than models that do not experience Slingshot Effects and grokking (shown in red).
223
225
VIT with CIFAR-10 For further validation of Slingshot Effects and generalization, we train
226
Vision Transformer (ViT) 4 on CIFAR-10 (8) The ViT consists of layers, width 384 and
227
12 attention heads trained on fixed subsets of CIFAR- 10 dataset [8]. The ViT model described above trained with 10K, 20K, 30K, 40K and 50K (full dataset) training samples. We train
228
models with the following learning rates: 0.0001, 0.00031 and 0.001 and with linear learning rate warmup for the epoch of optimization. We consider multiple learning rates study the impact of
230
this hyperparameter on Slingshot taking inspiration from B where the authors report observing

---

No Singshot Effects, no grokiing
Singshot Eflects, grokking
Singshoo Effects and grakking
100
c-1608
100
101007
0
c-1006
100
1e05
100
colepnyy Stp
cabapaty 4llp
cataganyy Top
catoapoly Blip
diviiim Stop
diviinn 60p
divitonn, 70p
molipicantin slop
mosipiicat sruarppoity Pip
gyuumppnyy Slop
auterertion dlp
Auseractinn 2:00
adkittonn,
addifien
addinsn,
eviti anliitirn esid suaraattmn,
Figure 5: Extended analysis on multiple grokking datasets. Points shown in green represent both Slingshot Effects and grokking, points shown blue indicate Slingshot Effects but grokking while points in red indicate no Slingshot Effects and no grokking. Adam varied shown text. Observe that increases, there are no Slingshot Effects or grokking behavior.
grokking over narrow range of learning rates Figure S plot of the highest test accuracy for set of hyperparameters (learning rate, number of training samples) as function of the number of training samples from which we make the following observations. The best test accuracy for a given set of hyperparameters is typically achieved after Slingshot phase begins during optimization. The checkpoints that achieve the highest test accuracy are labeled as "post-slingshot" and shown in green 6 While post-Slingsho checkpoints seem to enjoy higher test accuracy, there are certain combinations of hyperparameters that lead to models that show better test accuracy prior to the start of the first Slingshot phase. We label these points as "pre-slingshot" (shown in blue) in Figure 6 The above observations appear to be consistent with our finding that training long periods of time may lead to better generalization seen with grokking datasets [3
232
233
234
236
237
in Figure
238
240
242
Non- Transformer Models We conduct experiments with MLPs on synthetic data where the synthetic data a low dimensional embedding projected higher dimensions via random projections. This design choice is critical with showing the existence of the Slingshot Effect with synthetically generated data. We find that using low dimensional data does not lead any Slingshots. With this
243
244
245
dataset, we show that generalization occurs late in training with Adam. Specifically, we tune in Adam and show that the optimizer is highly sensitive to this hyperparameter. These observations are consistent with the behavior reported above with Transformers and on algorithmic datasets well as standard vision benchmark such as CIFAR-10. We refer the reader to Appendix for complete description and details of these experiments.
247
249
3.3
Drawbacks and Limitations
252
While the Slingshot Mechanism exposes an interesting implicit bias of Adam that often promotes generalization, due to its arresting of the norm growth and ensuing feature learning, also leads to some training instability and prolonged training time. In the Appendix we show that it is possible to
253
254
255
achieve similar levels of generalization with Adam on the modular division dataset [33] using the
256
same Transformer setup as above, while maintaining stable learning, in regimes that do not show
257
clear Slingshot Effect. First we employ weight decay, which causes the training loss values to
converge to higher value than the unregularized model. In this regime the model does not become unstable, but instead regularization leads to comparable generalization, and much more quickly.
259
However, it is important to tune the regularization strength appropriately. Similarly, find that is

---

16
no slingshot pre-slingshot post-silngssot
test accuracy
20
number of training samples (x1000)
Figure 6: Slingshot Effects subsets of CIFAR- 10 dataset. We train ViTs with multiple learning rates to verify the impact this parameter has on Slingshot. Power et al [13] note that grokking occurs over narrow range of learning rates. Note that the points marked in: (i) green correspond to test accuracy for an experiment after Slingshot Effect begins, (ii) blue are for trials where best checkpoint is observed prior to start of a Slingshot Effect and (iii) red are for trials with no Slingshot Effect.
possible to normalize the features and weights using the following scheme to explicitly control norm growth: [w] f(x)
[f(x)]| where and f(x) are the weights and inputs to the classification layer respectively, the norm used above is the T22 norm, and is the input the neural network. This scheme also results in stable training and similar levels of generalization. In all cases the effects rely on keeping the weight norms from growing uncontrollably, which may be the most important factor for improving generalization. These results suggest that while the Slingshot Mechanism may be an interesting self-correcting scheme for controlling norm growth, there are likely more efficient ways
f(x))
264
265
286
257
268
to leverage adaptive optimizers to similar levels of generalization without requiring the instability
269
that is a hallmark of the Slingshot effect.
270
Finally, we lack a satisfactory theoretical explanation for the Slingshot Mechanism, and hence
271
removed all attempts more rigorous mathematical definition, which we feel would only serve as distraction.
Conclusion
273
274
We have empirically shown that optimizing deep networks with cross entropy loss and adaptive optimizers produces the Slingshot Mechanism, a curious optimization anomaly unlike anything described in the literature. We have provided ample evidence that Slingshot Effects can be observed with different neural architectures and datasets. Furthermore, we find that Grokking [13 almost always occurs in the presence of Slingshot Effects and associated regions of instability in the Terminal Phase Training (TPT). These results in their pure form absent explicit regularization, reveal an intriguing inductive bias of adaptive gradient optimizers that becomes salient in the TPT, characterized by cyclic stepwise effects on the optimization trajectory. These effects often promote generalization
275
276
277
278
279
281
in ways that differ from non-adaptive optimizers like SGD, and warrant further study be able to harness efficiently. There are open question remaining to be answered, for instance 1) What's the causal factor of the plateau of weight norm growth? 2) Are there better ways of promoting generalization without relying on this accidental training instability? Answering these questions w ill allow us decouple optimization and regularization, and ultimately control and improve them independently.
283
285
287
5 Societal Impact
289
This is a fundamental work Deep Learning, it will impact the society via its effects on relevant models and applications.

---

References
291
292
[1] Zeyuan Alu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via over-parameteriiation.. ArXiv,
2019.
294
[2]
Anas Barakat and Pascal Bianchi. Convergence and dynamical behavior of the adam algorithm for nonconvex stochastic optimization. SIAM J. 31:1m,,1:224 31:2444 274, 2021.
295
296
[3] Jeremy M. Cohen, Simran Kaur, Yuanzhi Li, J. Zico Kolter, and Ameet Talwalkar. Gradient descent on neural networks typically occurs at the edge of stability. arXiv preprint arXiv: Arxiv-2103.000 2021.
297
298
299
[4] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv: Arxiv-2010.1 1929, 2020.
300
301
302
303
[5] John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12(61):121-2215) 2011.
305
[6] Elad Hoffer, Itay Hubara, and Daniel Soudry. Train longer. generalize better: closing the generalization gap in large batch training of neural networks. ArXiv, abs/ htsp//705.08744) 2017.
308
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1rxiv-1412.699 2014.
308
[8]
Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.
310
[9]
Aitor Lewkowycz, Yasaman Bahri, Ethan Dyer, Jascha Sohl-Dickstein, and Guy Gur-Ari. The large learning rate phase of deep learning: the catapult mechanism. arXiv preprint arXiv:2003.022/8, 2020.
311
312
313
[10] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint a7X1v=1711.05101, 2017.
315
[11] Kaifeng Lyu and Jian Li. Gradient descent maximizes the margin of homogeneous neural networks. arXiv preprint 190ivv/906.003 2019.
316
317
[12] Vardan Papyan, X. Han, and David L. Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences of the United States of America, 17:224652 24663, 2020.
318
319
320
[13] Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Generalization beyond overfitting on small algorithmic datasets. In ICLR MATH-AI Workshop, 2021.
322
323
[14] Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The implicit bias of gradient descent on separable data. The Journal of Machine Learning Research, 19(1):2222-2778. 2018
324
326
[15] Tijmen Tieleman and Geoffrey Hinton. Lecture 6.5 6.5-rmsprop, coursera: Neural networks for machine learning. University of Toronto, Technical Report, 6, 2012.
327
[16] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. arXiv preprint arXiv: Arxiv- 2017.
329
330 /
331
[17] Bohan Wang, Qi Meng, Wei Chen, and Yan Liu. The implicit bias for adaptive optimization algorithms on homogeneous neural networks. In International Conference on Machine Learning. pages 10849-10858. PMLR, 2021.
332
333
334 [18] J. Zhang, Tianxing He, Suvrit Sra, and Ali Jadbabaic. Why gradient clipping accelerates training: theoretical justification for adaptivity. arXiv: Optimization and Control, 2020.
justtiicaaisttiifccatioo
335
10

---

Checklist
336
337
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No], , or [N/A]- - You are strongly encouraged to include a justification your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
338
339
340
341
Did you include the license to the code and datasets? [Yes] See Section ??.
342
Did you include the license the code and datasets? [No] The code and the data are proprietary.
343
344
Did you include the license to the code and datasets? [N/]]
345
Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below,
346
347
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
350
351
(b) Did you descrihe the limitations of your work? [Yes]
352
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
354
355
2. If you are including theoretical results...
356
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
357
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
359
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes]
360
381
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
362
363
(c) Did you report error bars (e.g.. with respect to the random seed after running experi- ments multiple times)? [Yes]
364
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
386
387
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
368
(a) If your work uses existing assets, did you cite the creators? [Yes]
369
(b) Did you mention the license of the assets? [Yes]
370
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
371
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes]
372
373
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes]
374
375
5. If you used crowdsourcing or conducted research with human subjects...
376
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [Yes]
378
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [Yes]
379
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [Yes]
381
11


Reviews:
[{"review_id": "tvlyTbF4r4Y", "reviewer": "Reviewer_BLwe", "summary": "The paper presents an empirical study on an optimization anomaly, which is referred to as the Slingshot Mechanism/Effect by the authors, when adaptive optimizers (e.g., Adams) are used for solving supervised learning problems. Specifically, it was found that, when adaptive optimizers were used, cyclic phase transitions between stable and unstable training occurred and were correlated with how the norm of the last-layer weights changed. The authors also reported findings on the correlations between the onset of the Slingshot Mechanism and the glokking phenomenon. ", "questions": "**Is the reported phenomenon unique to \"deep models\"?**\n\nI did not see any numerical experiments on \"shallow models.\" From the appendix, I only see experiments with a transformer (12 layers and 10 million parameters), a CNN with VGG-like architecture (which I assume to be a deep model), a deep MLP with 6 layers, and a deep linear model with 6 layers. Without experiments with simple/shallow models, it is premature to draw the conclusion that Slingshot is a general phenomenon as the authors claimed in Lines 154-162. \n\nAs a matter of fact, given that the authors tried to illustrate the phenomenon using simple quadratic functions (Line 174), could the authors conduct a set of numerical experiments using quadratic functions to see whether the phenomenon still persists?\n\n**Some simple numerical diagnoses could have been done.**\n\nAs evident from Line 178, one issue with the adaptive optimization schemes considered is that the effective step size $\\mu / (|g| + \\epsilon)$ can be much larger than the unnormalized step size $\\mu$ when the magnitude of the gradient $g$ is small and $\\epsilon$ is small. (Here, a small $\\epsilon$ is assumed; otherwise, the update rule will not adapt much to the magnitude of $g$.) This implies that, when the magnitude of $g$ is small, i.e., when the current iterate is near a stationary point, the next iterate of the gradient-based update can be very far from the stationary point, potentially leading to a sudden increase in the loss. \n\nFor gradient-based updates, when the objective function is smooth, it is known that a sufficient condition for decrease is to choose the step size to be less than $2/L$, where $L$ is the Lipschitz constant of the gradient (and is related to the Hessian). It would be useful to plot in Figure 3 the effective step size, given by $\\mu / (|g| + \\epsilon)$, along with the sharpness to see if the step size is still small enough for the sufficient condition for decrease to hold. If the violation of the condition coincides with the increase in the loss, then this would provide a simple explanation of the Slingshot Effect.\n\n**Comments on presentation and typos:**\n\n* Are the Slingshot Mechanism and the Slingshot Effect the same thing? Both terms are used throughout the paper.\n* Figure 1 and its first citation are too far apart: Figure 1 appears on the first page (even before the main text), but it citation does not appear until Line 158. \n* A typo in Line 190: \"...evidence of **Elingshot** Effects.\"\n* Line 230: What is $3.10e^{-04}4$? Also, it uncommon to write $10^{-04}$ instead of $10^{-4}$. This has occurred several times throughout the paper.", "limitations": "I do not have any comments on this.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n* The reported findings hint that extra care may be required when adaptive optimizers are used. The results may also promote further theoretical studies on the limitations of using adaptive optimizers in supervised learning.\n\nWeaknesses:\n\n* Additional numerical experiments could have been conducted to help future theorists better understand the source of issues. In particular, some rather simple diagnoses could have been done but were missing from the paper.\n* There is much room for improvement in presentation of the paper. See the specific comments in the Questions section.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "s96PB6Fbfjs", "reviewer": "Reviewer_8hZg", "summary": "The authors describe a \"slingshot\" effect present in the use of Adam (and cousins). This purportedly cyclic effect can be observed when the norm of the final layer sharply increases, often accompanied by an increase in train loss. The authors offer the slingshot effect as a possible explanation for the ability of Adam to induce generalization even without explicit regularization.", "questions": "1. Is there a reason validation error/loss is not depicted in Fig 1? Given the emphasis in this paper on generalization, it seems odd not to depict in the headline figure.\n2. Figure 2(c) looks either incorrect or misgenerated—the validation loss should not be zero at the beginning of training, unless I am missing something.\n3. Do the authors have any explanation for why Figure 3(d) doesn't show a high-curvature update on the second plateau?\n4. Can the authors elaborate more on the implications of the larger $\\epsilon$ result? To me, larger $\\epsilon$ ruining the grokking effect sort of implies that non-adaptive methods (e.g. plain GD) don't benefit from slingshotting either (which is exactly what the authors found empirically). But it would still be good to explicitly connect the dots here.\n5. I am also left wondering about the result on the paragraph starting on line 243. It's cool that the authors discovered it but what is it supposed to tell me? What about low-dim data makes the slingshot effect less likely to occur?\n6. Finally, is it fair to call this effect cyclic when most empirical examples only have two cycles (with a few having 3)? Is there a setup where we can see $n$ distinct plateaus and sharp norm increases for some $n >> 3$?", "limitations": "The authors do a good job not overclaiming, and they explicitly enumerate a number of other regimes that achieve the results despite lack of \"slingshot\". Certain items are left unaddressed (see \"Questions\").\n\nMinor\n----\n\n- Line 84: missing author name\n- Line 112: what is \"Adam without momentum\" and how is it different from RMSProp?\n- Line 145: \"layers\" $\\rightarrow$ \"layer\"\n- Line 170: \"has\" $\\rightarrow$ \"have\"\n- Line 245: \"choices\" $\\rightarrow$ \"choice\"\n- Line 254: \"it's\" $\\rightarrow$ \"its\"", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This is overall a nicely executed empirical paper that reveals a moderately interesting phenomenon. A cyclic regime of generalization would be quite remarkable and this paper offers preliminary evidence that supports the authors' hypothesis. It is also refreshing that the authors did not attempt to handwave nonsense poorly-motivated math into the paper, which is unfortunately a severe problem even in 2022.\n\nMy main concerns are the following:\n\n- Significance: Given that this effect explains generalization in unregularized optimization, the overall impact to practical deep learning is somewhat muted.\n- Strength of evidence: the results in the paper leave me with certain questions (most notably #6 below) as to whether the authors have sufficiently validated their hypothesized effect.\n- Implications: the paper enumerates a number of interesting facts (e.g. dependence on $\\epsilon$) but doesn't really dive deeper beyond some of those facts, leaving the reader wondering as to why those facts were presented.\n\nOverall, I enjoyed reading this paper. I don't think it's super noteworthy in terms of immediate impact, but it is (as far as I can tell) a novel discovery that may serve as a waypoint on our path to better understanding of oddball generalization phenomena in deep learning.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "GmMV6-olEoU", "reviewer": "Reviewer_KF2y", "summary": "The paper studies the grokking phenomenon introduced by Power et al [1], which is a curious phenomenon of delayed generalization. This paper describes a characteristic of training termed the \"slingshot mechanism\", that is found in a number of tasks and optimization settings. The slingshot mechanism refers to periods of instability found in adaptive optimizers in the late stages of training. These periods of instability are accompanied by norm increases in the classification layer, as well as an improvement in generalization performance for the grokking tasks introduced in [1]. The paper studies this phenomenon and finds that it occurs in a number of different tasks more general than those in [1], and find that it is controlled by the epsilon parameter in adaptive optimizers.", "questions": "- In Figure 3(d), why do we not see a corresponding spike in sharpness at the onset of the second plateau? Based on the hypothesis that the weights are flung to a new region in parameter space when the curvature grows too large, I would expect to see the curvature spike at each transition.\n\n- Figure 5, what is the accuracy being plotted? Is it training accuracy or test accuracy? If test accuracy, what is the training accuracy of these models? Do all the runs achieve 100% training accuracy? It is unclear whether correlation with better generalization comes from the appearance of slingshots or from the bad hyperparameter choices required to remove slingshot behavior from the training trajectory. \n\n- Dataset size is a crucial parameter in determining whether grokking / generalization occurs in [1]. How does the correlation between generalization vs slingshot behavior change when models are trained with differing data size? Does slingshots exist regardless of whether the model generalizes?", "limitations": "Beyond the weaknesses I listed, the authors were good at addressing several limitations of this work.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n- The paper has a good and comprehensive discussion of related work.\n- The paper characterizes an interesting phenomenon that is sometimes found in the optimization trajectory of adaptive optimizers.\n- Interesting observations regarding the effect of epsilon on the slingshot behavior.\n\n### Weaknesses\n\n- There is no explanation, empirical or otherwise, for how the slingshot mechanism may relate to generalization. Understanding the delayed generalization is an important aspect of why the grokking phenomenon in [1] is interesting to the community. I believe that in the current state, without a deeper understanding, this work may be more appropriate for a workshop submission.\n- The paper claims that the slingshot mechansim is a general phenomenon, however on CIFAR10 the phenomenon was mostly observed when training with extremely small training data (e.g. 200 examples), and no correlation to better generalization was shown. There is one experiment in the Appendix where the full CIFAR10 dataset is used in training a ViT model, but this model only achieves a final test accuracy of <80% and it's not clear how much test accuracy is improved due to the slingshot mechanism due to a single run and increasing test loss. This calls into question how relevant or general the identified behavior is to modern day neural networks.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Reject