The Slingshot Effect: A Late-Stage Optimization Anomaly in Adaptive Gradient Methods
Authors: Vimal Thilak
Etai Littwin
Shuangfei Zhai
Omid Saremi
Roni Paiss
Joshua M. Susskind
Venue: Accepted by TMLR ()

Abstract: Adaptive gradient methods, notably Adam ~\citep{kingma2014adam, loshchilov2017decoupled}, have become indispensable for optimizing neural networks, particularly in conjunction with Transformers ~\citep{vaswani2017attention, dosovitskiy2020an}. In this paper, we present a novel optimization anomaly called the \emph{Slingshot Effect}, which manifests during extremely late stages of training. We identify a distinctive characteristic of this phenomenon through cyclic phase transitions between stable and unstable training regimes, as evidenced by the cyclic behavior of the norm of the last layer's weights. Although the Slingshot Effect can be easily reproduced in more general settings, it does not align with any known optimization theories, emphasizing the need for in-depth examination.
Moreover, we make a noteworthy observation that Grokking, as reported by ~\citet{power2021grokking}, occurs predominantly during the onset of the Slingshot Effects and is absent without it, even in the absence of explicit regularization. This finding suggests a surprising inductive bias of adaptive gradient optimizers at late training stages, urging a revised theoretical analysis of their origin.
Our study sheds light on an intriguing optimization behavior that has significant implications for understanding the inner workings of adaptive gradient methods.

Published in Transactions on Machine Learning Research (02/2024)
The Slingshot Effect: A Late-Stage Optimization Anomaly in Adam-Family of Optimization Methods
Vimal Thilak Apple
uthilak uthilak&@ppler Gapale.cppppooommmmm
Etai Littwin Apple
elitboin com
Shuangfei Zhai
Sthat@apple.com
Apple
Omid Saremi
oseremi@apple.ccom
Apple
Roni Paiss
puiss.ronii@gaasl.com
Joshua Susskind
jsusskind@apple.com
Apple
Reviewed on OpenReview: https: // openreview. net/ net/forum? forum? id= OZhn@ULouY
Abstract
Adam Kingma & Ba 2014 and Adam-family [Loshchilov & Hutter 2017 Ticleman & Hinton 2012) of adaptive gradient methods have become indispensable for optimizing neural networks, ks, particularly conjunction with Transformers (Vaswani et al. 2017 Dosovitskiy et al. 2020). In this paper, we present novel optimization anomaly called the Slingshot Effect, which manifests during extremely late stages of training. We identify distinctive characteristic of this phenomenon through cyclic phase transitions between stable and unstable training regimes, evidenced by the cyclic behavior of the norm of the last layer's weights. Although the Slingshot Effect can be easily reproduced in more general settings, it does not align with any known optimization theories, emphasizing the need for in-depth examination. Morcover, we make a noteworthy observation that Grokking, as reported by Power al. (2021), occurs predominantly during the onset of the Slingshou Effects and is absent without it, in the absence of explicit regularization. This finding suggests surprising inductive bias of adaptive gradient optimizers at late training stages, urging a revised theoretical analysis of their origin. Our study sheds light on an intriguing optimization behavior that. has significant implications for understanding the inner workings of Adam- fhan-family of gradient methods.
1 Introduction
Adaptive optimizers Kingma & Ba 2014 Loshchilov Hutter 2017 Ticleman Hinton 2012) are widely used to train deep neural networks including ResNets (Wightman al.) [202]] Dosovitskiy et al.) 20200)
and Transformers /Vaswani et al. 2017
Dosovitskiy al. 2020) they empirically demonstrate st.oong

---

Published in Transactions on Machine Learning Research (02/2024)
train loss
last layer norm
Slingshot
100
0.0785
10
layer norm
0.0780
loss
10
0.0975
10
0.0770
10
1000
1200
1400
1600
1800
2000
0.0765
growth
Nomn
spake
plateou
epoch
Figure 1: Slingshot Effects observed with fully-comnected ReLU network (FCN). The FCN is trained with 200 randomly chosen CIFAR-10 samples with Adam. Multiple Slingshot Effects occur cyclic fashion as indicated by the dotted red boxes. Each Slingshot Effect is characterized by period of rapid growth of the last layer weights, ensuing training loss spike, and period curtailed norm growth.
performance. Recently, several works Chowwheer et al. 2022 Molybog et al. 2023 Wortsman el al. 2023
Courtois et al. 2023 Zhu et al. 2023) have reported optimization instability of Adam or AdamW (Kingma & Ba 2014 Loshchilov & Hutter 2017 when used train deep neural networks. Specifically, the training instabilities are characterized by several spikes in the loss during optimization. This curious behavior has motivated studies that attempt to understand and explain the source of these instabilities (Molybog et al.
2023 or provide mitigation strategies to stabilize training [Chowdhery et al. 2022
Wortsman et al. 2023
In this paper, we present an optimization anomaly that we call the Slingshot Effect wherein the training loss of neural networks optimized with Adam-family of adaptive gradient methods (see Algorithm 1 for a generic description of methods used in this work) exhibit loss spikes. Figure [ shows an example the training loss spikes observed during neural network optimization. As can be seen from Figure [ the instabilities are observed at extremely late stages of training or in the so-called Terminal Phase of Trarning (TPT) In detail, leveraging the basic sctup in Power et al. (2021) with Adam (Kingma & Ba 2014 and without weight decay, we make the following additional observations visualized in Figure? 2
1. During the TPT, training exhibits cyclic behaviour between stable and unstable regimes. A prominent artifact of this bchaviour can be seen in the norm of the classification layer's weights, which exhibits a cyclical behavior with distinct, sharp phase transitions that alternate between rapid growth and growth curtailment over the course of training.
2.
The norm grows rapidly sometime after the model has perfect classification accuracy on training data. A sharp phase transition then occurs in which the model misclassifies training samples. This phase change is accompanied by sudden spike in training loss, and deceleration in the norm growth of the final classification layer.
3. The features (pre-classffication layer) show rapid evolution as the weight norm transitions from rapid growth phase to growth curtailment phase, and change relatively little at the norm growth phase.
4. Phase transitions between norm growth and norm curtailment phases are typically accompanied by sudden bump in generalization as measured by classification accuracy on a validation set, as observed in a dramatic fashion in Power et. al. (021)
is empirically observed with the sctup described above (Adam with no weight decay) that grokking as reported in Power et al. (2021) almost exclusively happens at the onset of Slingshots, and is absent without it.
2

---

Published in Transactions on Machine Learning Research (02/2024)
We denote the observations above as the Slingshot Effect (or simply slingshot), which is defined to be the full cycle starting from the rapid norm growth phase, and ending in the curtailed norm growth phase. Empirically, a single training run typically exhibits multiple Slingshots. Moreover, while grokking as described in Power ct al. 2021 might be data dependent, we find that the Slingshot Effect is pervasive, and can be easily reproduced in multiple scenarios, enconpassing variety of models (Transformers and MLPs) and dalasets (both vision, algorithmic and synthetic datasets). Since we only observe Slingshots when training classification models with adaptive optimization methods such as Adam Kingma & Baj (2014) and RMSProp (Tieleman) implicit bias of such optimizers.
& Hinton 2012), our work can be seen empirically characterizing Finally, we focus on Adam (Kingma & Ba 2014) optimization method in the main paper, and relegate all experiments with additional optimizers to the appendix that suggest that our observations and conclusions hold for other methods under the Adam-fanily of optimizers 4 We note here that we limit our empirical
analysis Slingshot Effects with Adam (Kingma & Ba 2014), RMSProp (Tieleman & Hinton 2012) and
Adagrad (Duchi ct al., (2011) while acknowledging that there are a vast number of other adaptive optimization
methods including AMSGrad (Reddi et al. 2018). Adax (Li et al. 2020), AdaBound (Luo et al., 2019 and
AdaBelief (Zhuang et al. 2020 that are not used in analysis.
Algorithm Adam-family of Adaptive Gradient Methods
Input: 21 E J initial parameters,
Input: step size (t,
Input: 81 and $2 C [),
Input: R
Output: Optimized parameters 27411
Initialize mg and
for t I..., do
s 9t = f((ee)). (Get gradients g w.r.t function f)
my B1m4-1 + (1 - B1)9t
5
my I-f
me
Ve 32Vi-1 (1 2
Ve V4
1-88
2141
2t
Vi+e
pma
(Update parameters)
1.1 Implications of Our Findings
The findings in this paper have both theoretical and practical implications that go beyond characterizing the effect. A prominent feature of the Slingshot Effect is the repeating phase shifts between stable and unstable training regimes, where the unstable phase is characterized by extremely large gradients, and spiking training loss. Furthermore, we find that learning at late stages of training have cyclic property, where non trivial feature adaptation only takes place at the onset of a phase shift. This abrupt feature learning characteristic is observed in a dramatic fashion when considcring the grokking phenomena, where we observe the slingshot effect at the onset of grokking, pointing lowards possible generalization ion benefits of the effect. From theoretical perspective, this is contradictory to common assumptions made in the literature of convergence of adaptive optimizers, which typically require L smooth cost functions, and bounded stochastic gradients, either in the L2 or Lxo norm, decreasing step sizes and stable convergence (Zhang et al. 2020 Ahu et al. 2019 Barakat & Bianchi 2021). The generalization behavior observed with the grokking setup (Power et al. 2021) casts doubt on the ability of current working theories to explain the Slingshot Effect.
Practically, our work presents additional evidence for the growing body work indicating the importance of the TPT stage of training for optimal performance (Hoffer et al., 2017 Power et al., 2021 Papyan al., 2020). In an era where the sheer size of models are quickly becoming out of reach for most practitioners, our work suggests focusing improved methods to prevent excessive norm growth either implicitly through Slingshot Effects or through other forms of explicit regularization or normalization which we study in the appendix.
We the " "Adammfamily" refer to those methods whose update equations are defined in Kingma Ra 2014
3

---

Published in Transactions on Machine Learning Research (02/2024)
2
Related Work
The Slingshot Effect we uncover here is reminiscent of the catapult mechanism described in Lewkowycz et al. (2020). Lewkowycz et al. (2020) show that loss of a model trained via gradient descent with an appropriately large learning rate shows non monoonnn behavior the loss initially increases and starts decreasing once the model "catapults" to a region of lower curvature early in training. However, the catapult phenomenon differs from Slingshot Effects in several key aspects. The catapult mechanism is observed with vanilla or stochastic gradient descent unlike the Slingshot Effeet that is seen with adaptive optimizers including Adam (Kingma & Ba 2014) and RMSProp (Tieleman & Hinton, 2012). Furthermore, the catapult phenomenon relates to a large initial lcarning rate, and docs not exhibit a repeating cyclic behavior. More intriguingly, Slingshot Effects only emerge late in training, typically long after the model reaches perfect accuracy on the training data.
Cohen et al. 2021) describe a "progressive sharpening" phenomenon in which the maximum eigenvalue of the loss Hessian increases and reaches value that is at equal to or slightly larger than 2/7 where 7) is the learning rate. This "progressive sharpening" phenomenon leads the model to enter regime Cohen et al.] (2021) call Edge of Slability (EoS) where-in the model shows non-monotonic training loss behavior over short time spans. In a recent work, Cohen et al. (2022) demonstrate a similar phenomenon with full-batch (or with large batch empirically) adaptive optimizers called Adaptive Edge of Stability (AEoS) where-in the "raw" sharpness quantificd by the maximum cigenvalue of the loss Hessian increasos and oscillates around a certain value for a version of Adam [ohhen et al. (2021) call "Frozen Adam". However, Cohen et al. (2022) observe that the 'raw" sharpness continues to increase during training with regular Adam (Kingma & Ba 2014]. Edge of Stability and Adaptive Edge of Stability are similar to Slingshot Effects in that the instablities are observed later on in training. However, while EoS Cohen et al. (2021) and AEoS (Cohen et al.
2022) aim to
shed light, on neural network training dynamics they do not show or analyze the cyelic behavior of the last, layer norm weights and do not examine generalization that we observe in our experiments.
As noted above, the Slingshot Effect emerges late in training, typically longer after the model reaches perfect accuracy and has low loss on training data. The benefits of continuing to training a model in this regime has been theoretically studied in several works including (Soudry et al. 2018 Lyu & Li, 2019). [Soudry et al. (2018) show that training linear model on separable data with gradient using the logistic loss function leads to max--arggh solution. Furthermore, Soudry et al. (0188) prove that the loss decreases at a rate of O while the margin increases much slower O (ooge) where is the number of training stops. Soudry ct al. 20088 also note that the weight norm of the predictor layer increases at logarichmic rate, i.e., O(log(4)) Lyu &
Li (2019) generalize the above results to homogeneous neural networks trained with exponential-type loss function and show that loss decreases at a rate of ((1/(log(tt)2222
where L is defined as the order of the homogenous neural network. Although these results indeed prove the benefits of training models, their analyses are limited to gradient descent. Morcover, the analyscs developed by (Soudry et al. (2018) do not predict any phenomenon that resembles the Slingshot Effect. Wang et al. (2021) show that homogenous neural networks trained with RMSProp (Tilemaa & Hinton 2012) or Adam (Kingma & Ba 2014] without momentum converge in direction to the max-margin solution. However, none of these papers can explain the Slingshot Effect and specifically the cyclical behavior of the norm of the last layer weights.
Understanding Grokking The grokking phenomenon (Power et al. 2021 has been studied in several recent works that aim to understand and explain the empirical findings by Power et al. (2021). Liu et al. (2022) use a toy model to study grokking and show four phases of learning and that feature learning occurs at slower rate that leads to delayed generalization. In a follow-up work, Liu et al. (2023) analyze the train and test loss landscapes and suggest that grokking occurs due to mismatch between the train and test loss landscapes. Nanda et al. (2023) analyze the modular addition example proposed by (Power et al. (2021) and identify three phases of learning and propose a "hidden measure" that track the development of struct ure in the weights of a model. Barak et al. (2022) analyzes the behavior of neural networks on subset parity task and show that grokking-like. phenomenon can be soen in that sctup even with multi-layer perceptrons (MLPs). We empirically analyze grokking (Power et al., 2021 Barak et al. 2022) in a setup that does not inchude any explicit regualarization like weight decay unlike the other works that use weight decay.
4

---

Published in Transactions on Machine Learning Research (02/2024)
Relationship to Training Instability or Loss Spikes Several papers (Chowdhery et al.
2022 Molybog
et al. 2023: Wortsman et al., 2023 Zhu et al. 2023 Lobacheva et al., 2021 and references therein have
reported optimization instability of Adam (Kingma & Ba 2014) or AdamW (Loshchilov & Hutter 2017] that is characterized by spikes in the loss during training. Molybog et al.
(2023) theoretically analyze the unstable behavior observed while training large scale models with Adam (Kingma & Ba 2014] and propose several mitigation approaches to stabilize training. Wortsman et al. (2023) also report loss spikes while training large scale vision models and observe that the loss spikes occur due to either an out-of-date second order gradient moment estimate or due to lower precision floating point (16-bit) training. Wortsman et al. (2023) provide mitigation strategies based on these observations. We note here that most literature related to training instability is focused on stabilizing optimization with the notable exception of
Zhuu et al. (2023
who provide empirical evidence that training instabilities may lead to better generalization ion. While Zhu el al.
2023) use stochastic gradient descent (SGD) in their work, we empirically analyze the optimization and gencralization bchavior of neural networks that exhibit training loss spikes with adaptive optimizers.
3
The Slingshot Effect
3.1
Experimental Setup
We use the training setup studied by Power et al. 2021 in the main paper as a working example to illustrate the Slingshot Effect. In this setup, we train decoder-only Transformers (Vaswani et al.) [017] on modular division dataset (Power et al. 2021) of the form a + b = c, where a, b and c are discrete symbols and + refers to division modulo p for some prime number p, split into training and validation sets. The task consists of calculating e given a and experiments are described in Appendix B. The Transformer consists of layers, of width 128 and 4 attention heads with approximately 450K trainable parameters and is optimized by Adam (Kingma & Ba 2014 Loshchilov & Hutter 2017 For these experiments we set learning rate to 0.001, weight decay to 0, B1 0.9,
The algorithmic operations and details of the datasets considered in our
32- 0.98, = 10-s, linear learning rate warmup for the first 10 steps and minibatch size to 512 which are in line with the hyperparameters considered in (Power et al. 2021
3.2 Experimental Observations
Figure shows the metrics of interest that we record on training and validation samples for modular division dataset. Specifically, we measure 1) train loss; 2) train accuracy; 3) validation loss; 4) validation accuracy; 5) last layer norm: denoting the norm of the classification layer's weights and 6) feature change: the relative
change of features of the 1-th layer (h') after the t-th gradient update step [lb] --tll We observe from A:ll
Figure 25 that the model is able to reach high training accuracy around step 300 while validation accuracy starts improving after 10 steps as seen Figure 2d Power et al. (2021) originally showed this phenomenon and refer to it as grokking. We observe that while the validation accuracy does not exhibit any change until much later in training, the validation loss shown in Figure [c] exhibits double descent behavior with an initial decrease, then a growth before rapidly decreasing to zero.
Seemingly, some of these observations can be explained by the arguments in (Soudry et al. 2018 and their extensions to adaptive optimizers (Wang et al. (202]). Namely, at the point of reaching perfect classification of the training set, the cross-entropy (CE) loss by design pressures the classification layer to grow in norm at relatively fast rate. Simultaneously, the implicit bias of the optimizer coupled with the CE loss, pushes the dircction of the classification layer to coincide with that of the maximum margin classifier, albeit at a much slower rate.
These insights motivate to measure the classifier's last layer norm during training. We observe in Figure 2a that once classification reaches perfect accuracy on the training set, the classification layer norm exhibits a distinct cyclic behavior, alternating between rapid growth and growth curtailment, with a sharp phase transition between phases. Simultaneously, the training loss retains a low value in periods of rapid norm growth, and then wildly fluctuates time periods when there is a curtailment of norm growth. Figure 2cc and Figure shows the evolution of the relative change in features output by each layer in the Transformer. We observe that the feature maps are not updated much during the norm growth phase. However, at the
5

---

Published in Transactions on Machine Learning Research (02/2024)
Tain
artiager Wor
letiayer rem
wilderin
103
200
101
Lyd Sayer oeitt
1HC SYA AOTEL
60
3004
100m
20
12-11
102
30
10
137
10
10
30
12"
16'
101
10'
2385
(a)
(b)
(c)
dedci accurecy
last layur ror
10/or2 Nestars
l0c layst com
300
0.000
0,0:0
0.00
anl laye 7tmm
Isturo Charg.
2.020
facure
0.00
5.035
5,005
0.2
0.00
5.000
201
204
30
30
sipp
stpp
(d)
Figure 2: Division dataset: Last layer weight norm growth versus a) loss on training data b) accuracy on training data (c) loss on validation data d) accuracy on validation data e) normalized relative change in features of first Transformer layer (I) normalized relative change in features of second Transformer layer. Note that the feature change plots are shown starting 10K step to emphasize the feature change behavior during rapid norm growth and norm growth curtailment phases, revealing that the features stop changing during the norm growth phase and resume changing during the curtailed norm growth phase.
phase transition, we observe that the feature maps receive rapid update, which suggests that the internal representation of the model is updating.
Are Slingshot Effects a general phenomenon? In an atlempt to ascertain the generality of Slingshot Effects as an optimization artifact, we run similar experiments with additional architectures, datasets, optimizers, and hyperparameters. We use all algorithmic datasets as proposed in (Power et al. 2021 as well as frequently used vision benchmarks such as CIFAR-10 (Krizhevsky) 2009). and even synthetic Gaussian dataset. For architectures, we use Transformers, MLPs and deep linear models (see figure [ We find abundant evidence of Slingshot Effects in all of our experiments with Adam and RMSProp. We are unable to observe Slingshot Effects with Adagrad (Duchi al. 2011) and also with stochastic gradient descent (SGD) or SGD with momentum, pointing to the generality of the effect across architectures and datasets. We refer the reader to Appendix A for the full, detailed description of the experiments.
Why do Slingshot Effects happen? We hypothesize that the norm growth continues until the curvature of the loss surface becomes large, effectively "flinging" the weights to a different region in parameter space as small gradient directions get amplified, reminiscent the mechanics of slingshot flinging a projectile. We attempt to quantify how far a model is flung by measuring the cosine distance between a checkpoint. during optimization and initial parameters. Specifically, we divide the model parameters into representation (pre-classifier) parameters and classifier (last. layer) parameters and calculate how far these parameters have moved from initialization. We show that checkpoints collected after model experiences Slingshot have larger representation cosine distance. We defer the reader to the appendix for further details.
By design, adaptive optimizers adapt the learning rale on a per parameter basis. In loy, convex scenarios, the parameter provably determines whether the algorithm will converge stably. To illustrate this, we take inspiration from Cohen et al. (2021), and consider a quadratic cost function L(A, B)C) = f Ar + B + C, A E Raxd xx, E Rd. Ce R, where we assume A is symmetric and positive definite. Note that the global minimum of this cost. is given by x² - B. The gradient of this cost with respect to is
6

---

Published in Transactions on Machine Learning Research (02/2024)
update sharpness
layer
update sharpness
last layer norm
1.75
1.50
2.0
Sharpness
50
last Sayer nOrTh
20
0.75
1.2
0.50
0.25
10
0.0
2000
4000
6000
8000
10000
2000
4000
6000
8000
10000
epoch
epoch
(a))
(b)
update sharpness
last layer norm
last layer
2.00
2.00
1.75
25.0
1.50
200
30
sharpness
1.2
lavd layer homm
1.00
20
0.75
0.50
0.50
0.25
0
2000
10000
2000
4000
10000
epoch
opach
(c)
(d)
Figure 3: Curvature metric (denoted "update sharpness") evolution norm growth (a) addition, (b) subtraction, (c) multiplication, and (d) division dataset. Note the spike in the sharpness metric near the phase transitions between rapid and curtailed norm growth.
given by g Ar + B. Consider optimizing the cost with adaptive optimization steps of the simple form A T++1 = 11 1' = 1 Bl= where is learning rate, and the division and absolute operations are taken clement wisc. Starting from some Th, the crror 6 Fx x² evolves according to:
e++1 4+11= -I-judiag [ + A)e; def Mee
Note that the condition ||||| < 2c where || Ils denotes the spectral norm, implies that the mapping Me is a contraction for all values of i, and hence convergence to the global optimum is guaranteed (This is in contrast to gradient descent, where the requirement is ||A]] < 2). Note that the choice of e crucially controls the requirement on the curvature of the cost, represented by the the spectrum of A in this case. In other words, the smaller e, the more restrictive the requirements on the top eigenvalue A. Cohen et al. (2021) observed that full batch gradient descent increases the spectral norm of the Hessian its maximum allowed value. We therefore hypothesize that for deep networks, a small value for requires convergence to a low curvature local minimum, causing Slingshot Effect when this does not occur. Morcover, we may reasonably predict that increasing the value of would lift the restriction on the curvalure, and with it evidence of Slingshot Effects.
Figure 3 shows evidence consistent with the hypothesis that Slingshot Effects occur in the vicinity of high loss curvature, by mcasuring the local loss surface curvature along the optimization trajectory. Let H4 denote the local Hessian matrix of the loss, and u, the parameter updale al time 1 given the optimization algorithm of choice. We use the local curvature along the trajectory of the optimizer, given by | curvature measure. Across the arithmetic datasets from (Power et al. 2021). whenever the last layer weight norm growth is curtailed, the curvature measure momentarily peaks and settles back down.
1 uf H,u,, a
We next observe from Figure 2a that the training loss value also spikes up around the time step when the weight norm change transitions from rapid growth to curtailed growth phase. A low training loss
Varying
7

---

Published in Transactions on Machine Learning Research (02/2024)
value suggests that the gradients (and their moments) used as imputs to the optimizer are small, which in turn can cause the hyperparameter value to play a role in calculating updates. Our hypothesis here is that the Slingshot Effect should eventually disappear with a sufficiently large e. To confirm this hypothesis, we run an experiment where we vary e while retaining the rest of the sctup described in the previons section.
Figure 4 shows the results for various values of e considered in this experiment. We first observe that the number of Slingshot Effect cycles is higher for smaller values of e. Secondly, smaller values of e cause grokking to appear an carlier time step when compared to larger values. More intriguingly, models that show signs of grokking also experience Slingshot Effects while models that do not experience Slingshot Effects do not show any signs of grokking. Lastly, the model trained with the largest = 10-5 shows no sign of generalization even after receiving 500K updates.
3.3 Effects on Generalization
In order to understand the relationship between Slingshot Effects and neural networks generalization, we experiment with various models and datasets. We observe that models that exhibit Slingshot tend to generalize better, which suggests the benefit of training models for a long time with Adam (Kingma & Ba 2014 More surprisingly, we obscrve that Slingshots and grokking tend to come in tandem.
Transformers with algorithmic datasets We follow the setting studied by Power et al. 2021 and generate several datasets that represent algorithmic operations and consider several training and validation splits. This dataset creation approach is consistent with the methodolog used to demonst rate grokking (Power et al. 2021). The Transformer is trained with Adam (Kingma & Ba 2014 Loshchilov & Hutter (017) with a learning rate of 0.001, weight decay set to O, and with learning rate warmup for 500K steps. We consider e of Adam as hyperparameter in this experiment. Figure summarizes the results for this experiment where the x-axis indicates the algorithmic operation followed by the training data split size. As can be seen in Figure 5 Slingshot Efffets are seen with lower values of and disappear with higher values of e which confirms the observations made in Section 3 with modular division dataset. In addition, models that exhibit Slingshot Effects and grokking (shown in green) tend to generalize better than models that do not experience Slingshot Effects and grokking (shown in red).
VIT with CIFAR-10 For further validation of Slingshot Effects and generalization, we train a Vision Transformer (ViT) (Dosovitskiy et al.) 2020) on CIFAR-10 (Krizhevsky) 2009). The VIT consists of 12 layers, width 384 and 12 attention heads trained fixed subsets of CIFAR-10 10 dataset (Krizhevsky 2009 The ViT model described above is trained with 10K, 20K, 30K, 40K and 50K (full dataset) training samples. We train the models with the following learning rates: 0,0001, 0.00031 and 0.001 and with a linear learning rate warmup for the 1 epoch of optimization. We consider multiple learning rates to study the impact of this hyperparameter on Slingshot taking inspiration from Power et al. 22021] where the authors report observing grokking over a narrow range of learning rates Figure 1 hyperparameters (learning rate, number of training samples) as a function of the number of training samples from which we make the following observations. The best test accuracy for given set of hyperparameters is typically achieved after Slingshot phase begins during optimization. The checkpoints that achieve the highest test accuracy are labeled as "post-slingshot" and shown in green in Figure 6 While post-Slingshot checkpoints seem to enjoy higher test accuracy, there are certain combinations of hyperparameters that lead to models that show better test accuracy prior to the start of the first Slingshot phase. We label these points as "pre-slingshot" (shown in blue) in Figure 6 The above observations appear to be consistent with our finding that training long periods of time may lead to better generalization seen with grokking datasets (Power et al., 2021
shows a plot of the highest. test accuracy for a set of
Non- Transformer models with subset parity We train MLPs with Adam on the (n, subset parity lask. This family of lasks is notoriously challenging since it poses strict computational lower bounds on learning (see (Barak et al. 2022) for more details). We refer the reader to Appendix(A.6 A.6 for a full description of the datasot and the model used in this sctup. We observe multiple Slingshot Effects in Figure [3] Additionally, we observe an improvement in the test accuracy around the vicinity of each Slingshot effect until the loss stops changing deep into optimization. We empirically study the dynamics of the effective step size and
8

---

Published in Transactions on Machine Learning Research (02/2024)
10
c
10
train
1xtt lyee narm
tran
byernomm
10
10
4.36
107
27.00
1.75
20
0.50
10
M00 D27m
10
10"
10
10-1
10
9.08
2.25
10
3.50
10000
50300
Spech
epoch
ssech
(a)
(b)
(c)
training loss vs epochs
calicatooo on boss
bait HgTeT nom
valicttinn 1ox
lndd upu nom
2.50
0.55
107
1.75
1.50
6X.10
0.66
laget
3x.10P
2.25
10-1
0.06
3.00
10600
opocs
(d)
(e)
(f)
validation loss vs epochs
art syer 10mm
accuaay
layer
tran arcurnty
bet aneernom
100
1.0
00
1.50
03
1911 Sye ALT
60
60
0.75
20
0.36
10000
20075
70006
10000
10330
15050
20505
35000
40000
50350
0.
T0026
7003:
10000
40030
10500
(g)
(h)
(i)
training accuracy vs epochs
QUAMFACY
kyer
vulidation eccurac
uyernom
vDl soddo acoraay
los layo nom
100
2.00
0.26
1.75
3.50
60-
0.8
3.00
118 Wi
03
4.20
0.23
0.2
0.00
20505
as0oc
(k)
(I)
validation accuracy epochs
Figure 4: Varying in Adam on the Division dataset. Observe that as increases, there is no Slingshot Effect or grokking behavior. Figure (a) correspouds to default suggested (Kingua & Ba, 2014) where the model trained with smallest value undergoes multiple Slingshot cycles.
curvature in Appendix A.6.1 to understand what occurs during a phase transition from an optimization perspective.

---

Published in Transactions on Machine Learning Research (02/2024)
Slingshot Effects, no grakking
Slingshot no grokking
Slingshot grokking
100
c-10.08
100
z=1e07
accuracy
k=1e:05
100
1.00
E-1e-04
addition_ 50p
addition 60p
addition 70p
cutspoly Sap
cubapplyy Bop
cutepply 70p
cuberplyy Rop
divsion, 50p
division 50p
mullppixttion 50p
muliplicatton 60p
squarepialy S0p
Subtraction 50p
subtraction 70p
subtraction sop
Evanaddoiiirr add subtractionn
division
squarepoly
Figure Extended analysis on multiple grokking datasets. Points shown in green represent both Slingshot Effects and grokking, points shown blue indicate Slingshot Effects but not grokking while points in red indicate Slingshot Effects and no grokking. in Adam is varied as shown in text. Observe that as increases, there are no Slingshot. Effects grokking behavior.
Non-Transformer Models with synthetic data We conduct experiments with MLPs on synthetic data where the synthetic data is a low dimensional embedding projected to higher dimensions via random projections. With this dataset, we show that generalization occurs late training with Adam. Specifically, we tune e in Adam and show that the optimizer is highly sensitive this hyperparameter. These obscrvations are consistent with the behavior reported above with Transformers and algorithmic datasets as well standard vision benchmark such as CIFAR-10. We refer the reader to Appendix A.9 for complete description and details of these experiments.
3.4 Drawbacks and Limitations
While the Slingshot Effect exposes an interesting implicit bias of Adam that often promotes generalization, due to its arresting of the norm growth and ensuing feature learning, it also leads to some training instability and prolonged training time. In the Appendix we show that it is possible achieve similar levels of generalization with Adam on the modular division dataset (Power et al. 2021) using the same Transformer setup as above, while maintaining stable Icarning, in regimes that do not show clear Slingshot Effect. First we employ weight decay, which causes the training loss values to converge lo a higher value than the unregularized model. In this regime the model does not become unstable, but instead regularization leads to comparable generalization, and much more quickly. However, it is important to tune the regularization strength appropriately. Similarly, we find that it is possible normalize the features and weights using the following scheme explicitly control norm growth: w = |T f(x) = I where w and f(z) are the weights and inputs to the classification layer respectively, the norm used above is the L2 norm, and v is the input to the neural network. This scheme also results in stable training and similar levels generalization. In all cases the effects rely on keeping the weight norms from growing uncontrollably, which may be the most important factor for improving generalization. These results suggest that while the Slingshot Effect may be an interesting ceff-correcting scheme for controlling norm growth, there are likely more efficient ways to leverage adaptive optimizers to
20
f(z)
10

---

Published in Transactions on Machine Learning Research (02/2024)
1c
slingshot pre-slingshot
30
75
post-slingshot
3edg
1o.03
test accuracy
70
11-03
65
1e-34
1e-03
3c 96
11 03
60
55 3e-0
10 15
20
25
30
35
40
45
number of training samples (x1000)
Figure 6: Slingshot Effects subsets of CIFAR-10 dataset. We train ViIs with multiple learning rates to verify the impact this parameter has on Slingshot. Power et al. (2021) note that grokking occurs over narrow range of learming rates. Note that the points marked in: (i) green correspond to test accuracy for experiment after the Slingshot Effect begins, (ii) blue are for trials where best checkpoint is observed prior to start of Slingshot Effect and (iii) red for trials with Slingshot Effect..
similar levels generalization without requiring the instability that is a hallmark the Slingshot effect. Finally, we lack a satisfactory theoretical explanation for the Slingshot Effect, and hence removed all attempts more rigorous mathematical definition. It an open problem formally define and mathematically analyze the mechanism behind Slingshot Effects.
4 Conclusion
We have empirically shown that optimizing deep networks with cross entropy loss and adaptive optimizers produces the Slingshot Effect, curious optimization anomaly unlike anything described in the literature. We have provided ample evidence that Slingshot Effects can be observed with different neural architectures and datasets. Furthermore, we find that Grokking (Power et al. 2021 almost always occurs in the presence of Slingshot Effects and associated regions of instability in the Terminal Phase of Training (TPT). These results in their pure form absent explicit regularization, reveal an intriguing inductive bias of Adam-family of optimizers that bocomes salient the TPT, characterized by cyclic stepwise effects on the optimization trajectory. These effects often promote generalization in ways that differ from non-adaptive optimizers like SGD, and warrant further study to be able harness efficiently. There are open question remaining to be answered, for instance 1) What is the causal factor for the weight norm exit rapid growth and enter curtailed growth phase? 2) Are there better ways of promoting generalization without relying this accidental training instability? Answering these questions will allow us to decouple optimization and regularization, and ultimately to control and improve them independently.
Acknowledgments
We thank Preetum Nakkiran for insightful feedback and discussions and the Apple Machine Learning Research team for supporting and engaging with this work.
References
Kwangjun Ahn, Jingzhao Zhang, and Suvrit Sra. Understanding the unstable convergence of gradient descent. Xix preprint ar orXiv:2204.011 01050, 2022.
11

---

Published in Transactions on Machine Learning Research (02/2024)
Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via over- parameterization. Arii abs/1811.03000 2019.
Sanjeev Arora, Zhiyuan Li, and Abhishek Panigrahi. Understanding gradient descent on edge of stability in doep learning. arxiv preprint XXw:22255009755 2022.
Boaz Barak, Benjamin L. Edelman, Surbhi Goel, Sham M. Kakade, eran malach, and Cyril Zhang. Hidden progress in deep learning: SGD Iearns parities near the computational limit. In Alice H. Oh, Alckh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022.
URL httts::/
Anas Barakat and Pascal Bianchi. Convergence and dynamical behavior of the adam algorithm for nonconvex stochastic optimization. SIAM J. Optim., 31:244-274, 2021.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Miilne and Qiao Zhang. JAX: composable transformations of Python+NummPy programs. 2018. URL http::/github: com/ foom/goole//ax
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphnee Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowyez, Erica Morcira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Calasla, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways, 2022.
Jeremy M. Cohen, Simran Kaur, Yuanzhi Li, Zico Kolter, and Ameet Talwalkar. Gradient descent on neural networks typically occurs at the edge of stability. arXiv preprint ar Arziv-2103.000 2021.
Jeremy M. Cohen, Behrooz Ghorbani, Shankar Krishnan, Naman Agarwal, Sourabh Medapati, Michal Badura, Daniel Suo, David Cardoze, Zachary Nado, George E. Dahl, and Justin Gilmer. Adaptive gradient methods at the edge of stability. arXiv preprint Xiv: Arcio-2207 14484, 2022.
Adrien Courtois, Damien Scieur, MicheMichel Morel, Pablo Arias, and Thomas Eboli. Sing: A plug- and-pay dnn learning tochnique, 2023.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv: Arrio-2010.1190 11999, 2020.
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12(11)22111-2159, 2011. URL http
//jmlrrorg/papersssee. http:///ppp/////////aaaaaaaa html
Elad Hoffer, Itay Hubara, and Daniel Soudry. Train longer, generalize better: closing the generalization gap in large batch training of neural networks. ArXiv, abs/1705.08741, 2017.
Sergey Toffe and Christian Szegedy. Batch normalisation: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448 456. PMLR, 2015.
Jeevesh Juneja, Rachit Bansal, Kyunghyun Cho, João Sedoc, and Naomi Saphra. Lincar connectivity reveals generalization strategics. In The Eleventh International Conference on Learning Representations, 2023.
URL http:///gggaaeeeeeeeeee/o//nnnaa
12

---

Published in Transactions on Machine Learning Research (02/2024)
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint or orx: 1rriiu-112.00 6980, 2014.
Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.
Aitor Lewkowycz, Yasaman Bahri, Ethan Dyer, Jascha Sohl-Dickstein, and Guy Gur-Ari. The large learning rate phase of deep learning: the catapult mechanism. arxiv preprint orXi:22003.02116 2020.
Wenjie Li, Zhaoyang Zhang, Xinjiang Wang, and Ping Luo. Adax: Adaptive gradient descent with exponential long term memory. arXiv preprint or orxi::2004.000 09740, 2020.
Ziming Liu, Ouail Kitouni, Niklas Nolte, Eric J Michaud, Max Tegmark, and Mike Williams. Towards understanding grokking: An effective theory of representation learning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022.
URL https::/ .ntt://oorumm7ddd6666633331
Ziming Liu, Eric J Michaud, and Max Tegmark. Omnigrok: Grokking beyond algorithmic data. In The Eleventh International Conference on Learning Representations, 2023. URL forum?id=zDiHo1WWaay
Ekaterina Lobacheva, Maxim Kodryan, Nadezhda Chirkova, Andrey Malinin, and Dmitry P Vetrov. On the periodic behavior of neural network training with batch normalization and weight de- cay. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vanghan (eds.), Advances in Newral Information Processing Systems, volume 34, pp. 21545-21550 Cur- Associates, Inc., 2021. URL https://proceedngss.ee wwt.pppccccppppprrrrrroc. wttpppperftlee/eeeeeeeeeeeeeeeee Eille//paper/2021/51l// .ddp
Ilya Loshchilov and Frank Hutter. Decoupled weight docay regularization. arXiv preprint crXi::1711.05101, 2017.
Ekdeep Singh Lubana, Eric Bigelow, Robert P. Dick, David Scott Krueger, and Hidenori Tanaka. Mechanistic mode connectivity, In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (ods.), International Conference on Machine Learning, ICML 2029, 23- 29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 22965 23004. PMTR, 2023. URL htt.::ttp/s///t httttp:://proceednggs mll. www.ppresssssse presss/v202/uuaaa222.3 html
Liangehen Luo, Yuanhao Xiong, Yan Liu, and Xu Sun. Adaptive gradient methods with dynamic bound of learning rate. arXiv preprint arXiv:1903 09843, 2019.
Kaifeng Lyu and Jian Li. Gradient descent maximizes the margin of homogeneous neural networks. arXiv preprint Xiv: 05890, 2019.
Igor Molybog, Peter Albert, Moya Chen, Zachary DeVito, David Esiobu, Naman Goyal, Punit Singh Koura, Sharan Narang, Andrew Poulton, Ruan Silva, Binh Tang, Diana Liskovich, Puxin Xu, Yuchen Zhang, Melanie Kambadur, Stephen Roller, and Susan Zhang. A theory on adam instability in large-scale machine learning, 2023.
Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretabilltyy In The Eleventh International Conference on Learning Representations, 2023. URL
Vardan Papyan, X. Y. Han, and David L. Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences of the United States of America, 117:24652 24663, 2020.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach,
13

---

Published in Transactions on Machine Learning Research (02/2024)
H. Larochelle, A. Beygelzimer, F. d' d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Informa- lion Processing Systems 32, pp. 8024 8035. Curran Associates, Inc., 2019. URL. http::t/pppprsssss///////////s neur 1ps
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapcau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825 2830, 2011.
Alcthca Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Generalization beyond overfitting on small algorithmic datasets. In ICLR MATH-AT Workshop, 2021.
Sashank J. Reddi, Satyen Kale, and Sanjiv Kumar. On the convergence of adam and beyönd. ArXie, abs/1904.09237, 2018.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arxiv: Arzic- 140io-1409.156, 2014.
Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya. Gunasekar, and Nathan Srebro. The implicit bias of gradient descent separable data. The Journal of Machine Learnèng Research, 19(1):2822-25) 28778, 2018.
Tijmen Tieleman and Geoffrey Hinton. Lecture 6.5-rmspro,, coursera: Neural networks for machine learning. University of Toronto, Technical Report, 6, 2012.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkorcit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Atlention is all you need. arXiv preprinl arXiv: Arriv- 10riv-1706.000 03762, 2017.
Bohan Wang, Qi Meng, Wei Chen, and Tic-Yan Liu. The implicit bias for adaptive optimization algorithms on homogeneous neural networks. In International Conference on Machine Learning, pp. 10849-10858. PMLR, 2021.
Ross Wightman, Hugo Touvron, and Hervé Jégou. Resnet strikes back: An improved training procedure in timm. arXio preprint or 1rx::21100000 2021.
Mitchell Wortsman, Tim Dettmers, Luke Zettlemoyer, Ari Morcos, Ali Farhadi, and Ludwig Schmidt. Stable and low-precision training for large-scale vision-language models, 2023.
J. Zhang, Tianxing Hc, Suvrit Sra, and Ali Jadbabaic. Why gradient clipping acceleratos training: A theoretical justification for adaptivity. arXiv: Optimization and Control, 2020.
Yushun Zhang, Congliang Chen, Naichen Shi, Ruoyu Sun, and Zhi-Quan Luo. Adam can converge without any modification update rules. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and
A. Oh (eds.). Advances in Newral Information Processing Systems, volume 35, pp. 28386-28399. Cur- ran Associates, Inc., 2022. URL
wwwp:eennnnrnnrceieeeeeee caaeerrffeesppppppe /illss/paper/202//111
Libin Zhu, Chaoyue Liu, Adityanarayanan Radhakrishnan, and Mikhail Belkin. Catapults in sgd: spikes in the training loss and their impact on generalization through feature learning, 2023.
Juntang Zhuang, Tommy Tang, Yifan Ding, Sekhar C Tatikonda, Nicha Dvornek, Xenophon Papademetris, and James Duncan. Adabelief optimizer: Adapting stepsizes by the belief in observed gradients. Advances in neural information processing systems, 33:18795-18806, 2020.
14

---

Published in Transactions on Machine Learning Research (02/2024)
The Slingshot Effect: A Late-Stage Optimization Anomaly in Adam-Fanily of Optimization Methods Appendix
Contents
Introduction
1
Implications of Our Findings
3
Related Work
4
The Slingshot Effect
5
Experimental Sctup
5
3.2
Experimental Observations
5
3.3 3
Fffects on Generalization
8
Drawbacks and Limitations
10
Conclusion
11
Appendix A Slingshot Effects across Architectures, Optimizers and Datasets
17
Vision Transformers on 1000 samples from CIFAR-10
17
A.2
CNN on 200 samples from CIFAR-10
17
MLPs on 200 samples from CIFAR-10
Shallow models
19
A.5
Deep linear models
20
A.6
Learning Subset Paritics
21
A.6.1
Eflective Step Size and Curval ure Dynamics
21
A.7
Different Optimizers
23
Vision Transtormers and Full CIFAR-10
24
A.9
Slingshot with MLP and Synthetic Dataset
24
A.9.1 Ablation Study
24
Appendix B Slingshot and Grokking
26
B.1
Analysis of Parameter Dynamics
27
................................
B.2 SGD Optimization
28
B.3 Tuning Adam Optimizer's s 32 Hyperparameter
28
B.4
Slingshots with Additional Datasets
28
Appendix C Controlling Instability Through Normalization and Norm Constraints
44
C.1
Weight decay
44
15

---

Published in Transactions on Machine Learning Research (02/2024)
C.2 Features and parameter normalization
47
Appendix D Slingshot Effects and Linear Mode Connectivity
51
16

---

Published in Transactions on Machine Learning Research (02/2024)
train loss
last layer norm
train acc
last layer norm
0.20
0.20
10
99.75
0.16
0.18
99.50
10
last accuracy
103
0.16
99.25
10-9
99.00
0.14
98.75
0.12
98.50
98.25
1000
1200
1400
1600
1800
2000
1000
1200
1400
1600
1800
2000
epech
opech
(a) Norm growth versus loss on training data
(b) Norm growth versus accuracy on training data
Figure 7: Vision Transformer trained on 1000 samples from CIFAR-10.
A Slingshot Effects across Architectures, Optimizers and Datasets
This section provides further evidence of the prevalence of Slingshot across architectures and optimizers subsets of CIFAR-10, testing setups beyond the specific setup consider by Power et al. (2021). In these experiments, we focus solely on characterizing the optimization properties of various setups described below. The small sample sizes are used in order to more easily find regimes where different architectures can converge to fit the training data fairly quickly.
We use cross-entropy loss to optimize the models with Adam (Kingma & Ba 2014 in the following experiments. The following experiments are implemented in PyTorch Paazkee et al. 2019
A.1 Vision Transformers on 1000 samples from CIFAR-10
For further validation, we train a Vision Transformer (ViT) Dosovitskiy et al. 2020 with 12 layers that has 10 million parameters
small sample of the CIFAR-10 dataset (Krizhevsky 2009: In this setup, we learning rate to 0.001, no weight decay, 31 0.9, $2 = 0.95, = 10- and minibatch size of 128. We choose sample size of 1000 training samples for compulational reasons, as wish to observe multiple cycles of the Slingshot Effect extremely late in training. The input images are standardized to be in the range [O, 1]. No data augmentation used in our training pipcline. Due to the extremely small sample size, we focus our attention on the training metrics since no generalization is expected. Figure 7a (respectively Figure 7b) shows a plot of training loss (respectively training accuracy) and last layer norm evolution during the latter stages of training. Multiple Slingshot stages are observed in these plots (5 clear eycles), which can be seen by the sharp transition of the weight norm from high growth to curtailed growth phase.
A.2 CNN on 200 samples from CIFAR- 10
We consider a VGG-like architecture (Simonyan & Zisserman 2014) that has been adapted for CIFAR-10 dataset P The model is trained with 200 randomly chosen samples from CIFAR-10 training split and with
full-batch Adam (Kingıa & Ba 2014
Loshchilov & Hutter 2017 The hyperparameters used for the
optimizer include learning rate of 0.001, weight decay= 0, B1 = 0.9, B2 = 0.95, and c = 10-8, As with VIT, no data augmentation is used in these experiments other than standardizing the input. to be in the range
[o, 1]. We observe the presence of multiple Slingshot stages with CNN from Figure Sa and Figure Sb These experiments suggest that Slingshot effect is not restricted to Transformers architecture alone.
We the VGG11 architecture without batch normalization Tofff & Szegedy 2015 from http:://ggttuub.comkkuass pytorch-cifar in this experiment.
17

---

Published in Transactions on Machine Learning Research (02/2024)
layer norm
train acc
last: layer norm
10a
100
100
80
60
last layer
40
20
1000
2000
3000
4000
5000
1000
2000
3000
4000
5000
opach
(a) Norm growth loss training data
(b) Norm growth accuracy on training data
Figure 8: CNN without batch nomralization trained on 200 samples from CIFAR-10.
train in loss
last layer norm
acc
layer norm
10
1200
100
2
1000
80
1000
800
accuracy
10
60
last
40
last
10-4
200
20
1000
2000
3000
4000
5000
epoch
epoch
(a) Norm growth versus loss on training data
(b) Norm growth versus accuracy on training data
Figure 9: CNN with batch nomralization trained 200 samples from CIFAR-10.
With BatchNorm We repeat the CNN-based described above but with a VGG-like model that includes batch normalization (loffe & Szegedy 2015 The training setup is identical to the one described for CNN wihtout batch normalization. We observe the presence of multiple Slingshot stages with CNN from Figure Ya and Figure 9b. The weight norm does not decrease during training opposed to the weight norm dynamics for CNN without batch normalization seen in Figure These experiments suggest that Slingshot Effects can be seen with standard neural network training components including batch normalization.
A.3 MLPs on 200 samples from CIFAR-10
The next architecture we consider is deep (6 layers) fully connected network trained on a small sample of 200 samples belonging to the CIFAR-10 dataset (Krizhevsky 2009) with full-batch Adam (Kngmma & Ba 2014 Loshchilov & Hutter 2017) imizer The optimizer's 's hyperparameters are set as following: learning 0.001, weight decay 0, B 0.9, 32 0.95, and e 10 8 As with the ViT setup above do no use data augmentation for training this model. Figure 10a (respectively Figure 10b] shows plot of training loss (respectively training accuracy) and last layer norm evolution during the latter stages of training. Multiple Slingshot stages are observed in this setup well. These experiments further suggest that the Slingshot. Effect is prevalent in simple models as well.
We the VGG11 architecture with batch normalization [Toffe & Szegedy 2015 from https://gsthhb. http://sthubb.coom/uuan.g.. pytorch-cifar in this experiment.
18

---

Published in Transactions on Machine Learning Research (02/2024)
train loss
last layer norm
100.0
10P
0.0785
10
95.0
acourscy
0.0780
10
loyer
0.0775
90.0
10.6
0.0770
10.9
85.0
0.0765
0.0765
1000
1800
1000
epoch
apoch
Norm growth loss training data
Norm growth accuracy training data
Figure 10: MI.P trained on 200 samples from CIFAR-10.
3000
70-1
100
2 20--1
20-1
200
1007
20003
30230
40000
3002
02000
(a) Training loss for 1 layer model
(b) Training loss for 2 layer model
(c) Training loss for layer model
2000
100
400
200
(d) Training accuracy for layer model
Training accuracy for layer model
(f) Training accuracy for layer model
Figure 11: Effect of depth on Slingshots. Norm growth behavior versus training metrics for models whose depth is indicated above. All models are trained with all-hatch Adam with learning rate 0.001 on 200 CIFAR-10 samples.
A.4 Shallow models
We consider the behavior of shallow models including linear, 2- and 3-layer MLPs with Adam optimizer.
As with the previous setup, train these models small sample of 200 samples belonging to the CIFAR-10 dataset [Krizhevsky] 2009) with full-batch Adam (Kingma & Ba 2014 optimizer. The optimizer's
hyperparameters are set as following: learning rate = 0.001, weight decay = 0. 81 = 0.9, 3g = 0.95, and
= 10-8 No data augmentation is used in these experiments as well. Figure 11a Figure 11b Figure H1
show the training loss and last layer norm evolution during training for the linear, 2-layer and 3-layer models respectively while Figure Figure Tic IT] the training accuracy and last layer norm evolution. Slingshot Effects are observed in 2-layer and 3-leyer MLPs whereas no Slingshot Effects are seen with the linear model. These experiments suggest that depth appears to be necessary condition observe Slingshots.
19

---

Published in Transactions on Machine Learning Research (02/2024)
Dan los
lax layst norm
Brin Mod
laxt layst rorm
10"1
1
10-1
10-4
10
12-4
2503
5000
7500
2000
12500
15000
37590 20000
2502
5000
1900 15000
12503 15M30
175.00 200000
2507 3902 7530 1000 12500 15590 17500 2000)
(a) Training loss Adam
(b) Training loss for RMSProp
(c) Training loss for SGD
crai acc
120 layor norm
COm D0c
lst layer norm)
100
100
aut laperr romm
80
20
2500 3000 1900 10300
115.0 15002 17500 23020 esech
2500 5000 7500 10593
12502 15920 17500 20072
2000 5020 7302 10300 esech
12595 15000 17520 20393
(d) Training accuracy for Adam
(e) Training accuracy for RMSProp
(f) TTriiing accuracy for SGD
Figure 12: Optimizer choice and Slingshots. All optimizers train a 6-layer linear model full-batch on 200 CIFAR-10 samples.
A.5 Deep linear models
We train a layer lincar model with 200 samples belonging to CIFAR-10 (rizhovsky 2009) with full-batch Adam KKnggma & Ba 2014 Loshchilov Hutter 2017]. The optimizer's hyperparameters are set as following: learning rate 0.001, weight decay 81 0.9, B2 0.95, and 10 6 Figure 12a and Figure 12d show the training loss and accuracy behavior observed during optimization. Multiple Slingshot stages are observed with this architecture well.
20

---

Published in Transactions on Machine Learning Research (02/2024)
A.6 Learning Subset Parities
In this section we use the k-sparse parilies of n bits lask as a test bed. Theoretically, this family of tasks is notoriously challenging since it poses strict computational lower bounds on learning (see (Barak et al.) 2022) for more details). For the (n, k) subset parity task, cach input is a random n dimensional voctor such that each component is randomly sampled from ~ Unif{-1,11 The label is then given by a parity function over a predefined sparse set of < n bits. For the following experiments, we use = 3, n = 50. For the model, we use a 3 layer MLP with relu activations, and the cross entropy loss. We use a dataset of 1000 samples, and test set of 8000 samples. We train each network with Adam using batch size of 32, learning rate of 1 {0.004, 0.003, 0.002) and e E {10-8, 10-2, 10- Our results are summarized in Figures 13 14 For c = 10-8, multiple Slingshots appear past the perfoct fitting of the training set, with a bump in generalization lization post most Slingshots. For larger values of no Slingshots are observed, while generalization
and 15
remains poor.
1.0
90
90
90
70
70
70
50
4obo
6000
8000 spach
Loc00 12000 14000
50
2000
6000
80b0
12000 14000
50
2000
6000
8000
12000
14000
(a)
(b)
train
tost
test
test
10a
2.01
10
123
10
100
30-4
30-5
10
30-
10-9
10-1
2000
4000
6000
8000
10000
22000 14000
2000 4000
6000
B000
10000
12000 14000
0
2000
4000
6.00
6000
12000
12000 14000
opoch
60och
650ch
(d)
(e)
(f)
Figure 13: Learning a (3, 50) subset parity with Adam with e = 10-8 and a learning rate of (a).(d) r == 0.004, (b).(e)
1) 0.003 and (f), 0.002. Multiple Slingshots are visible, resulting in improved generalization.
A.6.1 Effective Step Size and Curvature Dynamics
A classical results pertaining to optimizing smooth functions with gradient descent states that sufficient. condition for convergence requires that the learning rate does not exceed 3, where L is the Lipschitz constant of the gradient. Due to the sufficiency of the condition, we expect it to be violated at the phase transitions of the slingshots, when the training loss spikes. We quantify the effective step size of parameter VV2+t
where the terms are defined in Algorithm 1. To approximate L in a local region, we use the maximum eigenvalue of the loss Hessian in this analysis as is done by series of recent works including Cohen et al. (2021), Ahn et al. (0022) and Arora et al. (2022). We use the same setup described for training parity dataset to conduct this empirical analysis. The hyperparametors used for the optimizer include 17 = 0.004, 10-8 and 3, = 0.9 and B2- = 0.999. Figure 16a shows the dynamics of the training and validation loss while Figures 16b Figure [6c and Figure [6d shows the evolution of the effective step size as well as the maximum allowable step size for a few paramcters chosen randomly from the three layers the neural network. We observe from these plots that the effective step size is smaller than the maximum allowed step size in the vicinity of ShingShot Effeetts. however, at the phase transitions clearly see that the effective step
21

---

Published in Transactions on Machine Learning Research (02/2024)
40
90
90
70
70
70
50
50
50
6000
8000
10000
12000
6000
8000
12000
14000
2000
8200
14000
(a)
(b)
(c)
test
167
1.0
1.0
10-3
10-1
10-1
30-3
1.-
10-3
10
10
10
111
110010110
2000
4000
6000 8300
10000
12000 14000
2000 4000
5000
8000
10000 12000 14000
2000
4000
6000
6000
10000
12000 14000
(d)
(e)
(f)
Figure 14: Learning a (3, 50) subset parily with Adam with e 10-7 and a learning rate of (a).(d) rj = 0.004, (b).(e)
1) 0.003 and (f), 0.002. No Slingshots are visible.
90
30
90
70
X12
70
50
50
50
4000
6nbe
10000 12000
12000
n003 epech
1050n
14000
spech
opoch
(b)
train
test
tra
test
train
test
30"h
30"4
10-1
20
200
-0-
10-4
10-1
10-5
2000
4000
8000 10000 12000 14200
2000 4000
6200
8000 15000 12500 14500 coach (c)
2000
4000
6200
8000
13000 12000 14300
(d)
(f)
Figure 15: Learning (3, 50) subset parity with Adam with 1 and a learning rate of (a), rj 0.004, (b).
0.003 and (f), 0.002. No Slingshots are visible.
size is larger than the maximum allowed, causing the loss to spike. After a few Slingshot cycles, we observe that the maximum allowed step size increase dramatically, and additional Slingshots follow.
22

---

Published in Transactions on Machine Learning Research (02/2024)
maximum stepsize
effective step size (param 10)
trainn loss
validation loss
effective stop size (param a)
effective step size (param 20)
10
10
1012
10a
10a
10a
10-7
10a
0
2000
4000
6000
8000 epoch
10000
12000
14000
2000
4000
6000
8000
10000
12000
14000
epoch
(a)
(b)
maximum stepsize ettective
effective step size (param 1 10000)
ettective 33000)
affective step size (param 15200)
affective step size (param 15260)
103h
10
10a
10P
10a
10a
10a
10a
6000 12000 14000
epoch
(d)
Figure 16: Empirical analysis of the relationship between Slingshot Effeett and loss surface sharpness, Above plots include (a) training and validation loss; evolution of effective step size and curvature of parameters from (b) first layer, (c) second layer and (d) classification layer in a 3-layer MLP trained with Adam. At the phase transitions, effective step size is larger than 3. initiating the slingshots. After few cycles, the Lipschitz constant of the gradients decreases substantially, and the Slingshots cease.
Table Optimizers hyperparameters. Learning rate 0.001 and weight decay for optimizers
Optimizer
Other hyperparameters
Adam
31 = 0.9, 3, 0.95
RMSProp
a= 0.95, momentum-0.0 momentum=0.9
GD
A.7 Different Optimizers
In this set of experiments, study the training loss behavior of deep linear models optimized full-batch with Adam (Kingma & Ba 2014 Loshchilov & Hutter 2017). RMSProp TTellmman & Hinton 2012 and full-batch gradient descent (GD). The six layer model is trained with 200 samples. The hyperparameters used for optimizing the model with various optimizers are described in Table Figure 12 shows the training loss and accuracy behavior of the three optimizers considered in this experiment. We observe Slingshot behavior with Adam and RMSProp from Figure 12 while Slingshot behavior absent with standard gradient descent. This observation suggests that the normalization used in adaptive optimizers calculate the update from gradients may lead to Slingshot behavior.
23

---

Published in Transactions on Machine Learning Research (02/2024)
A.8 Vision Transformers and Full -0FAR-10
In Appendix A] we have empirically shown that the existence of the Slingshot phenomenon on small subset of CIFAR-10 dataset (Krizhevsky 2009) with Vision Transformers (ViTs). We now study the impact that Slingshot has on the gencralization ability of ViTs by training a model on all 50000 samples in CIFAR-10 training dataset. The ViT used here a larger model than the one considered in A to account for larger dataset size. The ViT model consists of 12 layers, width 384 and 12 attention heads and is optimized by Adam (Kingma & Ba 2014 Loshchilov & Hutter 2017). For this experiment, we set the learning rate 0.0001, weight decay to 0, 31 = 0.9, 31 = 0.95 and <= 10-8, minibatch size of 512 and linear learning rate warmup for 1 epoch of optimization. Figure 17 shows the results of experiment with full CIFAR-10 dataset. Multiple Slingshots can be observed in these plots similar to the plots described in Appendix A We observe from Figure [7d] that the test accuracy peaks in epochs following Slingshot with the maximumn recorded test accuracy occurring very late in optimization. This observation suggests that the Slingshot can have a favorable effect on generalization consistent with the behavior observed in the main paper with division dataset.
layer norm
train acc
layer norm
1.6
100
10-1
90
1.4
10-3
80
10-3
socursy
70
1.0
0.0
60
10-7
0.4
0.4
0.2
10-41
1.00
0.0
6000
8000
2000
4000
6000
8000
10000
(a)
(b)
loss
test acc
last layer norm
1.6
102
14
1.4
1.2
65
1.2
1ast layer north
1.0
atcuracy
1053
0.8
60
0.8
40.6
0.6
0..4
45
10a
0.2
0.0
2000
4000
6000
8000
10000
2000
4000
6000
8000
10000
epoch
epoch
(c)
(d)
Figure 17: Slingshot generalization full CIFAR-10 dataset: Norm growth versus a) loss training data b) accuracy on training data (c) loss on lest data d) accuracy on test. data.
A.9 Slingshot with MLP and Synthetic Dataset
In this section, we provide empirical evidence that Slingshot Effects are observed with a synthetic dataset. in fully-connected architecture. The small dimensional dataset, like the Grokking dataset of Power et al. (2021), allows us to easily measure of sharpness, given by Tw] Heu, where u, is the optimizer's update vector and H, the Hessian at step f, to examine the interplay between Slingshot Effects and generalization.
A.9.1 Ablation Study
In this section, we train toy model on a synthetically generated dataset with the aim of analysing the effect of different hyper parameters on the Slingshot Effect. We construct. 128-dimensional dataset with
24

---

Published in Transactions on Machine Learning Research (02/2024)
nain looe
bdt ayerncm
graclett Sharpees
bet 9om
26.0
250
26.6
175
0.0
15.0
150
15.0
10
12.5
12.
10'
12.5
100
10
1.3
7.5
10-
10
25
0.5
3.0
0.
15
165
(scc)
1w0
opech
14c
(a)
(c)
val 153
last layer norm
1000
0.0
val aCC
lust layer nom
spdate tharorast
byee nom
10
175
175
17.5
15.0
110
10-
12.5
0.5
125
suppree
10-
881 aatleg
10.0
0.4
150
10-3
5.0
5.0
0.0
05
1.0
1.5
2.0
0.0
0.
1.3
35
7.0
255
5.5
1.0
(b)
(d)
(f)
Figure 18: Slingshot generalization on synthetic dataset: Norm growth versus a) loss on training data b) accuracy on training data (c) loss on validation data d) accuracy validation data. Note that the vertical line in green shows location maximum test accuracy. Adam hyperparameters are 81 = 0.9, 31 = 0.95, e 10-8
Scikit-learn (Pedregosa et al. 2011) that has 3 informative dimensions that represents 8-class classification problem. The class centers the edges of 3-dimensional hypercube around which clusters are data are sampled from standard normal distribution. The other 125-dimensions also filled at random to create a high-dimensional dataset used in our experiments. We generate 256 training and validation samples for this dataset and use a minibatch size of 128 in all the experiments described in the following.
A Architecture and Optimizer Figure 18 shows the training and validation metrics when we optimize a 4-layer fully-connected network (FCN) with Adam using a learning rate of 0.001, 31 = 0.9, 81 = 0.95, no weight docay and c = 10-8, Note that we use this value of in our first experiment as this is the default value proposed in Kingmn and Ba (Kingma & Ba 2014). These experiments are implemented in JAX (Bradbury al. 2018
Tuning In the next set of experiments with synthetic data, tune value for Adam to understand its impact test accuracy. Figure 19 shows plot of the maximum validation accuracy achieved by models trained with Adam as function of time (epoch). We observe that Adam reaches its best test accuracy late in optimization with e = 10-5 yielding the highest validation accuracy. Furthermore, the best. accuracy is achieved with a model that experiences Slingshot during optimization. This observation is consistent with our findings for VIT training with CIFAR-10 dataset described in the main paper and Appendix A.8
Influence of 81 and B2 In these experiments, we aim to study the impact of Adam optimizer's 31 and 32 hyperparameters on Slingshot. We use the synthetic data described above and set the learning rate of 0.001 and 10- for this analysis. Figure 20 and Figure 21 shows the results of this study. We observe from Figure 20 that the Slingshot Effect is fairly robust to the values of 81 and 83. Figure 20a Figure 20c show that Slingshot is even observed with Bn and Br sot to 0 which effectively disables exponential moving averaging of gradient moments in Adam (Kinguna & Ba 2014). Figure 20g Figure 200] provide an example of hyperparameters that fail to induce Slingshot. We observe from Figure 20 that models that experience Slingshot tend to reach their best test accuracy during the later stages training. Specifically, we observe from Figure 20b] Figure 20c and Figure 20k that the best validation accuracy occurs after 60000 epochs. These examples provide further evidence about an interesting implicit bias of Adam. Figure 21 shows more
25

---

Published in Transactions on Machine Learning Research (02/2024)
0.80
adam aps 1e405
0.78
dam 1e-04
adam 1c.06
adam eps 1e-08
validation accuracy
0.76
ødam eps 1e-02
adam eps 1e-07
0.74
0.72
0.70
0.68
adam eps 1e:03
104
105 epoch
106
Figure 19: Slingshot syntehtic dataset. Note that the points marked in: (i) green correspond to Adam-trained models that undergo Slingshot, (ii) red correspond to Adam-trained models that do not experience Slingshot; Adam's hyperparameters are given by 31 0.9, 0.95, no weight decay and shown in parentheses.
examples of hyperparameters that do not induce Slingshot Effects. Finally, we observe from Figure 21 that hyperparameters that provide higher validation accuracy are from models that experience Slingshot Effects.
B Slingshot and Grokking
We use the empirical setup described by Power et al. (2021) to describe the Slingshot Effect. The following section describes relevant details including datasets, architecture and optimizer used in our experiments.
Architecture The model used a decoder-only Transformer (Vaswani et al. 2017 with causal attention masking. The architecture used in all our experiments consists of 2 decoder layers with each layer of width 128 and attention heads.
Optimization We train the architecture described above with Adam optimizer (Kingma & Ba 2014 Loshchilov & Hutter (017)) in most of our experiments unless noted otherwise. The learning rate is set 0.001 and with linear learning rate warmup for the first 10 steps. We use 31 = 0.9, 82 0.98 for Adam's hyperparameters. The Transformers optimized with cross-entropy (CE) loss that is calculated the output tokens for given binary operation.
Algorithmic Datasets The Transformer is trained on small algorithmic datasets that consists of sequences that represent a mathematical operation. The following operations are used in our experiments:
+ (mod p) for 0 < a,6
c = a (mod p) for < a, <<p
(mod p) for 0 < a.b p
c = a + (mod p) for < a, <
a + b (mod p) for 0 < a.bpp <p
26

---

Published in Transactions on Machine Learning Research (02/2024)
a² +b (mod p) for 0 1 a,b <p
a² + b² (mod p) for 0 a. <p
a + 2 + ab (mod p) for a, <
a + 2 + ab + b (mod for 0 s a, <p
8² + ab (mod p) for 0 S a b <
8 at (mod p) for a, p
c b (mod p) if odd, otherwise a (mod p)] for S a. < p
c a b for a, E Ss
a b a- for a, b E Ss
= x a for a, Ss
c + (mod p) if a is even, otherwise a (mod p)] for 0 S a, <p
c = a + (mod p) if a is even, otherwise a b (mod p)] for 0 s a.b<p
where p 97 and with the dataset split in training and validation data. Each equation in the dataset is of the form (=) where (x) represents the token used to represent x. We refer to Power al. 2021 for detailed description of the datasets
B.1 Analysis of Parameter Dynamics
A common observation is that intermediate representations tend to evolve beyond simple scale increase during phase transitions from rapid norm growth phase to curtailed norm growth phase. In order to empirically quantify this effect, we train the Transformer described in Appendix B with modular addition, multiplication and division datasets using Adam with learning rate set to 0.001 and 31 0.9 and {3 0.98. We calculate the cosine distance between the representation and classification parameters from their initial values where the cosine distance is given by drepr 1.0
repr
repr W 1
WA
|
w el
w
d = 1.0
|
otf
1 cf 11
where drepr (d denotes cosine distance for representation (respectively classification) parameters, wi repr (resp. w denotes representation (resp. classification) parameters at time with wgerr (we²) indicating the initial representation (resp. classification) parameters where the norm used above is the Euclidean norm.
Figure 22 shows the dynamics of the loss, accuracy and cosine distance recorded during training. We observe that the classification parameters move farther away from initialization faster than the representation parameters. More interestingly, observe from Figure 22c and Figure 22 that the representation parameters travel farther from initialization for training runs that experience Slingshot. These trials use e 10 and - 10 and experience Slingshot Effects. In contrast, we see from Figure 22i and Figure 221 that the representation distance remains low for models trained with e = 10-5 and 10-4 The models trained with higher e values do not experience Slingshot Effects. These results suggest that Slingshot may have a beneficial effect in moving the representation parameters away from initialization which eventually helps with model generalization. Figure 23 and Figure 24 show a similar trend for multiplication and division datasets respectively.
27

---

Published in Transactions on Machine Learning Research (02/2024)
B.2 SGD Optimization
In this appendix, we show that Slingshot Effects are not seen during Transformer training with stochastic gradient descent (SGD) with momentum to support our claim in the main paper. To this end, we use train the Transformer described in Appondix B on modular division dataset with a 50/50 train/validation split using SGD with momentum. We use a mini-batch size 512 which requires the optimizer to take 10 steps per epoch for dataset split described above. We set um to 0.9 and use the following learning rates: 0.001, 0.01 and 0.1 and run the optimizer for 1500000 steps. The number of steps used here is 3 times larger than the steps used to run Adam in this work which is chosen to give SGD additional time to reach convergence. Figure 25 shows the usual loss and accuracy metries calculated on training and validation data as well as the weight norm of the classifier layer. We observe that there is no evidence of Slingshot with SGD. Lastly, we do not see any evidence of Grokking or generalization with this setup as well.
B.3 Tuning Adam Optimizer's 32 Hyperparameter
Zhang et al. (2022) showed in recent paper that vanilla Adam (Kingma & Ba 2014) can converege without any modifications to its update equations. A key message in (Zhang et al. 2022] is to tune the B hyperparamoters that are used to smooth momentum and second order moment terms in Adam (Kingma & Ba 2014) with a focus on tuning B2 which controls the denominator in the update equations. We study the behavior of Adam optimizer (Kingma & Ba, 2014) by training a Transformer described in Appendix B with modular division dataset split evenly into training and validation datasets. Figure 26 shows the behavior of optimizing Transformers with Adam for three values of Bg including 0.98 which is the default used both by Power el al. (2021) and in this paper well as very high B2 values including 0.9995 and 0.9997. We observe training instabilities and Slingshot Effects in all of these experiments from Figure 26 These observations suggest that Slingshots can occur during neural networks optimization despite carcful tuning of B2 hypermater.
B.4 Slingshots with Additional Datasets
In this appendix, we provide evidence of Slingshot Effects on additional datasets from Power et al. (022 Grokking work. The dalasets are created by a subset of mathematical operations defined in Appendix B Each operation can have multiple datasets that depends on the vaainvalidation split ratio. We use the training spt described in B on 18 separate datasets. Figure 27 Figure 44 shows the results the datasets described in this appendix. We observe Slingshot Effects and generalization with all 18 datasets. These results suggest the prevalence of Slingshot Effects when large models are trained with adaptive optimizers,
specifically Adam Kingma & Ba 2014)
28

---

Published in Transactions on Machine Learning Research (02/2024)
loss
accuracy
sharpness
1767
18s1 Myrt PATT
uodate
1.0
60000
EACT bayer 10mm
50600
0.6
10
40020
330000
10020
20030
10300
00000
00030
105630
20030
45000
00003
130000
21003
40000
50035
50003
100050
(a)
(b)
B1
0. 12 0. Observe multiple Slingshots
2221
layu
toan
19
last uyyr nort!
gF3tlmmt
1ptat>
byer nomm
120033
14
0..6
00032
10
20030
40300
50000
00000
102020
20020
40000
50000
80002
120000
20020
40035
60100
3C603
103000
(d)
(e)
(f)
31 0.5, A2 0.5. Observe multiple Slingshots
bun
last nomm
tain
last layer nor
gradlent
update
bast layer nom
1.6
13a
1.4
1.2
0.6
3.5
60003
03000
20000
40000
103030
20303
00702
00000
(h)
0.9, 83 0.8. Observe no Slingshot
byeer
bain
10
last layrr nor.
gradeent
upsate
ast aper norm
3.7
-12
0.0
40005
40032
20002
20030
40300
50000
$0000
103000
20030
41000
620000
80002
150000
70030
40032
60300
30003
103000
(j)
(k)
(1)
B1 = 0.9, = 0.95. Observe multiple Slingshots
Figure 20: Varying 81, B2 in Adam synthetic dataset. FCN is trained with Adam using learning rate 0.001 and
10 -8 The validation accuracy of models that experience Slingshot reach their highest accuracy later in training.
29

---

Published in Transactions on Machine Learning Research (02/2024)
0.999
70
0.95
60
0.8
0.65
50
0.5
B2
40
0.4
-3
0.3
20
0.2
0.1
10
0.0
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
B1
Figure 21: Extended analysis B1, O2 in Adam on synthetic dataset. Plot shows the highest validation accuracy
achieved with various values of B1, 32 with learning rate set 0.001 and 10-6 Hyperparameters that do not induce Slingshot Effects are marked with diagonal line in black. Models trained with 3 > 0.2 and 82 = diverged during training due instability. These trials have their validation accuracy set to chance level.
30

---

Published in Transactions on Machine Learning Research (02/2024)
loss
accuracy
cosine distance
valcotion
100
23
2.0
120
60
40
12
20
0.3
10-11
0.0
0.0
10:00
20300
30020
40000
50003
10000
20003
30d00
40000
50400
15000
20020
10002
00300
50070
(a)
(b)
(c)
epsilon 10
bain
vuldation
lost layer norm
Norractlaiion
lal koyor nom
10-1
80
1axt normh
60
layer
10
0.8
0.2
62
15650
28203
10450
A0500
50350
0.0
opact
4po55
620ch
(e)
(f)
epsilon = 10-7
vall idvionn
Hype 10mm
9.16
10'
0.14
5.4
6.14
C00 deacce
0.33
9.30
310
40
2.20
3.00
30-1
0.6
0304h
k20ch
(g)
(h)
epsilon = 10 5
Imin
val detion
Nyyer 1cmm
3.54
0.25
9.12
0.22
10-
0.25
2.2
20-4
3.6
9.06
3.00
0.06
Gauch
s20ch
(k)
(I)
epsilon = 10
Figure 22: Cosine distance evolution for Transformer described in Appendix B trained on modular addition. Observe that the cosine distance from initialization increases with models that experience Slingshot Effects.
31

---

Published in Transactions on Machine Learning Research (02/2024)
loss
accuracy
cosine distance
Dui
Sull2tiin
lapt norm
layer
100
101
80
0.0
10
-00
20
0.5
5.2
0.5
0.0
20200
20220
30020
40032
30003
13050
20000
30450
50-50
12000
20035
30002
40220
50020
(a)
(b)
(c)
epsilon = 10
Irah
volldcion
lust syar nom
last
rsposatiation
daafficatrr
lollaysramm
1.07
1.75
150
10
80
125
3.25
1.00
2.80
0.75
10
0.50
10
00
13250
20003
20320
#2000
50022
1000
7003
10000
00320
10:00
12000
22000
30020
(d)
(c)
(f)
epsilon = 10-7
last NEM
daaficater
0.34
3.12
012
30
60
3.30
12
WUMP
3.08
5.1
30-
3.56
0.06
0.06
00
13000
20000
20000
41003
52000
120.0
7002
100.00
00010
10.00
12000
22000
(g)
(h)
epsilon 10-5
last
dusffcater
0.39
30F
012
0.22
10-
60
0.0
20.4
0.00
0.0
12200
20300
10010
72003
73000
15650
20030
10000
40350
10000
12000
23000
50300
(i)
(k)
(1)
epsilon 10
Figure 23: Cosine distance evolution for Transformer described in Appendix trained modular multiplication. Observe that the cosine distance from initialization increases with models that experience Slingshot Effects.
32

---

Published in Transactions on Machine Learning Research (02/2024)
loss
accuracy
cosine distance
valcodaan
100
102
10
60-
400
20
0.3
10
0.0
0.0
10:00
20300
30020
40000
S0003
10000
20003
30d00
40000
50400
15000
20020
10002
00300
50070
(a)
(b)
(c)
epsilon 10
bran
vuldation
lost layer norm
Norractlaiion
lal koye nom
2.0
127
80
last POTTS
60
Tast lager
0.5
0.0
15650
28203
10450
A3500
50350
0.0
opact
4pa55
02ech
(e)
(E)
epsilon = 10
Tmin
vall devion
Myper 10mm
3.16
034
0.14
012
0.33
9.30
010
10
2.20
3.00
066
0324h
k20ch
(g)
(h)
epsilon = 10 5
Imin
vall detion
Nyyer 10mm
1..44
3.12
512
0.30
0.23
0.25
118 mam
010
0.0
10-1
2.56
0.06
0.06
Gasch
s20ch
(k)
(I)
epsilon = 10
Figure 24: Cosine distance evolution for Transformer described in Appendix trained on modular division. Observe that the cosine distance from initialization increases with models that experience Slingshot Effects.
33

---

Published in Transactions on Machine Learning Research (02/2024)
train
Validition
last
train
validation
layer norm
10
0.11
100
0.31
0.10
10
0.09
last layer nom
accuracy
60
0.09
layer nomm
0.08
0.08
10
0.07
0.07
10
20
0.06
0.05
0.05
20000 40000 60000 воñоо 1000001 200001 40000
20000 40000 6аoоо 80600 1060001 200001 40000140000 40000
epoch
(a)
(b)
train
Validation
last
train
validation
last layer norm
10:
0.32
102
0.11
0.11
80
0.10
10
0.09
byar nmm
acourscy
60
0.09
100
Tayer
0.08
10
0.07
0.07
10
20
0.06
0.06
10-1
0.05
0.05
20000
40000 60000 80000
1000001 200001 40000
20000 40000 6000a
BоñоD 20000000000000000000000000000000 40000
epoch
(d)
train
Validation
last
train
validation
layer norm
10:
10a
80
0.12
10-1
1ast layer narm
18st layer nomm
10
0.10
accuracy
60
0.10
10
20
10:
0.06
0.06
20000 40000 60000 80000 1000001 20000 40000
20000 40000 60000 80000 1000001 40000000000000000000000000
epoch
(e)
(f)
Figure 25: Optimizing a Transformer with SGD on modular division dataset: Norm growth (a), (c), (e) training and validation loss, (b), (d), (f) training and validation accuracy. Note the lack of Slingshot Effects, Grokking and generalization seen with Adam optimizer.
34

---

Published in Transactions on Machine Learning Research (02/2024)
loss
validation accuracy
validation accuracy
last layer norm
train loss
last layer
100
2.0
2.0
80
10
last layer norm
1.5
last lastllayer layer noms
10
accuracy
60
loss
10
1.0
1.0
10
0.5
20
0.5
10 10
0..0
20000
40000
60000
80000
100000
40000
80000
100000
epoch
epoch
(a)
(b)
B2 = 0.98, E 10-8
82 = 0.98, 6 = 10
train loss
layer
validation accuracy
last layer norm
1..6
100
10a
0.5
10
0.4
Last layer norm
accuracy
60
0.4
last layer norm
loss
0.3
10-6
0.2
0.2
20
10-1
0.1
0.1
20000
40000
60000
80000
100000
20000
40000
60000
80000
100000
epoch
epoch
(c)
(d)
82 0.9995,
10 18
B2 0.9995,
10-8
train loss
last layer norm
validation accuracy
last layer norm
10
70
1.0
60
10
0.8
0.8
last layer norm
jast layer norm
accuraEy
40
1055
10-1
0.6
0.6
30
10
0.4
20
0.4
10-
10
20000
40000
60000
80000
100000
20000
40000
60000
80000
100000
epoch
epoch
(e)
(f)
3/ 0.9997, 10
82 = 0.9997, 10-8
Figure 26: Effect of varying Adam optimizer's 32 hyperprameter. Plot shows train loss and validation accuracy of a Transformer trained on modular division dataset described Appendix B Observe that Slingshot Effects and training spikes still persists at very high values of 82.
35

---

Published in Transactions on Machine Learning Research (02/2024)
train validation
last layer norm
train
validation
last layer norm
100
10a
1.75
1.75
1.50
1.50
10
188 layer norm
Bayleyrr norm
1.25
accuracy
10
loss
1.00
10
0.75
0.75
10
0.50
10-10
0.25
0.25
0.00
0.00
10000
20000
30000
40000
50000
10000
20000
30000
40000
50000
epoch
opoch
(a)
(b)
Figure 27: Addition dataset with 50/50
split. Training and validation (a) loss and (b) accuracy.
train
validation
last layer norm
train
validation
last
100
2.0
2.0
layer norm
last laeer norm
10
accuracy
60
1.5
loss
1.0
40
1.0
10
0.5
20
10-10
0.0
10000
20000 epoch
30000
40000
10000
20000
30000
40000
epach
(a)
(b)
Figure Addition dataset with 60/40 train/validatinn validation split. Training and validation (a) loss (b) accuracy.
train
validation
last layer norm
train
va validation
last layer norm
10a
2.5
100
2.5
2.0
last layer nomm
last layer norm
accuracy
1.5
loss
10
1.0
40
1.0
10-8
0.5
20
0.5
10
0.0
5000 10000 15000 20000 25000
30000
35000
40000
5000
10000 15000 20000 25000 30000 35000 40000 epach
epoch
(a)
(b)
Figure 29: Addition dataset with 70/30 train/ train/validdan /alittiinoi split. Training and validation (a) loss and (b) accuracy.
36

---

Published in Transactions on Machine Learning Research (02/2024)
train validation
last layer norm
train validation
last layer norm
10a
100
10a
2.0
10
last last-laye nom
last layer hor
11.5
accuracy
1.5
lods
10
1.0
40
1.0
10
0.5
20
0.5
10-10
0.0
10000
20000
30000
40000
5.0000
10000
20000
30000
40000
50000
epoch
epoch
(a)
(b)
Figure 30: Cubepoly dataset with 50/50 train/ train/validation split. Cubepoly operation given by (a + (mod p) for 0 S a, b p). Training and validation (a) loss and (b) accuracy.
train
validation
last layer
layer
2.00
100
2.00
10a
1.75
1.75
1.50
1.25
accuracy
1.25
lo3s
layer
1.00
10
0.75
0.75
10
0.50
20
0.50
10 10
0.25
0.00
0.00
10000
20000
30000
40000
10000
20000 epoch
30000
40000
opoch
(a)
(b)
Figure 31: Cubepoly dataset with 60/40 train/valict iain/vvaidaaiin split. Cubepoly operation is given by (a + (mod p) for a, b p). Training and validation (a) loss and (b) accuracy.
37

---

Published in Transactions on Machine Learning Research (02/2024)
train
validation
last layer
train
validation
last layer
100
10
1.75
10
1.50
10
1.25
18 layer norm
accuracy
nomm
105s
1.00
1.00
10
0.75
0.75
10
0.50
10
0.25
0.25
0.00
0.00
5000 10000 15000 20000 25000
30000
35000
40000
5000
10000 15000 20000 25000 30000 35000 40000
epoch
(a)
(b)
Figure 32: Cubepoly datasel with 70/30 train/vale /wti/vaaiiaiiooo split. Cubepoly operation is given by (a + b (mod p) for a b p). Training and validation (a) loss and (b) accuracy.
train
validation
last layer
train
validation
last layer norm
100
100
1.75
1.75
1.50
1.50
1.25
23 layer norm
1.25
nom
10
accuracy
lass
1.00
10
0.75
10
0.50
0.50
10-10
0.25
0.25
0.00
0.00
5000
10000
15000
20000
25000
30000
5000
10000
15000 20000
25000
30000
epoch
(a)
(b)
Figure 33: Cubepoly dataset with 80/20 orain/validattinn split. Cubepoly operation is given by (a + b (mod p) for a p). Training and validation (a) loss and (b) accuracy.
38

---

Published in Transactions on Machine Learning Research (02/2024)
train validation
last layer
train
validation
last layer norm
10a
100
10a
1.75
1.75
1.50
1.25
accuracy
1.25
lass
10
layer
playe
1.00
1.00
10
0.75
0.50
0.50
0.25
10-10
0.00
0.00
10000
20000
30000
40000
50000
10000
20000
30000
40000
50000
epoch
epoch
(a)
(b)
Figure 34: Division dataset with 50/50 train/ vali/validattio split. Training and validation (a) loss and (b) accuracy.
validation
layer norm
layer norm
10a
100
1.75
1.75
109
1.50
1.50
10
1.25
last layer norm
accuracy
lo0s
10
1.00
layer
1.00
10-4
0.75
40
0.75
10-1
0.50
20
0.25
10.00
0.00
10000
10000
20000
30000
40000
epoch
epoch
(a)
(b)
Figure 35: Division dataset with 60/40 train/ train/validation validation split.
Training and validation (a) loss and (b) accuracy.
train
va validation
last layer narm
train
validation
last layer norm
100
2.5
10a
2.5
80
10 10
2.0
2.0
101 laye horm
aast layer norm
accuracy
1.5
loss
10
40
1.0
0.5
20
0.5
10-10
0.0
0.0
25000
spoch
epoch
(a)
(b)
Figure 36: Division datasel with 70/30 mnin/validation split. Training and validation (a) loss and (b) accuracy.
39

---

Published in Transactions on Machine Learning Research (02/2024)
train validation
last layer norm
train
validation
last layer norm
100
10
1.50
10
1.25
1.25
10
1.00
layer
accuracy
1.00
10
0.75
0.75
10
0.50
10
0.25
0.00
0.00
5000
10000 15000 20000 25000 30000 epoch
35000
40000
5000 10000 15000 20000 25000 30000 35000 40000
(a)
(b)
Figure 37: Eveer-aldd-odd---bbaaciiii dataset with 70/30 train/validation /alidation split. Even-adddooddd-atbtaaciii operation is given by (a b (mod p) if is even, otherwise b (mod p)] for S a, < Training and validation (a) loss and (b) accuracy.
train
va didation
last layer
train
validation
last layer norm
100
2.0
2.0
108
1.5
1.5
last yyer norm norm
accuracy
60
loss
layer
11.0
1.0
10
40
10
0.5
20
10
10000
20000
30000
40000
50000
10000
20000
30000
40000
50000
opoch
epach
(a)
(b)
Figure 38: Multiplication dataset with 50/50 train veainvvalidation split. Training and validation (a) loss and (b) accuracy.
40

---

Published in Transactions on Machine Learning Research (02/2024)
train validation
last layer norm
train
validation
last layer norm
10a
MAh
2,00
100
2.00
1.75
1.75
80
10
1.50
10
1.25
accuracy
1.25
l00s
layer
1.00
10
0.75
0.75
10
0.50
10
0.25
0.25
10000
20000
30000
40000
10000
20000
30000
40000
epoch
epoch
(a)
(b)
Figure 39: Multiplication dataset with 60/40 train/ /alidation split. Training and validation (a)
and (b) accuracy.
train
va Ialidation
last layer norm
train
validation
last layer norm
2.5
100
2.5
10a
80
10-2
aatt layer nomm
1.5
accuracy
60
1.5
10
lexss
last layer
10
1.0
1.0
20
0.5
10-10 -10
0.0
0.0
5000 10000 15000 20000 25000 30000 35000
40000
5000
10000 15000 20000 25000 30000 35000 40000 epoch
epoch
(b)
Figure 40: Squarepoly dataset with 70/30 crain/validation split. Squarepoly operation is given by a² + b (mod p) for S a, b < p. Training and validation (a) loss and (b) accuracy.
41

---

Published in Transactions on Machine Learning Research (02/2024)
train validation
last layer norm
train
validation
layer
100
10a
2.0
2.0
80
layer norm
1.5
last layer norm
accuracy
60
10
loss
10
1.0
1.0
10
0.5
20
0.5
10-10
0.00
0.0
5000
10000
15000
20000
25000
30000
5000
10000
15000
20000 epoch
25000
30000
epoch
(a)
(b)
Figure 41: Squarepoly dataset with 80/20 train/ nalidavaidttion split. Squarepoly operation is given by a² + (mod p) for
a, <pp. p. Training and validation (a) loss and (b) accuracy.
train
validation
last layer norm
train
validation
last layer norm
100
10P
80
10
1.5
bast layer norm
1.5
last layer norm
accuracy
10
lass
10
1.0
1.0
10
0.5
20
10
0.0
0.0
10000
30000
10000
20000
30000
40000
epoch
epoch
(a)
(b)
Figure 42: Subtraction dataset with 60/40 train/validann split. Training and validation (a) loss and (b) accuracy.
validation
last layer
validation
layer norm
1.75
1.75
10a
Va Ch Mlo
1.50
1.50
1.25
125
Joyer norm
accuracy
1o5s
1.00
last laye
10
0.75
40
0.75
10
0.50
0.50
20
10-10
0.25
0.25
1.000
0.00
10000 15000
20000 epoch
25000
30000 35000
40000
5000
10000 15000 20000 25000 30000 35000 40000 epoch
(b)
Figure 43: Subtraction dataset with 70/30 meiinvallddttinn split. Training and validation (a) loss and (b) accuracy.
42

---

Published in Transactions on Machine Learning Research (02/2024)
train
va lalidaaoon
last layer norm
validation
layer norm
100
10a
1.4
1.4
80
1.2
10
1881 layer mom
10
1.0
accuracy
60
loss
0.8
10
0.6
10
0.4
20
0.4
0.2
0.0
5000
10000
15000
20000 epach
25000
30000
5000
10000
15000 20000 epach
25000
30000
(a)
(b)
Figure 44: Subtraction dataset with 80/20 train/validation /valddaaioo split. Training and validation (a) loss and (b) accuracy.
43

---

Published in Transactions on Machine Learning Research (02/2024)
C
Controlling Instability Through Normalization and Norm Constraints
Training instability is the hallmark of the Slingshot Effect, yet as seen in previous sections, the Slingshot Effect typically results in improved performance, and Grokking. In this section, we explore whether is possible to maintain stable training, without sacrificing performance. To this end, explore how constraining and regularizing the weights of the network affect the Slingshot behaviour, and overall performance.
C.1
Weight decay
weight decay = 0
weight decay = 0.1
weight decay = 1.0
Irsh
valdd..n
3.50
10
5.75
10Th
10-4
331
10
0.5
334
9.12
10
2.75
10"
20
3.50
3.08
10
2.0
19550
20003
30300
42000
30072
5020
10000
23005
20020
25:00
30200
2020
10020
15900 G24ch
20005
23050
50000
spech
69047
(a)
(b)
(c)
train and validation loss vs epochs
valdation
valcation
tra
valication
last laye
0.16
1.50
0.1
4
60
1.25
60
a0cunce
1.00
0.12
0.75
0.10
23
0.05
10000
20020
30000
40300
39720
5203
10303
15930
20020
25:003
30020
5000
10000
13000
20030
25020
30000
spech
4po.h
(d)
(c)
(f)
train and validation accuracy vs epochs
Figure 45: Division dataset: Norm behavior with different weight decay values. Training and validation loss vs epochs with weight decay (a) 0.0, (b) 0.1, (c) 1.0; Training and validation accuracy vs epochs shown in (d), (e) and (f). The evolution of classifier weight norm shows instability as increase in weight decay strength.
Weight decay is commonly used regularization approach to improve the generalization performance of neural networks. Power et al. (2021) show that weight decay has the largest positive effect on alleviating Grokking. Weight decay naturally controls the size of the parameters and consequently their norm growth. We study the effect weight decay on stability of training Transformers with Grokking datasets in this section. We use weight decay values from 0, 0,0.1, 0.4, 0.8and1.0 with AdamW (Loshchilov Hutter 2017) optimizer. Figure 45 shows the results for division datasct. We observe from Figure 45) that weight decay strength increases, both Slingshot Effects and Grokking phenomenon disappear with the model reaching high validation accuracy quickly seen in Figure 15e and Figure 45f However, observe that the model experiences instability as can been seen with the loss plots in Figure 45b and Figure 45c or the accuracy plots in Figure 45e and Figure 451 A similar trend is observed for addition and multiplication datasets in
Figure 46 and Figure respectively.
The results shown above indicate that Slingshot may not be the only way to achieve good generalization. Both Slingshot and weight decay prevent the norms from growing unbounded and achieve high validation accuracy seen in plots described above. While weight decay shows different. weight norm dynamics, this
44

---

Published in Transactions on Machine Learning Research (02/2024)
weight decay 0
weight decay 0.1
weight decay 1.0
218
127
107
70
5.16
511
20"
316
0.40
16-1
12-4
5.05
10
5300 16320 35003 20300 25003 30000 35500
5050
15000
15000 23000
25:00
30300
35300
5005
10503
15300
20300
25300
50300 35500
(b)
(c)
train and validation loss vs epochs
val detion
lut layer Torm
last lastlaee nom
102
100
0.20
0.15
0.14
60
40
0.03
0.00
15dea 25630 opech
20l00 25502 30005 opoch
spech
train and validation accuracy epochs
Figure 46: Addition dataset: Norm behavior with different weight decay values. Training and validation loss vs epochs with weight decay (a) 0.0, (b) 0.1, (c) 1.0; Training and validation accuracy vs epochs shown in (d), (e) and (f). The evolution of classifier weight norm shows instability as increase in weight decay strength.
regularization does not decrease training instability. These results suggest the need for alternative approaches improve training stability.
45

---

Published in Transactions on Machine Learning Research (02/2024)
weight decay- 0
weight decay 0.1
weight decay 1.0
vellcution
last lopst rem
trai
dazinn
lasi laper
laxt layer nom.
9.16
511
30"
12+4
3.12
22-4
10-
3.02
20
3.00
23000
15500
20003
50002
20002
(a)
(b)
(c)
train and validation loss epochs
val dation
but layer norm
tran
valcattor
last
100
30
0.14
62
03
0.14
0.20
5.5
0.07
0.05
20200
20300
30200
40300
10000
23000
20200
15020
3000
20000 25070
20030 23030 33000 35.000
spoch
opach
spoch
(d)
(e)
(f)
train and validation accuracy vs epochs
Figure 47: Multúplication dataset: Norm behavior with different weight decay values. Training and validation loss vs epochs with weight decay (a) 0.0, (b) 0.1, (c) 1.0; Training and validation accuracy vs epochs shown in (d), (e) and (f). The evolution classifier weight norm shows instability as increase weight decay strength.
46

---

Published in Transactions on Machine Learning Research (02/2024)
C.2 Features and parameter normalization
A second approach that we use to explicitly control weights and feature norm is by normalizing the leatures and weights via the following scheme: inputs to the classification layer respectively, the norm used above is the L2 norm, and æ is the input the neural network. We take the cosine similarity of the normalized weights and features and divide this value by temperature value that treat hyperparameter in these experiments. The operation is given by: y = me s==(() where represents the temperature hyperparameter. We use temperature values from 0.1, 0.25, 0.5, 0.75, 1.0 for these experiments.
||=]| [(x) f(x)
[/(x)]] where and f(x) the weights and
Figure 48 shows the results of Transformer training on division dataset described in Appendix B that. is split evenly into train and validation sets. We observe that the model displays training instability evidenced by norm behavior and also loss behavior in Figure 48a at lower temperature values. We observe that r = 0.25 provides a good compromise between fitting training data while showing no training instability as seen in Figure 48b This hyperparameter value also results in Grokking as validation accuracy improves late training can be seen from Figure [Se] These together suggest that bounding weights and features norm helps stabilize training without sacrificing training performance.
We validate the normalization scheme with two additional datasets namely multiplication and division from Appendix B Figure 49 shows the results for training Transformers with multiplication dataset that is split evenly into train and validation sets. We observe from Figure [9 that proper temperature value can stabilize training and with some tuning can provide a compromise wetween training stability and generalization. Specifically, r = 0.25 allows the model fit the training data and reach almost perfect validation accuracy as seen from Figure 49b and Figure 49e
Finally, we repeat the above experiments with subtraction dataset and show the results in Figure 50 This dataset shows that while properly tuned temperature can help the model achieve almost perfect generalization, training instability shows up very late in optimization. This observation can be seen from Figure [5b] and Figure [0d] This result suggests that more work remains to be done with understanding and stabilizing the training behavior large neural networks.
47

---

Published in Transactions on Machine Learning Research (02/2024)
temperature 0.1
temperature 0.25
temperature = 1.0
Maddon
last 4y07 nom
leat koys
0.35
0.30
12
0.5
10
0.4
0.30
0.3
20
0.15
643100
0.15
0.2
0.3
1000
1.7
12x
30000
0.16
apoc1
(b)
(c)
train and validation loss epochs
lust
layer norm
0am
0.10
0.8
102
0.35
0.7
4.5
0.30
0.5
20
0.4
0.20
0.13
0.2
20
0.10
4.1
5003
23000
spach
spech
(d)
(c)
(f)
train and validation accuracy epochs
Figure 48: Division dataset: Features and parameters normalization. Observe that smaller temperature allows the model the data better but experiences training instability. Temperature 0.25 allows the model to and achieve high validation accuracy without suffering training instability.
48

---

Published in Transactions on Machine Learning Research (02/2024)
temperature 0.1
temperature = 0.25
temperature = 1.0
koys
3.4
1x
12.112
3.25
10
4."4100
0.35
0.2
305
42
3030040832
apace
spoch
(b)
train and validation loss epochs
last
dation
bain
lastlayer
12
0.8
20
40
20
0.10
40000
50020
spoch
spech
(pech
(d)
(c)
(f)
train and validation accuracy epochs
Figure 49: Multiplication dataset: Features and parameters normalization. Observe that a smaller temperature allows the model to the data better but experiences training instability. Temperature = 0.25 allows the model fit and achieve high validation accuracy without suffering training instability.
49

---

Published in Transactions on Machine Learning Research (02/2024)
temperature 0.1
temperature 0.25
temperature 1.0
Mdttion
lest ayyernom
00.45
0.10
107
0.15
5.2
1x/00-
Sxite
150
0,4
20F
0.20
115
0.25
0.2
0.00
6.05
43x
10000
20000
30000
40000
5000 10020 13000 20100
22002
30000 35020
5739
19710
15000
20000
(a)
(b)
(c)
train and validation loss vs epochs
Baa
val da-daBin
lost Mper 10r7
Dan
last layer norm
00.45
0.43
0.255
1.0
0.33
0.200
80
0.30
9.315
0.23
4.300
80
0.20
9.325
0.15
0.100
0.10
135.00
20030
32000
40350
10036
154cn
250m0
75030
spech
opech
apoch
(e)
(f)
train and validation accuracy vs epochs
Figure 50: Subtraction dataset: Features and parameters normalization. Observe that a smaller temperature allows the model to fit the data better but experiences training instability. Temperature 0.25 allows the model to fit and achieve high validation accuracy. However, we observe training instability as can seen with weight norm dynamics.
50

---

Published in Transactions on Machine Learning Research (02/2024)
D Slingshot Effects and Linear Mode Connectivity
2023 have used mode connectivity analysis to explore the loss landscapes of neural networks. These works suggest the existence of multiple basins in the loss surface each with its own generalization behavior (Lubana et al. 2023; Juneja et al. 2023). Given the empirical observations we make with Slingshot Effects, one hypothesis to explore is that each Slingshot Effect leads the model to a different basin in the loss landscape that shows potentially differeing generalization behavior. In this section, we explore whether Adam optimizer (Kingma & Ba 2014) explores different basins in the loss landscape by plotting the loss along the linear path between different model checkpoints collected during optimization. We defer thorough study of the above hypothesis to future work.
Recent works Lubana et al. 2023 Juneja et al.
10a
train loss validation loss
1.0
0.9
10-1
accuracy
10-3
0.7
10-5
0.6
10-7
train accuracy
validation accuracy
2000
4000
6000
8000
10000
2000
4000
6000
8000
10000
spoch
epech
(a)
(b)
Figure 51: Subset parity dataset: train and validation (a) loss and (b) accuracy.
We train 3-layer MLP with hidden layer width 100 on a (50, 3) subset-parity dataset described in Ap-
pendix A.6 The training dataset consists of 1000 samples while the test dataset consists of 8192 samples. We optimize the network with Adam and cross-entropy loss using a learning rate 0.004, 33 0.9, B3 = 0.999, 10-8 and batch size set to 32. The model optimized with Adam (Kingma Ba 2014) for 10000 epochs. Figure shows the train and validation loss and accuracy metrics for this experiment.
The lincar mode connectivity (LMC) analysis is conducted by using two checkpoints input and interpolating between the checkpoints to generate model weights. We choose two checkpoints that are equidistant, (as measured in epochs) from Slingshot Effect event or training loss spike. Specifically, we use two checkpoints that are 25 epochs on cither slide of a Slingshot Effect as this value allows us to include periods of low train loss on either side of the loss spike. We tested other values for the number of epochs going as low as epochs and found that this hyperparameter has a negligible effect on our analysis.
We lincarly interpolate between the two checkpoints chosen using the method described above and use the interpolated weights to calculate train and validation metrics. Figure 52 shows example LMC plot that includes the train loss calculated via interpolation (top-left). We observe that the LMC train loss shows a peak that suggests loss barrie that in turn suggests that the model checkpoints are in different basins. Figure (3 and Figure 54] captured at different points during training, show additional examples of LMC analysis that intuitively suggest that the model jumps between different loss basins.
We borrow this terminology from
Lubana al.
2023 Juneja
2023
51

---

Published in Transactions on Machine Learning Research (02/2024)
Weights obtained by interpolating checkpoints at epoch 3448 & epoch 3498
Weights obtained using Adam optimizer
10
10
10
10
10 3
loss
10-5
10-5
10
train loss
10
train validation loss
validation loss
0.0
0.2
0.4
0.6
0.8
1.0
3450
3460
3470
3480
3490
linear interpolation coefficient
epoch
1.00
1.000
0.98
0.975
accuracy
accuracy
train accuracy
0.950
0.96
validation accuracy
0.925
0.94
0.900
train accuracy validation accuracy
0.92
0.0
0.2
0.4
0.6
0.8
1.0
3450
3460
3470
3480
3490
linear interpolation coefficient
epoch
Figure 52: Train and validation metrics calculated with model weights generated via linear interpolation (left) and via Adam optimization (right).
Weights obtained by interpolating checkpoints at epoch 6320 & epoch 6370
Weights obtained using Adam optimizer
10
10
10 -1
10 33
loss
10-3
10-5
10-5
10
train loss
10-7
train loss validation loss
validation loss
0.0
0.2
0.4
0.6
0.8
1.0
6320
6330
6340
6350
6360
6370
linear interpolation coefficient
epach
1.000
1.00
0.98
accuracy
0.995
accuracy
0.96
0.990
train accuracy validation accuracy
0.94
train accuracy validation accuracy
0.0
0.2
0.4
0.6
0.8
1.0
6320
6330
6340
6350
6360
6370
llnear interpolation coefficient
epoch
Figure 53: Train and validation metrics calculated with model weights generated via linear interpolation (left) and via Adam optimization (right).
52

---

Published in Transactions on Machine Learning Research (02/2024)
Weights obtained by interpolating checkpoints at epoch 7301 epoch 7351
Weights obtained using Adam optimizer
10
10
10
10-3
loss
10-3
10-5
10-5
10-7
train validation loss
10-7
train validation loss
0.0
0.2
0.4
0.6
0.8
1.0
7300
7310
7320
7330
7340
7350
linear interpolation coefficient
epoch
1.000
1.00
0.998
accuracy
0.996
accuracy
0.99
0.994
0.98
0.992
train accuracy
train accuracy validation accuracy
validation accuracy
0.990
0.97
0.0
0.2
0.4
0.6
0.8
1.0
7300
7310
7320
7330
7340
7350
linear interpolation coefficient
epoch
Figure 54: Train and validation metrics calculated with model weights generated via linear interpolation (left) and via Adam optimization (right).
53


Reviews:
[{"review_id": "XeLKIDQ7Bq", "reviewer": "Reviewer_5SKJ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Extensive experimentation with different choices of architecture, optimizer, and dataset in deep learning has been an important driver of progress in our understanding. One of the contributions of empirical research in understanding deep learning is to highlight phenomena that are unexplained by existing theories and models. In this paper the authors are able to identify the situations in which the slingshot effect emerges and show that varying the $\\epsilon$ parameter controls its emergence. The authors could however incorporate some theoretical insights to sharpen their description of the slingshot effect and isolate the exact anomaly in optimization with adaptive gradient methods.\n\n1. **Optimization observations:** Gradient based minimization (SGD/GD/adaptive gradient methods) using the cross-entropy loss should naturally increase the norm of models. This is because the minimizers of the unregularized cross entropy loss are at $\\infty$, and a simple model of the dynamics of SGD [1, Lemmas 1 and 2] also predicts the norm growth. Hence the dynamics of \"norm curtailment\" are not surprising.\n\nThe observation that is actually interesting (in my opinion) is the fact that adaptive gradient methods display sharp spikes in the training loss and norm. A hypothesis that could explain these observations is that in these large model/small dataset setups, adaptive gradient methods jump from one local/global minimum basin to another. This could indicate two regimes of operation for adaptive gradient methods - a) where the gradients are large (relative to $\\epsilon$) and SGD type dynamics dominate to find the local minimum of a basin and b) where the gradients are small (relative to $\\epsilon$) and the adaptive methods are pushed out of the local basin that they are currently in. More careful sharpness measurements could help uncover whether this is the case. I will note here that the observations about feature change (Fig 2) are interesting and could be an avenue for further theoretical/empirical investigation.\n\n2. **Generalization observations:** The authors claim that the slingshot effect is accompanied by improved generalization, and while they are careful to not claim that this is a causal link, they do not posit any mechanisms for how this improvement in generalization could be happening. Connecting back to the previous observation about different basins in the loss landscape, one mechanism for improving generalization is that the different basins have different generalization properties (similar to [2,3]). Exploring this hypothesis would motivate the slingshot effect better. The current exploration of connections to generalization (Figures 5,6) do not have enough models in which the slingshot effect is absent in order to draw effective conclusions.\n\n\nMy broad concern about this paper is that as it is framed currently, the slingshot effect could be explained entirely in terms of jumping between different basins around local minima with different generalization capabilities, and this hypothesis is under-explored in favor of tracking norm dynamics, which are reasonably well understood and will not be explanatory for generalization.\n\n[1] Poggio, Tomaso, Andrzej Banburski, and Qianli Liao. \"Theoretical issues in deep networks.\" Proceedings of the National Academy of Sciences 117.48 (2020): 30039-30045.\n\n[2] Juneja, Jeevesh, Rachit Bansal, Kyunghyun Cho, João Sedoc, and Naomi Saphra. \"Linear connectivity reveals generalization strategies.\" arXiv:2205.12411, ICLR 2023\n\n[3] Lubana, Ekdeep Singh, Eric J. Bigelow, Robert P. Dick, David Krueger, and Hidenori Tanaka. \"Mechanistic mode connectivity.\" In International Conference on Machine Learning, pp. 22965-23004. PMLR, 2023.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper describes a slingshot effect that occurs during the terminal phase of training models to minimize the unregularized cross entropy loss using adaptive optimizers. During the terminal phase of training, they identify that the training loss undergoes periodic spikes, which occur simultaneously with an explosion in the norm of the model. Following the explosion, the norm growth is curtailed and the training loss is minimized until the next \"slingshot\" occurs. Each subsequent slingshot also seems to be accompanied by an improvement in the validation loss/accuracy, which is reminiscent of the \"grokking\" phenomenon where changes in the validation loss can occur long after the training loss has been minimized.\n\nThe authors attempt to provide explanations for the emergence of the slingshot phenomenon by showing that variations in the stability parameter $\\epsilon$ of an adaptive gradient algorithm can control the emergence/disappearance of the slingshot phenomenon.\n\nThe authors also claim that the slingshot effect is indicative of better generalization and compare models that exhibit this effect.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Establish the slingshot effect in terms of whether adaptive optimizers progressively explore different basins in the loss landscape by plotting the loss along the linear path between different models.\n\n2. How do the basins differ in generalization? Are there differences in the features learned at different stages?", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "3fyocVRpWm", "reviewer": "Reviewer_aRAs", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper is easy to follow and well-written.\n2. The observed Slingshot effect is interesting and has potential impact in understanding Adam.\n3. The authors have conducted numerous experiments to show how common the Slingshot effect is.\n\nWeaknesses:\n\n1. First of all, I think the title (and of course, many parts of the paper) should be changed because most of the experiments in this paper are solely concentrated on \"Adam\". RMSProp is just a special case of Adam with first-order momentum removed ($\\beta_1 = 0$ in Adam). Therefore, claiming \"a late-stage optimization anomaly in adaptive gradient methods\" would be over-selling. \"Adaptive Optimization Algorithms\" is a term used to describe optimization algorithms that have the ability to change its stepsizes based on certain criterions during the optimization process. For example, AdaGrad, AMSGrad, AdaX, AdaBelief, and AdaBound are quite famous, just to name a few. (Please let me know if you need references for them) These works should not only be mentioned in the related works, but also conducted experiments with. Of course, it would be infeasible to conduct experiments with all of them, but some of them should be fine. The authors have also mentioned in page 6 that AdaGrad does not have the Slingshot effect, so claiming that such effect is universal is overselling.\n\n2. The paper is completely an experimental and observational work, with no theories to explain the proposed effect. As I mention in Weakness 1, it would be infeasible to try all the adaptive optimization algorithms. Therefore, a theoretical study of the proposed Slingshot effect should be desirable. Given the current paper, I cannot tell the reasons behind the proposed Slingshot effect, why it happens for Adam and not for AdaGrad, and how to prevent it/reproduce it.\n\n3. The authors have mentioned that in the experiments, weight decay is set to 0. Does weight decay prevent the proposed Slingshot effect? If so, I would doubt the importance of this effect since weight decay is commonly used in practice. Moreover, in page 6, the authors have mentioned AdamW. If weight decay is not used, AdamW is basically the same as Adam. Therefore I am quite confused by the usage of weight decay in this paper.\n\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper have shown an interesting phenomenon named the Slingshot effect during the optimization of deep neural networks using adaptive optimizers, specificially Adam (and RMSProp). The authors have conducted extensive experiments to show that this effect is predominant in different deep learning senarios, with models ranging from MLP to Transformers, and datasets from synthetic Gaussian data to real-life datasets such as CIFAR10. The authors have also shown an interesting connection between Grokking and the Slingshot Effect.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please refer to the weakness part. Specifically, I would consider accepting this paper if:\n\n1. Adam is highlighted instead of adaptive optimization algorithms in general.\n2. Theoretical understanding/intuition is provided. It does not have to be very hard. Even linear regression is fine.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "yeR0JchD3p", "reviewer": "Reviewer_zYov", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "See below", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper empirically verify a interesting pehonoon of Adam, called Slingshot effet, which stands for a repeating phase shifts between stable and unstable training regimes after the model overfits the training set.  As claimed by authors, slingshot is  a unique effect of Adam family. Further, as shown in Figure 6, the model seem to have better generalization power after the slingshot performance. As such, authors belive that the above discovery reveal an intriguing inductive bias of adaptive gradient optimizers. \n\n\n\nOn the algorithmic datasets by Power et al. , The authors also discuss the intersting relation between slingshot and  Grokking effect (Power et al., 2021) . \n\n\n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "\n**Major Questions:**\n\n1. **Universial training phenomenon?**  I am not sure whether the Slingshot Effect is a universal phenomenon. Authors only verify this phenomenon on special toy data set by  (Power et al., 2021) and small data set like cifar 10. We are not sure whether the pheononm exists in other real language or vision tasks. Further,  we are not sure whether sliingshot is related to the spike effect in LLM as reported by Chowdhery et al., 2022, either. \n\n\n\n\n\n2. **Relation with generalization power?**  The author claims that  \"the model has better generalization power after the Slingshot phase\". However, again, this claim is only verified on cifar10 and toy datasets. Without further evidence, the connection between Slingshot and generalization is quite weak, making the script less interesting than it claimed. \n\n\n\nRemark: Although both questions are related to the limited experiments demonstrated in the script, question1 and 2 are two fundamentally different questions. The first focus on training,  and the 2nd focus on generalization.  The authors need to provide more evidence to support their claim on \"discovering an important and universal training phenomenon\" and \"its relation to implicit bias of Adam/ generalization power\"\n\n**Other questions:**\n\n1. The coloring in Figure 6 is confusing. Correct me if wrong: From my understanding ,   every column \n\n(# samples)  should  have  three balls **with distinct color**, right?  But now many columns has three green balls. \n\n\n2. Is the instability phenomenon due to the improper choice of beta2 in Adam? As proved in recent work [1], Adam might not work well if beta2 is not sufficiently large. I suggest the authors spent some effort tuning beta2 according to the guidance in [1], then see if the spiking phenomenon will be mitigated. For instance, simply try larger beta2 like 0.9995, 0.9999 to see if the spike disappears. \n\n[1] adam can converge without any modification on update rules, Zhang et al., 2022\n\n\n**Writing suggestion:**\n\nThe logic of the paper is a bit confusing. I suggest the author introduce the messages one by one as follows: 1. discover a new training phenomenon with spikes, and provide evidence. 2. Discuss the relation with generalization power, and provide evidence (the cifar 10 part). 3. Discuss the relation with Grokking, and provide evidence.  \n\nCurrently, 12 and 3 are all mixed up without clear logical order. It is difficult to read for general readers.   \n\n", "broader_impact_concerns": "none", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The claims made in the paper are well supported by extensive experiments and two of the reviewers have recommended acceptance after the author response. Reviewer aRAs is learning towards reject -- they acknowledge that the paper identifies an interesting phenomenon but are concerned about the lack of any theoretical understanding about it in the paper. While the point on the lack of theoretical understanding is valid, the paper does acknowledge this under the limitations section. Since the paper supports all the claims with extensive experiments as acknowledged by the reviewers and the slingshot phenomenon itself is deemed as interesting by all three reviewers, AE thinks the paper meets the [TMLR criteria for acceptance](https://www.jmlr.org/tmlr/acceptance-criteria.html). 

Minor change: Authors should add the experiment suggested by the Reviewer 5SKJ on plotting the loss along the linear path between different models to provide intuition to the readers from the perspective of jumping between different loss basins.
Decision: Accept (poster)