Outliers with Opposing Signals Have an Outsized Effect on Neural Network Optimization
Authors: Elan Rosenfeld
Andrej Risteski
Venue: ICLR 2024 poster (2024)
Keywords: neural network optimization
progressive sharpening
edge of stability
adaptive gradient methods
batch normalization
Primary area: optimization
TL;DR: We show the influence of samples w/ large, opposing features which dominate a network's output. This offers explanations for several prior observations including the EoS. We analyze a 2-layer linear net, reproducing the observed patterns.

Abstract: We identify a new phenomenon in neural network optimization which arises from the interaction of depth and a particular heavy-tailed structure in natural data. Our result offers intuitive explanations for several previously reported observations about network training dynamics, including a conceptually new cause for progressive sharpening and the edge of stability. We further draw connections to related phenomena including grokking and simplicity bias.

Experimentally, we demonstrate the significant influence of paired groups of outliers in the training data with strong *Opposing Signals*: consistent, large magnitude features which dominate the network output and provide gradients which point in opposite directions. Due to these outliers, early optimization enters a narrow valley which carefully balances the opposing groups; subsequent sharpening causes their loss to rise rapidly, oscillating between high on one group and then the other, until the overall loss spikes. We carefully study these groups' effect on the network's optimization and behavior, and we complement this with a theoretical analysis of a two-layer linear network under a simplified model.

Our finding enables new qualitative predictions of training behavior which we confirm experimentally. It also provides a new lens through which to study and improve modern training practices for stochastic optimization, which we highlight via a case study of Adam versus SGD.

Published as a conference paper at ICLR 2024
OUTLIERS WITH OPPOSING SIGNALS HAVE AN OUT- SIZED EFFECT ON NEURAL NETWORK OPTIMIZATION
Elan Rosenfeld
Andrej Risteski
Carnegie Mellon University el alann& 2 edu
Carnegie Mellon University
aristeakkandreew cmu.
ABSTRACT
We identify new phenomenon in neural network optimization which arises from the interaction of depth and a particular heavy-tailed structure in natural data. Our result offers intuitive explanations for several previously reported observations about network training dynamics, including a conceptually new cause for progres- sive sharpening and the edge of stability. We further draw connections to related phenomena including grokking and simplicity bias. Experimentally, we demon- strate the significant influence of paired groups of outliers in the training data with strong Opposing Signals: consistent, large magnitude features which dominate the network output and provide gradients which point in opposite directions. Due these outliers, early optimization enters narrow valley which carefully balances the opposing groups; subsequent sharpening causes their loss to rise rapidly, oscil- lating between high on one group and then the other, until the overall loss spikes. We carefully study these groups' effect on the network's 's optimization and be- havior, and we complement this with a theoretical analysis of two-layer linear network under a simplified model. Our finding enables new qualitative predictions of training behavior which we confirm experimentally. It also provides a new lens through which to study and improve modern training practices for stochastic op- timization, which we highlight via case study of Adam versus SGD.
INTRODUCTION
There is steadily growing list of intriguing properties of neural network (NN) optimization which are not readily explained by prior tools from optimization. Likewise, there exist varying degrees of understanding of the mechanistic causes for each. Extensive efforts have led to possible explanations
for the effectiveness of Adam (Kingma & Ba 2014), Batch Normalization (loffe Szegedy 2015)
and other tools for successful training- but the evidence is not always entirely convincing, and there certainly little theoretical understanding. Other findings, such as grokking (Power et al.) 2022) or the edge of stability (Cohen et al. 2021), do not have immediate practical implications, but they provide new ways to study what sets NN optimization apart. These phenomena are typically consid- ered in isolation- they not completely disparate, it is unknown what specific underlying factors they may share. Clearly, a better understanding of NN training dynamics in a specific context can lead algorithmic improvements (Chen al. 2021): this suggests that any commonality will be a valuable tool for further investigation.
In this work, we identify a phenomenon in NN optimization which offers a new perspective on many of these prior observations and which we hope will contribute to a deeper understanding of how they may be connected. While we do not (and do not claim to) offer a complete explanation for this finding, we present strong qualitative and quantitative evidence which suggests more coherent picture of the origin of these prior results, offering a single high-level idea which naturally fits into several existing narratives. Speeifically, we demonstrate the prevalence of paired groups of outliers which have a significant influence on a network's optimization dynamics. These groups are characterized by the inclusion of one or more (relatively) large magnitude features that dominate the network's 's output at random initialization and throughout most of training. In addition to their magnitude, the other distinctive property of these features is that they provide large, consistent, and opposing gradients, in that reducing loss on one group (usually) increases loss on the other; because of this structure, we refer them as Opposing Signals. These features share non-trivial correlation

---

Published as a conference paper at ICLR 2024
Lats Nuto Oftr
Vehicular Features:
Geus
Bockaround Cobor
Nuto Track
Frog Otter
Background Colee
Lats of Red
Trals Les (Goog
Grep
Gmap
Trais ofrong
Lot
Grap
Gmay2
Vchicular Features
Gran Testure
Greep
Grep
Oreer
GOMD
2119
2013
1.00
11400
Training dynamics of neural networks are heavily influenced by outliers with op- posing signals. We plot the overall loss of a ResNet-18 18 trained with GD on CIFAR-10, plus the losses of a small but representative subset of outlier groups These groups have consistent opposing signals (e.g., wheels and headlights sometimes means car and sometimes truck). Throughout training, losses on these groups oscillate with growing amplitude-this oscillation has an obvious correspondence to the short term spikes in overall training loss and, in particular, appear to be the direct cause of the cedge-of-stabiltyy phenomenon.
Figure 1:
with the target task, but they are often not the "correct" (e.g. human-aligned) signal. In fact, in many cases these features perfectly encapsulate the classic conundrum of "correlation vs. causation" for example, bright blue sky background does not determine the label of CIFAR image, but it does most often occur in images of planes. Other features are relevant, such as the presence of wheels and headlights trucks and cars, or whether punctuation comes before after the end of quotation.
Opposing signals are most easily understood with an example, which we will give along with a brief outline of their effect on training dynamics; more detailed description is presented in Section 3 Fig. 1 depicts the training loss of a ResNet-18 trained with full-batch gradient descent (GD) on CIFAR-10, along with few dominant outlier groups and their respective losses. In the early stages of training, the network enters a narrow valley in weight space which carefully balances the pairs' opposing gradients; subsequent sharpening of the loss landscape (Jastrzebski et al. 2020 Cohen et al., 2021) causes the network to oscillate with growing magnitude along particular axes, upsetting this balance. Returning to our example of sky background, one step results in the class plane being assigned greater probability for all images with sky, and the next will reverse that effect. In essence, the "sky = plane" subnetwork grows and shrinks The direct result of this oscillation is that the network's loss images of planes with sky background will alternate between sharply increasing and decreasing with growing amplitude, with the exact opposite occurring for images of non non-planes with sky. Consequently, the gradients of these groups will alternate directions while growing in magnitude as well. As these pairs represent a small fraction of the data, this behavior not immediately apparent from the overall training loss-but eventually, progresses far enough that the overall loss spikes. As there is an obvious direct correspondence between these two events throughout, we conjecture that opposing signals are the direct cause of the "edge-of-stabile phe- nomenon (Cohen et al.) 2021). We also note that the most influential signals appear to increase in complexity over time (Nakkiran et al. (201)).
We repeat this experiment across range of vision architectures: though the precise groups and their order of appearance change, the pattern occurs consistently. We also verify this behavior for transformers on next-token prediction of natural text and small ReL.U MLPs simple 1D functions; we give some examples of opposing signals in text in Appendix [B] However, we rely on images for exposition because offers the clearest intuition. To isolate this effect, most of our experiments use GD but we observe similar patterns during SGD, which we present in Section
It would be more precise to say "strengthening connections between regions of the network's output and neurons which have large activations for sky-colored inputs". Though we prefer to avoid informal terminology. this example makes clear that the more relaxed phrasing is usually much cleaner. We therefore employ it when the intended meaning is clear.
2

---

Published as a conference paper at ICLR 2024
Contributions. The primary contribution of this paper is demonstrating the existence, pervasive- ness, and large influence of opposing signals during NN optimization. We further present our current best understanding, with supporting experiments, of how these signals cause the observed training dynamics-in particular, we provide evidence that it a consequence of depth and steepest descent methods. We complement this discussion with toy example and analysis of two-layer linear net on a simple model. Notably, though rudimentary, our explanation enables concrete qualitative predictions of NN behavior during training, which we confirm experimentally. also provides new lens through which to study modern stochastic optimization methods, which we highlight via case study of SGD vs. Adam. We see possible connections between opposing signals and a wide va- riety of phenomena in NN optimization and generalization, including grokking (Power et al. 2022). catapullin//lingshotting (Lewkowyez et al. 2020; Thilak et al., 2022), simplicity bias (Valle-Pere) et al. 2019), double descen! (Belkin et al.] 2019; Nakkiran et al. 2020), and Sharpness-Aware Min-
imization (Foret et al. 2021). We discuss these and other connections in Appendix
2 SETUP AND EXPERIMENTAL METHODOLOGY
Though their influence on aggregate metrics is non-obvious, identifying opposing signals is straight- forward. When training a network with GD, we track its loss on each individual training point. For a given iteration, identify the training points whose loss exhibited the most positive and most nega- tive change in the preceding step (there is large overlap between these sets in successive steps). This set will sometimes contain multiple opposing signals, which distinguish via visual inspection. This last detail means that the images we depict are not random. but we emphasize that it would not be correct to describe this process as cherry-picking signals consistently obey the maxim "I know it when see it"" To demonstrate this fact, Appendix HH contains the pre-inspecte insppecpettonn samples for a ResNet-18, VGG-11, and Vision Transformer at sev- eral training steps and for multiple seeds we believe the implied groupings are immediate, even not totally objective. We see algorithmic approaches to automatically clustering these samples as a direction for future study-for example, one could select samples by correlation in their loss time-series, or by gradient alignment.
though precise quantification is difficult, these
Given how these samples were selected, several other characterizations seem relevant. For instance, maximal one-step loss change is often a reasonable proxy for maximum gradient norm; we could also consider the largest eigenvalue of the loss Hessian of the individual point, or how much curva- ture it has in the direction of the overall loss's top eigenvector. For large networks these options are far more compute-intensive than our chosen method, but we can evaluate them post-hoc on specific groups. In Fig. 2 we track these metrics for several opposing group pairs and find that they are consistently much larger than that of random samples from the training set.
On the Possibility of Formal Definition. Though the features and their exemplar samples immediately recognizable, we do not attempt to exactly define a "feature", nor an "outlier" with respect to that feature. The presence of particular feature is often ambiguous, and it is difficult
Gradient Norm
Top Eipenvalue of Individal Loss
Curvorare On Top Pall Loss Eigeneecter
Signol
Signal
Signal 2
Signul
Randkm
2300
Doccation
Toeation
Toeration
Figure 2: Tracking other metries which characterize outliers with opposing signals. For com- putational considerations, we select outliers according maximal per-step change in loss. However, this quantity relates to other useful metrics, such as gradient norm and curvature. We see that the samples uncover are also significant outliers according these metrics.
3

---

Published as a conference paper at ICLR 2024
to define a clear threshold for what makes given point an outlier Thus, instead of trying to exactly partition the data, we simply note that these heavy tails exist and we use the most obvious outliers as representatives for visualization. In Figs. Figs.Tand[e [ we choose arbitrary cutoff of twenty samples per group. We also note that what qualifies as an opposing signal or outlier may vary over time. For visual clarity, Fig. depicts the loss on only the most dominant group pair in its respective training phase, but this pattern occurs simultancously for many different signals and at multiple scales throughout training. Further, the opposing signals are with respect to the model's internal representations (and the label), not the input space itself; this means that the definition is also property of the architecture. For example, following [Cohen et al. (202T) we train small MLP to fit a Chebyshev polynomial on evenly spaced points in the interval [I, (Fig. 36 in the Appendix). This data has no "outliers" in the traditional sense, and is not immediately clear what opposing signals are present. Nevertheless, we observe the same alternating behavior: we find a pair where one group is a small interval of x-values and the opposing group contains its neighbors, all in the range -0.5.. This suggests that the network has internal activations which are heavily influential only for more negative v-values. In this context, these two groups are the outliers.
3 UNDERSTANDING THE EFFECT OF OPPOSING SIGNALS
Our eventual goal will be to derive actionable insights from this finding. To do this, it is necessary to gain a better understanding of how these opposing signals lead to the observed behavior. this section we give simplified "mental picture" which serves as our current understanding this process: first a general discussion of why opposing signals are so influential, followed by a more mechanistic description with a toy example. This explanation is intentionally high-level, but will eventually see how it gives concrete predictions of specific hehaviors, which we then verify on real networks. Finally, we prove that this behavior occurs two-layer linear network under simple model.
3.1 PROGRESSIVE SHARPENING, AND WHY THESE OUTLIERS ARE SO INFLUENTIAL
At a high level, most variation in the input is unneeded when training a network to minimize predic- tive error- particcaallllarly with depth and high dimension. only a small fraction of information will be propagated to the last linear layer (Huh et al. 202I). Starting from random initialization, training a network aligns adjacent layers' singular values (Saxe et al. 2013 Mulayoff & Michaeli 2020) to amplify meaningful signal while downweighting noise growing sensitivity to the important signal. This sensitivity can be measured, for example, by the spectral norm of the input-output Jacobian, which grows during training (Ma & Ying. 2021]; it has also been connected to growth in the norm of the output layer (Wang et al.] (022). Observe that with this growth, small changes to how the net- work processes inputs become more influential. Hypothetically, small weight perturbation could massively increase loss by redirecting unhelpful noise to the subspace to which the network is most sensitive, by changing how the last layer uses it. The increase of this sensitivity thus represents precisely the growth of loss Hessian spectrum, with the strength of this effect increasing with depth Wang et al. 2016 Du et al. 2018 Mulayoff & Michaeli 2020
Crucially, this sharpening also depends on the structure of the input. If the noise is independent of the target, it will be downweighted throughout training. In contrast, genuine signals which oppose each other will be retained and perhaps even further amplified by gradient descent; this is because the "correct" feature may be much smaller in magnitude (or not yet learned), so using the large, "in- correct" feature is often the most immediate way of minimizing loss. As a concrete example, observe that a randomly initialized network will lack the features required for the subtle task of distinguish- ing birds from planes. But it will capture the presence of sky, which is very useful for reducing loss on such images by predicting the conditional p(lasss sky) (this is akin to the "linear/shallow -first" behavior described by Nakkiran et al. (2019); ; Mangalam & Prabhu (2019)). Thus, any method at- tempting to minimize loss as fast as possible (e.g., steepest descent) may actually upweight these
In the case of language-wlan where tokenization is discrete and more interpretable- precise definition is sometimes possible, such as those depicted in Appendix [
II this discussion we use the term "noise" informally. We refer necessarily to pure randomness, but more generally to input variation which is least useful in predicting the target.
'The coincident growth of these two measures was previously noted by Ma & Ying 2002 Gamba et al. 2023); Mac et al. (2023), though they did not make explicit this connection to how the network pro- cesses different types of input variance.
4

---

Published as a conference paper at ICLR 2024
loss on plans loss on o oterr loss on outliers inliers grudicnt steps
projection
weight space
p(plane | sky)
plother aky) ml
piplone sky) r
plotter sky) m1
Figure 3: A toy example illustrating the effect of opposing signals. Images with many blue pixels cause large activations and steep gradients. We project the loss to the hypothetical weight-space dimension "sky= plane". Left: Early optimization approaches the minimum, balancing the two opposing losses. Progress continues through this valley, further aligning subnetworks and growing the linear head. Right: The valley sharpens and iterates become unstable. Because these images a small minority and the others are less sensitive to this axis, the train loss is not noticeably affected first. Eventually either (a) the outlier gradients' growth forces the network downweight "sky", flattening the valley and returning to the first phase: or (b) the weights "catapult" to different basin.
features. Furthermore, amplified opposing signals will cause greater sharpening than random noise, because using signal to the benefit of one group is maximally harmful for the other- confi- dently predicting plane whenever there is sky will cause enormous loss on images of other classes with sky. Since random noise is more diffuse, this effect is less pronounced.
This description is somewhat abstract. To gain a more precise understanding, we illustrate the dynamics more explicitly toy example.
3.2 ILLUSTRAIING WITH HYPOTHETICAL EXAMPLE OF GRADIENT DESCENT
Consider the global loss landscape of a neural network: this is the function which describes how the loss changes as we move through parameter space. Suppose we identify a direction in this space which corresponds to the network's 's use of the "sky" feature to predict pl plane versus some other class. That is, we will imagine that whenever the input image includes bright blue background, moving the parameters in one direction increases the logit of the p ane class and decreases the oth- ers, and vece-versa. We will also decompose this loss among images with sky background, consider separately the loss on those labeled plane versus those with any other label. Because the sky feature has large magnitude, small change in weight space will induce large change in the network outputs i.e., small movement in the direction "sky plane" will greatly increase loss on these non-pl son-plane ane images.
Fig. 3 depicts this heavily simplified scenario. Early in training, optimizing this network with GD will rapidly move towards the minimum along this direction. In partieular, until better features are learned, the direction of steepest descent will lead to a network which upweights the sky feature and predicts p(class sky) whenever occurs. Once sufficiently close to the minimum, the gradient will point "through the valley" towards amplifying the more relevant signal (Xing et al., 2018). However, this will also cause the sky feature to grow in magnitude- as well as its potential influence under selective perturbation, as described above. Both these factors contribute to progressive sharpening.
Here we emphasize the distinction between the loss on the outliers and the full train loss. As images without sky (which comprise the majority of the dataset) are not nearly sensitive to movement along this axis, the global loss landscape may not at first be significantly affected. Continued op- timization will oscillate across the minimum with growing magnitude, but this growth may not be immediately apparent. Furthermore, progress orthogonal to these oscillations need not be af- fected-we find some evidence that these two processes occur somewhat independently, which present in Section 4 Returning to the loss decomposition, we see that these oscillations will cause the losses to grow and alternate, with one group having high loss and then the other. Eventually the outliers' loss increases sufficiently and the overall loss spikes (Fig. T), either flattening the valley or "catapulting" different basin (Wu et al. 2018 Lewkowycz et
2020 Thilak al. 2022).
Verifying toy example's predictions. Though this explanation lacks precise details, it does enable concrete predictions of network behavior during training. Fig. 4 tracks the predictions of a ResNet- 18 image of sky-to eliminate possible confounders, create a synthetic image
5

---

Published as a conference paper at ICLR 2024
lqqar leuge
Casc Logits
Class Proluhiiiics Daring Quolliion
Featore Nom
for 27
Her
last Teage
Mius Dver Data
Bie 29
hee 30
10-
5-
30.00
Figure 4: Passing a sky-colored block through ResNet during GD precisely tracks the predic- tions of our toy example. Left: In the first phase, the network rapidly learns to use the sky feature minimize loss. As signal is amplified, so too is the sky-colored input, and oscillation begins as de- picted in Fig. 3 Middle: During oscillation, gradient steps alternate along the axis "sky = plane" (and bit ship). Right: The initial phase and oscillation amplifies the sky input, as evidenced by the rapid growth in feature norm. The network then slowly learns to downweight this feature and rely on other signal (average feature norm provided for comparison).
single color block. Though the "plane vs. other" example seems almost too simple, we see ex- actly the described behavior-initial convergence to the minimum along with rapid growth feature norm, followed by oscillation in class probabilities. Over time, the network learns to use other signal and downweights the sky feature, as evidenced by the slow decay in feature norm. We reproduce this figure for many other inputs and for VGG- -1-BN in Appendix C with similar findings.
Our example also suggests that oscillation serves as a valuable regularizer that reduces reliance on easily learned opposing signals which may not generalize. When signal is used to the ben- efit of one group and the detriment of another, the disadvantaged group's ' gradient will have larger magnitude, and the network will be encouraged to downweight the feature. In Appendix C.3 reproduce Fig. 4 with a VGG-11-BN trained with a very small learning rate to closely approximate gradient flow. We see that gradient flow and GD are very similar until reaching the edge of stability. After this point, the feature norm under GD begins to slowly decay while oscillating, in contrast, in the absence of oscillation, the feature norms of opposing signals under gradient flow grow contin- uously. If opposing signals represent "simple" features which generalize worse, this could help to explain the poor generalization of gradient flow. A similar effect was observed by Dastrzębski et al.) (2020), who noted that large initial learning rate leads to better-conditioned loss landscape later.
3.3 THEORETICAL ANALYSIS OF OPPOSING SIGNALS IN A SIMPLE MODEL
To demonstrate this effect formally, we study misspecified linear regression on inputs x E I2 with two-layer linear network. Though this model is simplified, it enables preliminary insight into the factors we think are most important for these dynamics to occur. Since we are analyzing the dynamics from initialization until the stability threshold, it will be sufficient to study the trajectory of gradient flow-for reasonable step sizes 7. a similar result then follows for gradient descent. Our analysis reproduces the initial phase of quickly reducing loss on the outliers, followed by the subsequent growth in sensitivity to the way the opposing signal is used-i.e., progressive sharpening. We also verify this pattern (and the subsequent oscillation, which we do not formally prove) in experiments on real and synthetic data in Appendix F
Model. We model the observed features as a distribution over x E Rd,, assuming only that its co- variance E' exists-for clarity we treat Σ- = I in the main text. We further model vector vo R representing the opposing signal, with d2 > d1. We will suppose that on some small fraction of out-
a 1 1}} (1 the all-ones vector) for some a which governs the fea- liers p < 1, vo Unif {+v pdg
ture magnitude, and we let xi, = O on the remainder of the dataset. We model the target as the func- tion y = 33 x Va, | this captures the idea that the signal x, correlates strongly with the tar- get, but in opposing directions of equal strength. Finally, we parameterize the network with vectors b E IR bo E da and scalar e in one single vector 6, as fa = c- c- x x b,_ xo). Note the specific distribution of 2. is unimportant for our amalysis: only the mean and variance matter. Furthermore, in our simulations we observed the exact same behavior with cross-entropy loss. Our results suggest that depth and small signal-to-noise ratio are the only elements needed for this behavior arise.

---

Published as a conference paper at ICLR 2024
Setup. A standard initialization would be to sample [b, NO0, +2z I). which would then imply highly concentrated distributions for the quantities of interest. As tracking the pre- cise concentration terms would not meaningfully contribute to the analysis. we simplify by di- rectly assuming that at initialization these quantities are equal to their expected order of magnitude: [|bll2 = 1 ing that both layers have the same norm. We perform standard linear regression by minimizing the population loss L(0)- == (x) g) We see that the minimizer of this objective has b,- and cb = 3. However, an analysis of gradient flow will elucidate how depth and strong opposing signals lead sharpening as this minimum is approached.
di d= 11b.112 1 1bbb- d1
d, and b 3 |
vdi+ Likewise, we let c = 1, ensur-
Results. In exploring progressive sharpening, Cohen et al. (02I)) found that sometimes the model would have a brief decrease in sharpness, particularly for the square loss. In fact, this consistent with our above explanation: for larger a and a sharper loss (e.g. the square loss), the network will initially prioritize minimizing loss on the outliers, thus heavily reducing sharpness. Our first result proves that this occurs in the presence of large magnitude opposing signals:
Theorem 3.1 (Initial decrease in sharpness). Let k: := 41 and assume || > max vai+ds 21 Ar
initialization, the sharpness (2)(0)] 2 lies in (a, 30). Further, if Vā = 2211/1 In k). then both | and the overall sharpness will decrease as Ö at) from 0 until some time t S
In 191/
2|3
Proofs can be found in Appendix G After this decrease, signal amplification can proceed but this also means that the sharpness with respect to how the network uses the feature zo will grow, small perturbation to the parameters b, will induce a large increase in loss.
Theorem 3.2 (Progressive sharpening). If vā==5p
+ ||8||* k). then at starting at time t1 the
sharpness will increase linearly in [|,81| until some time t2 2 2 reaching at least 11 This lower bound on sharpness applies each dimension of ba-
क bo will start Oscillation will not occur during gradient flow-but for SGD with step size 1)
16
to increase in magnitude while oscillating across the origin. If this growth continues, it will rapidly reintroduce the feature, causing the loss on the outliers to grow and alternate. Such reintroduction (an example of which occurs around iteration 3000 in Fig. 4 seems potentially helpful for exploration. In Fig. 40 in the Appendix we simulate our model and verify exactly this sequence of events.
3.4 ADDITIONAL FINDINGS
Sharpness often occurs overwhelmingly in the first few layers. In Appendix D we track what fraction of curvature lies in each layer of various networks during training with GD. In ResNet- 18, sharpness occurrs almost exclusively in the first convolutional layer after the first few training steps; the same pattern appears more slowly while training VGG-11. In Vision Transformer curvature occurs overwhelmingly in the embedding layer and very slightly in the earlier MLP projection heads. GPT-2 (Radford et al. 2019) follows the same pattern, though with less extreme concentration in the embedding. Thus it does seem to be the case that earlier layers have the most significant sharpness- especially if they perform dimensionality reduction have particular influence over how the signal is propagated to later layers. This seems the likely cause of large gradients in the early layers of vision models (Chen al., 2021 Kumar et al. 2022). suggesting that this effect equally influential during finetuning and pretraining and that further study can improve optimization.
Batchnorm may smooth training, even if not the loss itself. Cohen et al. (202T) noted that batch- norm (BN) (loffe& & Szegedy, (015)) does not prevent networks from reaching the edge of stability and concluded, contrary to Santurkar et al. (2018), that BN does not smooth the loss landscape. We conjecture that the benefit of BN may be downweighting the influence of opposing signals and mitigating this oscillation. In other words, BN may smooth the optimization trajectory of neural networks, rather than the loss itself (this is consistent with the distinction made by Cohen et al. 2 between regularity and smoothness). In Section 4 we demonstrate that Adam also smooths
'The "fraction of curvature" is with respect to the top eigenvector of the loss Hessian. We partition this vector by network layer, so each sub-vector's 's squared norm represents that layer's contribution to the overall curvature.
7

---

Published as a conference paper at ICLR 2024
VGG-11 SCID Loss on Opgnning Groups
Projection Top Rigeneector
Schee
Sti
Adam
Sodt: Pajeess
MireeLL SGD
Sto S00
Adam Effective Step Siz (x 3o-4)
20
30
40
-00
20
30
10
10
25
Ietation
Ivonto
(b)
Figure 5: Outliers with opposing signals have a significant influence even during SGD. Left: We plot the losses of paired outlier groups a VGG-11-BN trained on CIFAR-10, along with the full train loss for comparison. Modulo batch randomness, the outliers' loss follow the same oscillatory pattern with large magnitude. See appendix for the same without batchnorm. Right (top): We train a small MLP on a 5k subset of CIFAR-10 with various optimizers and project the iterates onto the top Hessian eigenvector. SGD closely tracks GD, bouncing across the valley; momentum somewhat mitigates the sharp jumps. Adam smoothly oscillates along one side. Right (bottom): Adam's effective step size drops sharply when moving too close far from the valley floor.
the optimization trajectory and that minor changes to emulate this effect can aid stochastic optimiza- tion. We imagine that the effect of BN could also depend on the use of GD vs. SGD. Specifically. findings hint at a possible benefit of BN which applies only SGD: reducing the influence of imbalanced opposing signals in random minibatch.
For both GD and SGD, approximately half of training points go up in loss each step. Though only the outliers are wildly oscillating, many more images contain some small compo- nent of the features they exemplify. Fig. 37 the Appendix shows that the fraction of points which increase in loss hovers around 50% for every step to some extent, a small degree of oscillation appears be happening to the entire dataset. Fig. [38 depicts the distribution of changes loss of individual training points at each iteration: the density closely approximates normal curve, with mean consistently just below and a standard deviation around 0.3-0.5.
THE INTERPLAY OF OPPOSING SIGNALS AND STOCHASTICITY
Full-batch GD is not used in practice when training NNs. It is therefore pertinent to ask what these findings imply about stochastic optimization. We begin by verifying that this pattern persists during SGD. Fig. {())dissayys the losses for four opposing group pairs of VGG-11-BN trained on CIFAR-10 with SGD batch size 128. We observe that the paired groups do exhibit clear opposing oscillatory patterns, but they do not alternate with every step, nor they always move in opposite directions. This should not be surprising: we expect that not every batch will have a given signal in one direction or the other. For comparison, we include the full train loss in each figure- that is, including the points not in the training batch. We see that the loss on the outliers has substantially larger variance; to confirm that this is not just because the groups have many fewer samples, we also plot the loss a random subset of training points of the same size. We reproduce this plot with VGG-11 without BN in Fig. in the Appendix.
Having verified that this behavior occurs in the stochastic setting, we conjecture that current best practices for neural network optimization owe much of their success to how they handle opposing signals. As proof of concept, will make this more precise with preliminary investigation of the Adam optimizer (Kingma & Ba 2014).
4.1 How ADAM HANDLES GRADIENTS WITH OPPOSING SIGNALS
To better understand their differences, Fig. 5(b] visualizes the parameter iterates of Adam and SGD with momentum ReL.U MIP trained on 5k subset of CIFAR-10, alongside those of GD and
8

---

Published as a conference paper at ICLR 2024
SGD (all methods use the same initialization and sequence of training batches). The top figure is the projection of these parameters onto the top eigenvector of the loss Hessian of the network trained with GD, evaluated at the first step where the sharpness crosses 2/1, We observe that SGD tracks similar path to GD, though adding momentum mitigates the oscillation somewhat. In contrast, the network optimized with Adam markedly departs from this pattern, smoothly oscillating along one side. We identify three components of Adam which potentially contribute to this effect:
Advantage 1: Smaller steps along directions with higher curvature. Adam's normalization
causes smaller steps along the top eigenvector, especially near the minimum. The lower plot in Fig. (5(b)) shows that the effective step size in this direction-i.e. the absolute inner product of the parameter-wise step sizes and the top eigenvector-rapidlly drops to zero as the iterates move or- thogonal to the valley floor. Thus normalizing by curvature parameter-wise seems crucial; similarly, Pan & Li 2023) show that parameter-wiss gradient clipping improves SGD substantially.
Advantage 2: Managing heavy-tailed gradients and avoiding steepest descent. Zhang et al. (2020) identified the "trust region" important for Adam's success, pointing to heavy-tailed noise in the stochastic gradients. More recently, Kunstner et al.] (2023) argued that Adam's superiority does not come from better handling noise, supported with large batch experiments. Our result reconciles these contradictory claims by showing that the difficulty is not heavy-tailed noise, but strong, directed (and perhaps imbalanced) opposing signals. Unlike traditional "gradient noise", larger batch sizes may not reduce this effect-ther is, the full gradient is heavy-tailed across parameters. Furthermore, the largest steps emulate Sign SGD, which is notably not a descent method. Fig. ((b) shows that Adam's 's steps are more parallel to the valley floor than those of steepest descent. Thus seems advantageous to intentionally avoid steepest descent, which might lead to an ill-conditioned landscape. This point is also consistent with the observed generalization benefits of a large learning rate for SGD on NNs (Jastrzębski et al. 2020 in fact, opposing signals naturally fit the concept of "easy-to- to-fit features as modeled by Li et al. (2019).
Advantage 3: Dampening. Traditional SGD with momentum B < 1 takes a step which weights the current gradient by 1/(++#) > 1/2. Though this makes intuitive sense, our results imply that heavily weighting the most recent gradient can be problematic. Instead, we expect an important addition is dampening. which multiplies the stochastic gradient at each step by some (1 -)) 1. We observe that Adam's (unnormalized) gradient is equivalent to SGD with momentum and dampening both equal to 31, plus debiasing step. Recently proposed alternatives also include dampening in their momentum update but do not explicitly identify the distinction (Zhang et al.,
2020
Pan Li 2023 Chen et al. 2023).
Using these insights to aid stochastic optimization. To test whether our findings translate to practical gains, we design a variant of SGD which incorporates these insights. First, we use damp- ening = 0.9 in addition to momentum. Second, we choose global threshold: if the gradient magnitude for a parameter is above this threshold, we take a fixed step size; otherwise, we take a gradient step as normal. The exact method appears in Appendix [ We emphasize that our goal here is not to propose a new optimization algorithm; we are exploring the potential value gained from knowledge of the existence and influence of opposing signals.
Results in Appendix I show that this approach matches Adam when training ResNet-56/1 10 on CIFAR-10 with learning rates for the unthresholded parameters across several orders of magnitude ranging from 10-4 to 10². Notably, the fraction of parameters above the threshold is only around 10-25% per step, and the trajectory and behavior of the network is dominated by this small fraction; the remainder can be optimized much more robustly, but their effect on the network's 's behavior obscured. We therefore see the influence of opposing signals as a possible explanation for the "hidden" progress in grokking (Barak et al. 2022 Nanda et al., 2023). We also compare this method to Adam for the early phase training GPT-2 on the Web Text dataset (Gokaslan et al. 2019)- not only do they perform the same, their loss similarity suggests that their exact trajectory may be very similar (Appendix E.I). Here the fraction of parameters above the threshold hovers around 50% initially and then gradually decays. The fact that many more parameters in the transformer above the threshold suggests that the attention mechanism is more sensitive to opposing signals and that further investigation of how to mitigate this instability may be fruitful.
9

---

Published as a conference paper at ICLR 2024
5 CONCLUSION
The existence of outliers with such a significant yet non-obvious influence on neural network train- ing raises as many questions answers. This work presents an initial investigation into their effect on various aspects of optimization, but there is still much more to understand. Though it is clear they have a large influence on training, less obvious whether reducing their influence is necessary for improved optimization or simply coincides with it. At the same time, there is evidence that the behavior these outliers induce may serve as an important method exploration and/or regulariza- tion. If so, another key question is whether these two effects can be decoupled- or if the incredible generalization ability of neural networks somehow inherently tied to their instability.
ACKNOWLEDGEMEETT
We thank Saurabh Garg for detailed feedback earlier version of this work. Thanks also to Christina Baek and Bingbin Liu for helpful comments and to Jeremy Cohen for pointers to related work. This research is supported in part by NSF awards IIS-221 1907, CCF-2238523, an Amazon Research Award, and the CMU/PwC D Center.
REFERENCES
Sanjeev Arora, Zhiyuan Li, and Abhishek Panigrahi. Understanding gradient descent the edge of stability in deep learning. In International Conference on Machine Learning, pp. 948-10282 PMLR, 2022.
Boaz Barak, Benjamin L. Edelman, Surbhi Goel, Sham M. Kakade, eran malach, and Cyril Zhang. Hidden progress in deep learning: SGD learns parities near the computational limit. In Ad- vances in Neural Information Processing Systems, 2022. htt.tttsss//ppeeeeeeee /eettreeeeeeeee/eeeeeeee
Forum?1d=6XWP2ee:.
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine- learning practice and the classical varianer trade-off. Proceedings of the National Academy of Sciences, (16(3)::584991555
2019.
Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Yao Liu, Hieu Pham, Xu- anyi Dong, Thang Luong, Cho-Jui Hsieh, et al. Symbolic discovery of optimization algorithms. arXiv preprint arXiv:2002.06675, 2023.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self- cuff-supervieed vision transformers. arXiv preprint arXiv:2104.020 020557, 2(5);6, 2021.
Jeremy Cohen, Simran Kaur, Yuanzhi Li, Zico Kolter, and Ameet Talwalkar. Gradient descent on neural networks typically occurs the edge of stability. In International Conference Learning Representations, 2021. URL
Jeremy M Cohen, Behrooz Ghorbani, Shankar Krishnan, Naman Agarwal, Sourabh Medapati, Michal Badura, Daniel Suo, David Cardoze, Zachary Nado, George E Dahl, et al. Adaptive gradient methods at the edge of stability. arXiv preprint arXiv:2207.14453 2022.
Alex Damian, Eshaan Nichani, and Jason D. Lee. Self-stabilization:: descent at the edge of stability. In OPT 2022: Optimization for Machine Learning (NeurIPS 2022 Workshop), 2022. URL aettps::/opeenreellw..tttt net wtt.ooorrmmminooo.oooo B D27
The implicit bias of gradient
Simon Du, Wei Hu, and Jason Lee. Algorithmic regularization in learning deep homogeneous models: Layers are automatically balanced. In Advances in Neural Information Processing Syy- tems, volume 31. Curran Associates, Inc., 2018
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimiza- tion for efficiently improving gemeralization. In International Conference on Learning Represen- fations, 2021. URL ht..ttts::// U http://wwweee//ooo.eeeeeeeeo forem//orrrmiiia 6m mpos rM
10

---

Published as a conference paper at ICLR 2024
Stanislav Fort and Surya Ganguli. Emergent properties of the local geometry of neural loss land- scapes. arXiv preprint arXiv:1910.05999, 2019.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural
networks. In International Conference Learning Representations, 2019. URL https://
srenreview.nete 20-
Matteo Gamba, Hossein Azizpour, and Märten Björkman. On the lipschitz constant of deep net- works and double descent. arXiv preprint 0at::2231.123099 2023.
Behrooz Ghorbani, Shankar Krishnan, and Ying Xiao. An investigation into neural net optimization via hessian eigenvalue density. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Muchine Learning Research. PMLR, 09-15 Jun 2019. URL httpts::/ppooeeeeeeeeeeeee
ml www.sp.essss..... https//////ghhh/////e/t 1tt.rrbmnnn9bbmhmmm
Aaron Gokaslan, Vanya Cohen, Ellie Pavlick, and Stefanic Tellex. Openwebtext corpus, 2019.
Minyoung Huh, Hossein Mobahi, Richard Zhang, Brian Cheung, Pulkit Agrawal, and Phillip Isola. The low-rank simplicity bias in deep networks. arXiv preprint arXiv:2103, 10427, 2021.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448-456. pmir, 2015.
Stanistaw Jastrzębski, Zachary Kenton, Devansh Arpit, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amos Storkey. Three factors influencing minima in sgd. arXiv preprint arXiv: 2017.
Stanistaw Jastrzębski, Zachary Kenton, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amost Storkey. On the relation between the sharpest directions of DNN loss and the SGD step length. In International Conference on Learning Representations, 2019. URL /oLLhtttpp::/ppeeevviee.
Stanislaw Jastrzębski, Maciej Szymezak, Stanislav Fort, Devansh Arpit, Jacek Tabor, Kyunghyun Cho and Krzysztof Geras The break-even point on optimization trajectories of deep neural networks. In International Conference on Learning Representations, 2020. URL https://
sneerrview.net.o http://www...eeoooomo//ooo//o///
Stanistaw Jastrzębski, Devansh Arpit, Oliver Astrand, Giancarlo Kerg, Huan Wang, Caiming Xiong. Richard Socher, Kyunghyun Cho, and Krzysztof J Geras. Catastrophic fisher explosion: Early phase fisher matrix impacts generalization. In Proceedings of the 38th International Con- ference Machine Learning, Proceedings of Machine Learning Research. PMLR, 2021. URL
https: / http/ceeeii...ccpcp/p/ppppe mlr press / ////////////////////////// wtt33//asttzzeeel22.......
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXiv:1412.6980 2014.
Dmitry Kopitkov and Vadim Indelman. Neural spectrum alignment: Empirical study. In Artificial Neural Networks and Machine Learning- 2020: 29th International Conference on Artifi- cial Neural Networks, Bratislava, Slovakia, September 15-18, 2020, Proceedings, Part II 29, pp- 168-179. Springer, 2020.
Itai Kreisler, Mor Shpigel Nacson, Daniel Soudry, and Yair Carmon. Gradient descent monotoni- cally decreases the sharpness of gradient flow solutions in scalar networks and beyond. In Pro- ceedings of the 40th International Conference on Machine Learning, Proceedings of Machine Learning Research. PMLR, 23-29 Jul 2023. WRLLhttps:///ooot
ml mlr..ress/
k202//reisler225 html
Ananya Kumar, Ruoqi Shen, Sébastien Bubeck, and Suriya Gunasckar. How to fine-tune vision models with sgd, arXiv preprint arXiv:22/1 1.09359, 2022.

---

Published as a conference paper at ICLR 2024
Frederik Kunstner, Jacques Chen, Jonathan Wilder Lavington, and Mark Schmidt. Noise is not the main factor behind the gap between sgd and adam on transformers, but sign descent might be. In The Eleventh International Conference Learning Representations, 2023. URL https:
//openreviev. net. formmm d d=a65YK0cqHHgy
Aitor Lewkowycz, Yasaman Bahri, Ethan Dyer, Jascha Sohl-Dickstein, and Guy Gur-Ari. The large learning rate phase of deep learning: the catapult mechanism. arXiv preprint arXiv:2003.022 2020.
Xinyan Li, Qilong Gu, Yingxue Zhou, Tiancong Chen, and Arindam Banerjee. Hessian based analysis of sgd for deep nets: Dynamics and generalization. In Proceedings of the 2020 SIAM International Conference on Data Mining, pp. 190-198. SIAM, 2020.
Yuanzhi Li, Colin Wei, and Tengyu Ma. Towards explaining the regularization effect of initial large learning rate in training neural networks. Advances in Neural Information Processing Systems, 32, 2019.
Chao Ma and Lexing Ying. On linear stability of sgd and input-smoothness of neural networks In Advances in Neural Information Processing Systems, volume 34, pp. 16805-16817. Cur- ran Associates, Inc., 2021.
http://ss//proceeddnnnge..errrrs paper/2021/Eilee / 8c2 622 2ttp:///e..2..0000333ooo2.2222a0
Chao Ma, Daniel Kunin, Lei Wu, and Lexing Ying. Beyond the quadratic approximation: the
multiscale structure of neural network loss landscapes. arXiv preprint arXiv:2204.11336, 2022.
Lachlan Ewen MacDonald, Jack Valmadre, and Simon Lucey. On progressive sharpening, flat min- ima and generalisation. arXiv preprint 2rXiv:2305. 2023.
Karttikeya Mangalam and Vinay Uday Prabhu. Do deep neural networks learn shallow learnable examples first? In ICML 2019 Workshop on Identifying and Understanding Deep Learning Phenomena, 2019. http::htt/p///ttil
wwtee/fooumm/ id- -ib HkxHv4rn24
William Merril,, Nikolaos Tsilivis, and Aman Shukla. A tale of two circuits: Grokking as competi- tion of sparse and dense subnetworks. In ICLR 2023 Workshop on Mathematical and Empirical Understanding of Foundation Models, 2023. URL https: hwtp://///ppeeeee..eoooo net forum?
1d=8GZxtu4 KKx
Rotem Mulayoff' and Tomer Michacli. Unique properties of flat minima in deep networks. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceed- ings of Machine Learning Research. PMLR, 13-18 Jul 2020. URL htLptttts////roceeeee
mlr. wttes///119/mmmayoffee.t
Preetum Nakkiran, Gal Kaplun, Dimitris Kalimeris, Tristan Yang, Benjamin L Edelman, Fred Zhang, and Boaz Barak. Sgd on neural networks learns functions of increasing complexity. arXiv preprint arXiv: 1104, 2019.
Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang. Boaz Barak, and Ilya Sutskever. Deep double descent: Where bigger models and more data hurt. In International Conference on Learn- ing Representations, 2020.
net. fereeffoormm http://orummidiiemee 2ttee//fooommmlslsssssstttt
Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability.. In The Eleventh International Conference on Learn- ing Representationss 2023. /tthhtpss:/ pttttpss://ppenrevieee net. Fowoorrmmd=iisaaaaa
Pascal Notsawo Jr, Hattie Zhou, Mohammad Pezeshki, Irina Rish, Guillaume Dumas, et al. Predict- ing grokking long before happens: A look into the loss landscape of models which grok. arXiv preprint arXiv:2306. 00222306333300 13253, 2023.
Yan Pan and Yuanzhi Li. Toward understanding why adam converges faster than sgd for transform- ers. arXiv preprint trxiiv2306.002200 2023.
Vardan Papyan. The full spectrum of deepnet hessians at scale: Dynamics with sgd training and sample size. arXiv preprint arXiv: (00::////.00000 1.07062, 2018.
12

---

Published as a conference paper at ICLR 2024
Vardan Papyan. Measurements of three-level hierarchical structure in the outliers in the spectrum of deepnet hessians. In Proceedings of the 36th International Conference on Machine Learn- ing, volume 97 of Proceedings of Machine Learning Research. PMLR, 09-15 Jun 2019. URL ha.ml
wtw.eeeeeeiigi.e...ooo ngs m press httT:s/////p/p////
Vardan Papyan. Traces of class/cross-class structure pervade deep learning spectra. The Journal of Machine Learning Research, 21(1):10197-10260, 2020.
Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Gen- cralization beyond overfitting small algorithmic datasets. arXiv preprint arXiv:2201.02177, 2022.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, lly Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Elan Rosenfeld, Pradeep Ravikumar, and Andrej Risteski. Domain-adjusted regression or: Erm may already learn features sufficient for out-of-distribution generalization. arXiv preprint arXiv:2202.06836. 2022.
Levent Sagun, Leon Bottou, and Yann LeCun. Eigenvalues of the hessian in deep learning: Singu- 2016.
larity and beyond. arXiv preprint
Levent Sagun, Utku Evei, Ugur Guney, Yann Dauphin, and Leon Bottou. Empirical analysis of 2017.
the hessian of over-parametrized neural networks. arXiv preprint
Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Madry. How does batch normal- ization help optimization? Advances in neural information processing systems, 31, 2018
Andrew M Saxe, James L McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynam- ics of learning in deep linear neural networks. arXiv preprint arXiv:1312.6133 2013.
Harshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateck Jain, and Praneeth Netrapalli. The pitfalls of simplicity bias in neural networks. Advances in Neural Information Processing Systems, 33:9573-9583 2020.
Vimal Thilak, Etai Littwin, Shuangfei Zhai, Omid Saremi, Roni Paiss, and Joshua Susskind. The slingshot mechanism: An empirical study of adaptive optimizers and the grokking phenomenon. arXiv preprint 220.1:2200604817 2022.
Guillermo Valle-Perez, Chico Q. Camargo, and Ard A. Louis. Deep learning generalizes be- cause the parameter-function map is biased towards simple functions. In International Confer- ence on Learning Representations, 2019. URL https:/ /ttt::://ppenevvee..n. wtt.//forrmmiidoooo
Iye4ggAqFm
Shengjie Wang, Abdel-rahman Mohamed, Rich Caruana, Jeff Bilmes, Matthai Plilipose, Matthew Richardson, Krzysztof Geras, Gregor Urban, and Ozlem Aslan. Analysis of deep neural networks with extended data jacobian matrix. In International Conference on Machine Learning, pp. 718- 726. PMLR, 2016.
Zixuan Wang, Zhouzi Li, and Jian Li. Analyzing sharpness along GD trajectory: Progressive sharp- ening and edge of stability. Advances in Neural Information Processing Systems, 2022. URL net For
Kaiyue Wen, Tengyu Ma, and Zhiyuan Li. Sharpness minimization algorithms do not only minimize sharpness to achieve better generalization. arXiv preprint arXiv:2307.11000 2023.
Lei Wu, Chao Ma, and Weinan E. How sgd selects the global minima in over- parameterized learning: A dynamical stability perspective. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL
13

---

Published as a conference paper at ICLR 2024
Lei Wu, Mingze Wang. and Weijie J Su. The alignment property of SGD noise and how it helps select flat minima: A stability analysis. In Advances in Neural Information Processing Systems, 2022. URL http:///.eneeeeeeeee/eeee wttp://oooruuaadaattesssll.....
Chen Xing, Devansh Arpit, Christos Tsirigotis, and Yoshua Bengio. A walk with sgd. arXiv preprint arXiv:1802.08770, 2018.
Jingzhao Zhang, Sai Praneeth Karimireddy, Andreas Veit, Seungyeon Kim, Sashank Reddi, Sanjiv Kumar, and Suvrit Sra. Why are adaptive methods good for attention models? Advances in Neural Information Processing Systems, 33:15383-15393, 2020.
Xingyu Zhu, Zixuan Wang. Xiang Wang. Mo Zhou, and Rong Ge. Understanding edge-of- stability training dynamics with minimalist example. The Eleventh International Confer- Learning Representations, 2023. URL https:/ /wwp///eppeeee..eee//e/eeee net. forum?id= /EagBaMAAO,
Zhanxing Zhu, Jingfeng Wu, Bing Yu, Lei Wu, and Jinwen Ma. The anisotropic noise in stochas- tic gradient descent: Its behavior of escaping from sharp minima and regularization effects. In Proceedings of the 36th International Conference on Machine Learning, Proceedings of Machine Learning Research, pp. 7664-7663. PMLR, 09-15 Jun 2019, URL htLt::ts::///ttt httcttpps//rroeednggs...
mlr. www.speessvv. v97 zhul 9e html
14

---

Published as a conference paper at ICLR 2024
A RELATED WORK AND DISCUSSION
Many of the observations we make in this paper are not new, having been described in various prior works. Rather, this work identifies a possible higher-order cause which neatly ties these findings together. There are also many works which pursue a more theoretical understanding of each of these phenomena independently. Such analyses begin with set of assumptions (on the data, in particular) and prove that the given behavior follows. In contrast, this work begins by identifying condition the presence of opposing signals-whie we argue likely a major cause of these behaviors. These two are not at odds: we believe in many cases our result serves as direct evidence for the validity of these modeling assumptions and that it may enable even more fine-grained analyses. This work provides an initial investigation which we hope will inspire future efforts towards more complete understanding.
Characterizing the NN loss landscape. Earlier studies of the loss landscape extensively explored the spectrum of the Hessian of the loss, with common observation being a hierarchical structure
with a small group of very large outlier eigenvalues (Sagun et al. 2016 2017: Papyan 2018: 2019
Fort & Ganguli 2019; Ghorbani et al. 2019 Li et al. 2020 Papyan [020] Kopitkov & Indelman 2020). Later efforts focused on concretely linking these observations to corresponding behavior, often with emphasis on SGD's bias towards particular solutions (Wu et al., 2018 Jastrzębski
cet al. 2017 2020) and what this may imply about its resulting generalization (Jastrzębski et al., Zhu et al. [2199] Wu et al. (022). Our method for identifying these paired groups, along with Fig. indicates that this outlier spectrum is precisely the directions with opposing signals in the gradient, and that this pattern may be key to better understanding the generalization ability of NNs
2019
trained with SGD.
Progressive sharpening and the edge of stability. Shifting away from the overall structure, more recent focus has been specifically on top eigenvalue(s), where it was empirically observed that their
magnitude (the loss "sharpness") grows when training with SGD (Jastrzebski et al.] 2019: 2020)
and GD (Kopitkov & Indelman 2020 Cohen et al., 2021) (so-called "progressive sharpening"
This leads to rapid oscillation in weight space (Xing et al., 2018 Jastrzębski et al., 2019| Cohen et al., |021 2022). [Cohen et al. (2021) also found that for GD this coincides with consistent yet non-monotonic decrease in training loss over long timescales, which they named the "edge of stability";; moreover, they noted that this behavior runs contrary to traditional understanding of NN convergence. Many works have since investigated the possible origins of this phenomenon (Zhu et al. 2023 Kreisler et al. 2003);; several are deeply related our findings. Ma al. (2022) connect this behavior to the existence of multiple "scales" of losses; the outliers we identify corroborate this point. Damian et al. (2022) prove that GD implicitly regularizes the sharpness- we identify a conceptually distinct source of such regularization, as described in Section 3 Arora et al. (2022) show under some conditions that the GD trajectory follows a minimum-loss manifold towards lower curvature regions. This is consistent with our findings, and we believe this manifold to be precisely the path which evenly balances the opposing gradients. Waan et al. (2022) provide another thorough analysis of NN training dynamics at the edge of stability: their demonstrated phases closely align with our own. They further observe that this sharpening coincides with a growth in the norm of the last layer, which was also noted by MacDonald et al. (2023). Our proposed explanation for the effect of opposing signals offers some insight into this relationship.
Roughly, our results seem imply that progressive sharpening occurs when the network learns to rely on (or not rely on) opposing signals in a very specific way, while simultaneously amplifying overall sensitivity. This growth in sensitivity means small parameter change modifying how op- posing signals are used can massively increase loss. This leads to intermittent instability orthogonal to the "valley floor", accompanied by gradual training loss decay and occasional spikes as described by the toy example in Fig. 3 and depicted on real data in Fig.
Generalization, grokking, slingshotting, and subnetworks. We see ties between opposing sig- nals and many other phenomena in NN optimization, but a few specific concepts stand out to us as most clearly related. At least in images, the features which provide opposing signals match the traditional picture of "spurious correlations" surprisingly closely- we conjecture that the factors af- fecting whether a network maintains balance or diverges along a direction also determine whether it continues to use "spurious" feature or is forced find an alternative way to minimize loss.
15

---

Published as a conference paper at ICLR 2024
Indeed, the exact phenomenon of network "slingshotting" to new region has been directly ob- served, along with a corresponding change in generalization (Wu et al., 2018 Lewkowyez et al. 2020 Jastrzebski et al. 202T; Thilak et al., 2022). "Grokking" (Power al. 2002), whereby net- work learns to generalize long after memorizing the training set, is another closely related area of study. Several works have shown that grokking is a "hidden" phenomenon, with gradual amplifica-
tion of generalizing subnetworks (Barak et al. 2022 Nanda et al. [023] Merrill et al. 2023); it has even been noted co-occur with weight oscillation (Notswwo Jr et al. 2023). We observe that the opposing signal gradients dominate the overall gradient for most of training and our experiments on SGD in Section and Appendix[E]er two different scales, obscured by the network's behavior on outliers. We note that this also relates to the Lottery Ticket Hypothesis (Frankle & Carbin 2019): we think it not unlikely that a pruned winning ticket is one which is not as influenced by opposing signals, allowing it to more easily tra- verse the loss landscape. In fact, said ticket may already be following this trajectory during standard optimization- buttiioon-uuuuuuuuu the behavior (and loss) of the remainder of the network on these opposing signals would obscure its progress.
some further evidence that NN optimization occurs on
Simplicity bias and double descent. Nakkiran al. (0199) observed that NNs learn functions of increasing complexity throughout training. Our partincutttaattclllalyyyy the slow decay in the norm of the feature embedding of opposing signals-lead us believe would be more correct to say that they unlearn simple functions, which enables more complex subnetworks with smaller magnitude and better performance to take over. At first this seems odds with the notion of simplicity bias (Valle-Perez et al.) 2019; (hhah et al., 2020), defined broadly as a tendency of networks to rely on simple functions of their inputs. However, it does seem to be the case that the network will use the simplest (e.g.. largest norm) features that it can, so long such features allow it to approach zero training loss; otherwise it may eventually diverge. This tendency also suggests possible explanation for double descent (Belkin et al. 2019; Nakkiran et al.) 2020): even after interpolation, the network pushes towards greater confidence and the weight layers continue to balance (Saxe et al., 2013 Du et al. 2018), increasing sharpness. This could lead to oscillation,
pushing the network to leamn new features which generalize better (Wu et al. 2018 Li et al.) 2019 Rosenfeld et al. 2022 Thilak et al.] 2022). This behavior would also be more pronounced for larger networks because they exhibit greater sharpening. Note that the true explanation is not quite so straightforward: generalization is sometimes improved via methods that reduce oscillation (like loss smoothing), implying that this behavior is not always advantageous. A better understanding of these nuances is an important subject for future study.
Sharpness-Aware Minimization Another connection we think merits further inquiry is Sharpness-Aware Minimization (SAM) (Foret et al., (021). which is known to improve general- ization of neural networks for reasons still not fully understood (Wen et al.) (023). In particular, the better-performing variant is 1-SAM, which takes positive gradient steps on each training point in the batch individually. It it evident that several of these updates will point along directions of steep- est descent/ascent orthogonal to the valley floor (and, if not normalized, the updates may be very large). Thus it may be that 1-SAM is in some sense "simulating" oscillation and divergence out of this valley in both directions, enabling exploration in a manner that would not normally be possible until the sharpness grows large enough these intermediate steps would also encourage the network to downweight these features sooner and faster. In contrast, standard SAM would only take this step in one of the two directions, or perhaps not at all if the opposing signals are equally balanced. Furthermore, unlike 1 1-SAM the intermediate step would blend together all opposing signals in the minibatch. These possibilities seem a promising direction for further exploration.
16

---

Published as a conference paper at ICLR 2024
B EXAMPLES OF OPPOSING SIGNALS IN TEXT
Punctuation Ordering
the EU is \the best war-avoidance mechanism invented[").
because it was one of the few that still (dry-farmed(")
Re describes the taste as \almost minty["].
did receive several offers to Ahelp out bit[").
Nor is it OK to say \the real solution is in technological breakthrough(")
and that's s what they mean by \when complete["]. and that the next big investment bubble to burst is the \carbon bubble["].
he had been \driven by ideological and political motives("]. Prime Minister Najib Razak's persenal bank account was a \genuine conation!"].
exceptional intellect, unparalleled integrity, and record ef independence(")
was the \most consequential decision I've ever been invelved wi]
which some lawmakers have called the \filibuster of all filibusters"").
Democrats vowed to filibuster what openly called \stelen seat
His leather belt usually the delivery method of choice. ["] \it's laborious and boring. loved excitement and attention. ["] 'That little son-of-a-gun is playing favorites,' and turned it against him.[")
\For medicinal purposes, for medical purposes, absolutely, it's fine.
\That's a choice that growers make. It's on their side the issue.
to be good steward of your land. You have to make big decisions in hurry.
\But of course modern farming looks for maximum yield no what you have to put in. And the case of california, that input is water.
And we have been drawing down centuries of accumilation. Pretty soon those systems are not going be able to provide for us. not a luxury crop like wine. I'm really excited ab out the potential.
\e knew and still believe that it was the right thing to de. ["] spending lots of time in the wind tunnel, because it shows when test them. ["]
Compliance was low on the list, but I think it's pretty comfortable bike. ["]
the opportunity to do that.' just needed to take it and run with it. ["]
Figure 6: Examples of opposing signals in text. Found by training GPT-2 on a subset of Web- Text. Sequences are separate lines, the token in brackets is the target and all prior tokens are (the end of the) context. As both standards are used, before or after the end of quotation (we include the period after the quote for clarity-the model does not condition on it). Note that the double quotation is encoded as the pair of tokens [447, 251] 1, and the loss oscillation is occurring for sequences that end with this pair, either before (top) or after (bottom) the occurrence of the period token (13).
not always clear whether punctuation will come
17

---

Published as a conference paper at ICLR 2024
New Line or 'the' After Colon
In order to prepare your data, there are three things te de: [No]
in the FF 1ib of your choice, namely Scalaz cats. It lccks like this: [\n]
Salcedo said of the work: []n] Enter your email address: [[n] According to the CBO update: Here's s 1ist of 5 reasons successive Lambda invocations. It looks more or less like this:[|n)
why self diagnosis is valid: [n]
data, there are three things to do: do:[(n] [[]]
4.2 percent in early 2018. inlnAccording to the C8O update: [[n] other than being myself. "[n|TAATCH: [n]
is to make the entire construction plural, Joural.n//Iwo In|nTwo recent examples: [|]]
We offer the following talking points to anyone who is attending
the meeting: [[]]
is on the chopping block and at the worst possible moment: [\n]
as will our MPs in Westminster. But to me it is obvicus: [the] The wheelset is the same as that en the model above: [the] not get so engrained or in rut with what I had peen doing. Not to worry: [the]
robs this incredible title of precisely what makes it so wonderful; [the]
you no doubt not neticed something was missing: [the]
Neil Gorsuch's 'exist'' comments maternity leaver [the]
Figure 7: Examples of opposing signals in text. Found by training GPT-2 subset of Open-
WebText. Sequences are on separate lines, the token in brackets the target and all prior tokens are (the end of the) context. Sometimes a colon occurs mid-sentence-ann and is often followed by Sthe"-othe times announces the start new line. The model must unlearn + [[n]" versus [the]" and instead use other contextual information.
colon
"the" after colon
8
25
75
100
125
150
175
200
Iiccation
Figure 8: Loss of GPT-2 the above opposing signals.
18

---

Published as a conference paper at ICLR 2024
C REPRODUCING FIG. 4 IN OTHER SETTINGS
Though colors are straightforward, for some opposing signals such as grass texture it is not clear how to produce synthetic image which properly captures what precisely the model is latching on to. Instead, we identify a real image which has as much grass and as little else as possible, with the understanding that the additional signal in the image could affect the results. We depict the grass image alongside the plots it produced.
C.I RESNET-18 TRAINED WITH GD ON OTHER INPUTS
Imange
Class Logits
Class Protubilities During Oscillation
Festure Nerm
Over
pl.nc
30
70-
15-
-10
200
J00
500
600
Figure 9: ResNet-18 18 on red color block.
Closs Lngits
Class Profobilttiss Daring
Neem
Sapa lange
Moa: Over Daa
1000
Iter B116
lhe 817
Feg
14-00
10-
42)
1963
700
400
600
100
200
410
(00
beratton
Figure 10: ResNet-18 on a green color block. As this color seems unnatural, we've included two examples of relevant images in the dataset.
Figure 11: Examples of images with the above green color.
Ingst Inagg
Class Logies
Class Profutt liries Daring Oscilibtion
Festure Norm
Beer 538
Ber 539
17
piane
10
612
400
Figure 12: ResNet-18 18 on a white color block.
19

---

Published as a conference paper at ICLR 2024
Logits
Daring
kew Ineas
14-
Ovar Date
12-
Lasi
10-
lee 190
Ber
880)
15.05
200
100
000
900
1000
2000
5000
4000
5000
Docration
Jaraion
Figure 13: ResNet-18 on a black color block.
Daring
her 364
Her
10-
beage
660
000
Figure 14: ResNet-18 18 image with mostly grass texture.
20

---

Published as a conference paper at ICLR 2024
C.2 VGG- 11-11--BNN TRAINED WITH GD
For VGG-I1, we found that the feature norm of the embedded images did not decay nearly as much the course of training. We expect this has to do with the lack of residual component. However, for the most part these features do still follow the pattern of a rapid increase, followed by a marked decline.
Inpet Image
Closs Logits
Class Protobiiitiss During Oscilluzion
Festure Neem
Ner
Ite %2
14-
Aapha M/M
12-
10-
Ite
Ite
Figure 15: VGG-11-BN sky color block.
Imaage
Logits
Class Probubiitiie Duringg
14
Over
1000
300
1000
200
350
Danaticn
Figure 16: VGG-I1-BN on a red color block.
Closs
Profobiltiiss Daring
Neem
lee
Iee
1000
10-
Tux B1
Tie 82
100
14-00
Feg
Uegut Imuge
Mox Over
100
2003
100
100
300
100
borritionn
Figure 17: VGG-11-BN on green color block. See above for two examples of relevant images in the dataset.
Dasing
ler
lar
beags
230
200
Figure 18: VGG-11-BN on an image with mostly grass texture.
21

---

Published as a conference paper at ICLR 2024
C.3 VGG-11-BL WITH SMALL LEARNING RATE TO APPROXIMATE GRADIENT FLOW
Here we see that oscillation is a valuable regularizer, preventing the network from continuously upweighting opposing signals. As described in the main body, stepping far in direction causes an imbalanced gradient between the two opposing signals. Since the group which now has larger loss is also the one which suffers from the use of the feature, the network is encouraged to downweight its influence. If we use a very small learning rate to approximate gradient flow, this regularization does not occur and the feature norms grow continuously. This leads over-reliance on these features, suggesting that failing to downweight opposing signals is likely cause of the poor generalization networks trained with gradient flow.
The following plots depict VGG- I-BN trained with learning rate .0005 to closely approximate gradient flow. We compare this to the feature norms of the same network trained with gradient descent with learning rate 0.1, which closely matches gradient flow until becomes unstable..
Class
Nonn
30-
15
0000 100
100
W www.mmo
R12
lepus Image (Crandent Fany)
Frg
Mou Ovei Duta
1000
ligu Iinge (lealent Dovutl)
Tunc
1ine
Figure 19: VGG-11-BN on a sky color block with learning rate 0.005 (approximating gradient flow) compared to 0.1.
Trpse
20-
17.
10-
Inqux Image (Grsheez Flwes
Moxz Oheel Data
(Grd
Time
Tixe
Figure 20: VGG-11-BN on a white color block with learning rate 0.005 (approximating gradient flow) compared to 0.1.
Inper Image
Class Logis
Feature Nomm
www w
10
legua lamag (Ceaden Fanny Repar Trage Doscen))
Figure 21: VGG- 11-NN on black color block with learning rate 0.005 (approximating gradient flow) compared 0.1.
22

---

Published as a conference paper at ICLR 2024
Clas Tagis
22-
ligus lasac Cleaalent Plans
Moan Overr This
Figure 22: VGG-I1-BN on a red color block with learning rate 0.005 (approximating gradient flow) compared 0.1.
bryst Eoaage
Class Logis
Feature Nomm
plane
15-
W
(Crndert Flres
lapus lmnge Desenti
Figure 23: VGG-1 11-N green color block with learning rate 0.0005 (approximating gradient flow) compared to 0.1.
Cagi
Reauee Nom
12-
Miau Ores hut Lpar baagu Cladies Dossett
Teao
Figure 24: VGG-11-BN on an image with mostly grass texture with learning rate 0.0005 (approxi- mating gradient flow) compared to 0.I.
23

---

Published as a conference paper at ICLR 2024
C.4 RESNET- 18 TRAINED WITH FULL-BATCH ADAM
Finally, we plot the same figures for a ResNet- 18 trained with full-batch Adam. We see that Adam consistently and quickly reduces the of these features, especially for more complex features such as texture, and that also quickly reaches a point where oscillation ends. Note when comparing plots above that the maximum iteration on the x-axis differs.
Iraage
Class Logas
Nonn
20
Iqpx lauge
Moon Don
1.40 1000
10
2001
3001
1000
400
610
1900
Denakies
Figure 25: ResNet- 18 on a sky color block trained with Adam.
Class Logis
Feature Nomm
60-
Ippe truge
Moie Over thr
1/ggs Volte
30-
1r2
613
-30
30-
1807
-70
200
400
500
200
1000
400
(00
1000
Figure ResNet-18 18 on red color block trained with Adam.
Ingee Imaeee
Class Logits
Festure Nomm
Over
20-
10-
Figure 27: R8 on green color block trained with Adam.
Legits
Image
200
010
80C
1000
2.0
800
1000
Figure 28: ResNet-18 on an image with mostly grass texture trained with Adam.
24

---

Published as a conference paper at ICLR 2024
D TRACKING THE AMOUNT OF CURVATURE IN EACH PARAMETER LAYER
Here we plot the "fraction of curvature" of different architectures at various training steps. Recall the fraction of curvature is defined with respect the top eigenvector of the loss Hessian. We partition this vector by network layer and evaluate each sub-vector's squared norm. This represents the overall curvature. To keep the plots readable, we omit layers whose
that layer's contribution fraction is never greater than 0.01 at any training step (including the intermediate ones not plotted), though we always include the last linear layer. The total number layers is 45, 38, 106, and 39 for the ResNet, VGG-11, ViT, and NanoGPT respectively.
ResNet-18
5og 312
Exg
Sux Sua Parnece Mook
loockileon ponkieennrrecaa Lociaa
Figure 29: Sum squared entries the top eigenvector of the loss Hessian which lie in each parameter layer of ResNet-18 throughout training.
VGG-11-BN
Sta Pazoee Blok
Sop
(euares Swiph
dositeel welge
Figure 30: Sum of squared entries of the top eigenvector of the loss Hessian which lie in each parameter layer of a VGG-I1-BN throughout training.
25

---

Published as a conference paper at ICLR 2024
Suee P Bood
VIT
Sey
Sop
prypach anhokt puics siice 20o
tans nod lagess ELO coUre Uhwngg
tnasierac Aepee and songee
tras LO cnd Doweul
tnsaerrac Spoor and Aocud
tanelianee Lovon Lun Aloa
and wenge
wough
and Locuge
Loughe
tean ferace beyd and loukk
mast maaadooae Sapess net Loogl
haad bong
Figure 31: Sum of squared entries of the top eigenvector of the loss Hessian which lie in each parameter layer of a VIT throughout training.
NanoGPT
Sin Pe Pe ww B3ock
2015
Wasdooettto intg de setgi
trantteree Act met gros sougr
tondeoeehh arra cat oniga
cnutemeea uar uang wegga
tranterenhh Sette are oolins
and weigh
conntemer brok cut ouuga
Deaiferee Lut curnd weigar
unsiemaca weipst
Tasdomarh matee to suupr
wowffineeee Sab cundd weggar
kandomeric de
Figure 32: NanoGPT (6 layers, 6 head per layer, embedding dimension 384) trained on the default
shakespeare character dataset in the NanoGPT repository. Due to difficulty calculating the true top eigenvector, we approximate it with the exponential moving average of the squared gradient.
26

---

Published as a conference paper at ICLR 2024
E COMPARING OUR VARIANT OF SGD TO ADAM
As described in the main text, we find that simply including dampening and taking a fixed step size on gradients above a certain threshold results in performance matching that of Adam for the experiments we tried. We found that setting this threshold equal to the q = .1 quantile of the very first gradient worked quite well this was about le-4 for the ResNet-1 10 and 1e-6 for GPT-2.
For the ResNet we also ablate the use of dampening: we find that masking appears to be much more important for early in training, while dampening helpful for maintaining performance later. It is not immediately what may be the cause of this, nor if it will necessarily transfer to attention models.
Simply have something to label it with, we name the method SplitSGD, because it performs SGD and SignSGD on different partitions of the data. We emphasize that we do not intend to introduce this as a new optimization algorithm or assert any kind of superior performance our goal only to demonstrate the insight gained from knowledge of opposing signals' influence NN optimization.
Momentum
SpliSGD Dampening
3.5
3.5
SGD ii - Iw0000
SpiidGD r--d. lndd
5010 A. k..0l
Spli5GD =-d. 2-0.01
5GD0 0
7.5
Adbaz, Imdle-
Spi,80 r-0,, h-d.1
200
2.0
1.1
0.3
82-
0.2
100
150
200
250
100
130
200
230
(a) Adam versus standard SGD with Momentum
(b) Adam vs. SplitSGD, but with no dampening.
SpiuSGD Dumpening
Fraction Paromorars Masked Per Stop
3.5
1.0
Spi 9, Inu)00)
3.0
Syl:30D 9, imtt0
1206
255
5pl
0S-
20
1.5
0.0
0.5
0.2
0.6
50
100
150
300
(b) The fraction of parameters for which fixed-size signed step taken for each gradient step.
(a) Adam vs. SplitSGD with dampening
E.1 SPLITSGD ON GPT-2
For the transformer, use the public nanoGPT repository which trains GPT-2 on the Open dataset. As a full training run would be too expensive, we compare only for the early stage of optimization. Not only do the two methods track each other closely, it appears that they experience exactly the same oscillations in their loss. Though we do not track the parameters themselves, we believe it would be meaningful investigate if these methods follow very similar optimization trajectories as well.
27

---

Published as a conference paper at ICLR 2024
1222-120--Aaa 5012-128-550
3002-124N-A00
10
20
Stpp
5xp
Figure 35:
Adam versus SplitSGD on the initial stage of training GPT-2 on the OpenWee'Pee dataset, and the fraction of parameters with fixed-size signed step. All hyperparameters are the de- faults from the na repository. Observe that not only is their performance similar, they appear have exactly the same loss oscillations.
28

---

Published as a conference paper at ICLR 2024
F ADDITIONAL FIGURES
Training Points
Training Loss
Group
Group
0.5
Odlers
1.5
Vale
1.0
1.00
-05
0.25
0.0
1.0
0.0
0.5
300
1000
1200
1400
990
1000
1013
1020 1050
Inpor Value
Irensnon
Reratio
Figure 36: Opposing signals when fitting a Chebyshev polynomial with a small MLP. Though the data lacks traditional "outliers", it is apparent that the network has some features which are most influential only on the more negative inputs (or whose effect is otherwise cancelled out by other features). Since the correct use of this feature opposite for these two groups, they provide opposing signals.
Fraction of Points That Increase in Loss Per Step
GD
SGD
0.50
Pratten
0.40
0.35
0.25
ResNer
VGG
2000
The fraction of overall training points which increase in loss any given step. For both SGD and GD, it hovers around 0.5 (VGG without batchnorm takes a long time to reach the edge of stability). Though the outliers have much higher amplitude in their loss change, many more images contain some small component of the features they exemplify (or are otherwise slightly affected by the weight oscillations), and so these points also oscillate in loss smaller scale.
Figure 37:
29

---

Published as a conference paper at ICLR 2024
12
14
16
18
-55
0.0
Change Loss
Figure 38: Distribution of changes in loss of individual training points at each iteration (starting iteration 200 a ResNet-18). The density is nice bell curve, with consistent and standard deviation in [0.3, 0.5], sometimes bit more or less. The fraction of samples at loss change greater magnitude than μ + is consistently very near [0.055, 0.015, 0.005] for E [2, 3, 4].
near
VGG-11 SGD Loss on Opposing Groups
Group
Group
Full Train Loss
Rundom Subeet
Loss
4
Loss
2
160
180
200 Iteration
220
240
160
180
200
220
240
Iteration
Figure 39: We reproduce Fig.
5(a) without batch normalization.
30

---

Published as a conference paper at ICLR 2024
Loss
Loop
200
cmup2
100
175
0.0
150
175
Sharpness
Sharpness
102
150
175
250
175
Weight irojection on Top Eigenvector for
Weight Projection on Top Eigencector
000
3.-
102
175
250
175
(a) A 3-layer ReLU MLP trained on 5k-subset (b) Our model: 2-layer linear network trained on of CIFAR-10.
mostly Gaussian data with opposing signals.
Figure 40: We compare small ReLU MLP on subset of CIFAR-10 to our simple model of linear regression with two-layer network.
31

---

Published as a conference paper at ICLR 2024
G PROOFS OF THEORETICAL RESULTS
Before we begin the analysis, we must identify the quantities of interest during gradient flow and the system of equations that determines how they evolve.
We start by writing out the loss:
20(0) Ei(e(b] x bbb σo) - (B x + d2 dz 1/21 ( [soo))
E[((cb - 3) x))] )+ + E[((ccb - - 1/2 (ign(Fo))) 1) xo0) )
(2)
= |cbb - -112 + p
a
(chs -1)) + (VV a
(cho + +1)
(3)
2
p
= |cbb - 311 + a @ b 11).
(4)
This provides the gradients
VoL c(cb B),
(5)
Vo, L bc,
(6)
VeL. = bt Jcb- - B) + a 111bo11e..
We will also make use of the Hessian identify its top eigenvalue; it is given by
e² ed,
0d,xda ac La
2cb
VOL(O)- = Od 2cb"
20abo
(8)
2cab
||b|1 + a 1
The maximum cigenvalue Amax at initialization is upper bounded by the maximum row sum of this matrix, and thus Amax S 3d +ada
di+d2 3a. Clearly, we also have Amax 2 a.
We observe that tracking the precise vectors b, by are not necessary to uncover the dynamics when optimizing this loss. First, let us write b == '1al + ōv, where is the direction of the rejection of from 3 (i.c., T v 0) and d is its norm. Then we have the gradients
3
V. (V.b) (VoL.
(9)
3
c
+ 88
- c3
(10)
||8||
||31|
= e²c E 3|
(11)
V&L (vbb) (VoL)
(12)
8
T 2
+ 80
c3
(13)
||8||
-60
(14)
B
(s (c(c
3
V.L.
+ bv
+ +vv -13 + allb..12c
(15)
||
||
=fc22 + 8² 1bol13)
|||
(16)
Finally, define the scalar quantity o ||b.l12 noting that VoL = 26 VE, L = 2ac2o Minimizing this loss via gradient flow is therefore characterized by the following ODE on four scalars:
de de + +c|18]1.
(17)
dt
dō 288
(18)
dt do 20020.
(19)
dt dc
c(e² + 82 + ao) =|]B]].
(20)
dt
(21)
32

---

Published as a conference paper at ICLR 2024
Furthermore, we have the boundary conditions
<(0)
V d1 + d2
(22)
di 1
8(0) V d1 + d2
(23)
d2
(0) =
(24)
dp + d2
<(0) 1.
(25)
Given these initializations and dynamics, we make a few observations: (i) all four scalars are initial- ized at value greater than and remain greater than 0 at all time steps; (ii) ô and o will decrease towards 0 monotonically, and will increase monotonically until cc ||š]|: (iii) e will be decreasing at initialization. Lastly, we define the quantity == (e(0)² + +oo)))
dutodz and k := di
d1
Before we can prove the main results, we present lemma which serves as a key tool for deriving continuously valid bounds on the scalars we analyze:
Lemma G.I. Consider vector valued ODE with scalar indices U1, 02,..., where each index is described over the time interval [mmin (max] by the continuous dynamics dv(()) aa(v=<()))-v<)) by (v with 0; S. O, by 2 0 for all i, (v-i denotes the vector v without index i). That is, each scalar's 's gradient an affine function of that scalar with a negative coefficient, Suppose we define continuous fanctions à,, by R + R such that Vi,t, à,(t) I a:(v=((t) and b,(t) S b,(u=((t). Let è be the vector described by these alternate dynamics, with the boundary condition (tmin)) val ve(Imin) and Ve ve(Imin) 2 0 for all (if a solution exists). Then for E [lming bmax] holds that
b(1) S v((),
(26)
elementwise. if by upper bound 0;, by, the inequality is reversed.
Proof. Define the vector w())
a(t) -tt). This vector has the dynamics
dw; = dt
dō;
dui
(27)
dt
dt
(1) 6,(() 8,(1) a((v_(1)) v.(1) (((v=(())
(28)
< all)) (1( (()) (l).)
(29)
The result will follow by showing that w(t) S for all E [Emin tmaxl (this clearly holds at bmin). Assume for the sake of contradiction there exists time t' E (tmin tmax] and index i such that (t') > (let be the first such index for which this occurs, breaking ties arbitrarily). By continuity, we can define to == max {t E [tminn t'] (t(() I 0}. By definition of to it holds that
wto) and Ve > wto + +) (t(t wy(to + E) >> O. and thus dwn;(to) definition of w we also have
0. But by the
de
(ttt)) vt(tt) + w;((t) = (tto),
(30)
(31)
and therefore
dwa((t) dt
< àz(to) 0, (x(tt) at(o)((to)) vi(to)
(32)
(ar(to) - az (v(v)((t))) v,(to)
(33)
00,
(34)
with the last inequality following because (t) < (t)v=,(t) and v;(() > 0 for all i, & E [fmin mmxx
Having proven both (to) and dw,(fo) < O. we conclude that no such can exist. The other
direction follows by analogous argument.
L
We make use of this lemma repeatedly and its application is clear so we invoke it without direct reference. We are now ready prove the main results:
33

---

Published as a conference paper at ICLR 2024
G.1 PROOF OF THEOREM 3.1
Proof. At initialization, we have ||||| 2 Vd +22
= 11311e(0) 2 ã.+d; - c(0)(e(0)² + 6(0)3).
de Therefore, we can remove these terms from dt at time = 0. noting simple that de dt 2 -xoc.
Further, long as is still decreasing (and therefore less than c(0) = 1),
d(l] c(E² + 8²))
d([]3]le E- - (e² + 8²))
(35)
dt
dt
= (11 2E) de
28 dô
(36)
dt
dt
(11 ( + [1 - 28(-28))
(37)
(el1 - 2(e² + +0) + +|]1111 26))
(38)
Z -c(e||81| -2(c² + 6²) =[|13|2-2
(39)
<(]|31] 2e <|| 2(e² + 6²))
(40)
(11311 - <2 + 11))
(41)
Since e > O at all times, this is non-negative so long as the term in parentheses is non-negative,
which holds long S
Further, since ec |||||| have
de² 2c de dt 22 - 2Ec|
(42)
dt
(43)
21131|2.
(44)
This implies c(t)? < c0)) + 3|1. Therefore, for S
In 11/1
2||0 we have c(t) S di+d2 +
||]|]] In 11001/2 <
|| +1))+ (this inequality holds for |||||| 2 2). This satisfies the desired upper bound.
Thus the term in Eq. 441 is non-negative for all &
have de -ac under the de de above conditions. Since the derivative of o is negative in c, lower bound on gives us upper di bound on de. which in turn maintains valid lower bound de This allows solve for just the ODE given by
In 1/11/12 and
||
da
de²
dt 2nceo
(45)
do 220o2.
(46)
dt
Define by
1
Recalling the initial values of e² o, The solution to this system is given
dd 1+k
c(t)²
m71
(47)
(1 (l-m) expl2amm m
o(t)
(48)
exp(2amt)
I-m
m
(49)
exp(2amt)) k-1) -1
Since these are bounds on the original problem, we have c(t)? 2 m and ott) shrinks exponentially
fast in t. In particular, note that under the stated condition va 2 miIn|/l7775 (recalling k d 1),
|8 Ink
Va
we have In
181/2 2 Therefore we can plug in this value for 6, implying o(t) S m
di
2/am
dz
mk /a at some time before
In 11. 1/2
21 pl|
Now solve for the time at which de > 0. Returning to Eq. (1).
instead suppose that || - e(2 || ) 2 7 for some 7 0. If this quantity was non-negative and
2
13 +2
34

---

Published as a conference paper at ICLR 2024
has had a derivative of at least y until time t
2/am then its value that time must be at least
Ia k
In
For de to be non-negative, need this to be greater than c(t)Pa(()) it suffices to have
2yam
ylnkk
co exp(2aml)(1
1)-1 7 lnnk 2
2a42n
7 2
m 20=
Observe
2/am
(1+4-1)-1
la
that the stated lower bound on a directly implies this inequality. Finally, note that |]6] = e² + 82, and therefore
dl| b||
2e de
20 d8 dt
(50)
dt
dt
2c 2 + 82) + 2ce 2ce ||3||-
(51)
Since c(0) = 1 and de < ||B1) , this means |]6] will also be decreasing at initialization. Thus we have shown that all relevant quantities will decrease towards 0 at initialization, but that by time lnk we will have at 20.
t 2vam
G.2 PROOF OF PROOF OF THEOREM
Proof. Recall from the previous section that we have shown that some time t S In lk
2/amm c(t)²
will be greater than and increasing, and o(t) will be upper bounded by mk-va Furthermore, (tt) I di+d2 + 2|]| To show that the sharpness reaches particular value, we must demonstrate that grows large enough before the point ce z ||S||| where this growth will rapidly slow. To do this, we study the relative growth of e vs. e.
Recall the derivatives of these two terms:
de
dt = - (22 + 52 ao²)ee+ ||B||c
(52)
de
= - + |1.31|c.
(53)
dt
Considering instead their squares,
d
de
2c dt
(54)
dt
2(e² + 52 + ao²) + 211Bllec,
(55)
de
2E de
(56)
dt
222 + 2|cBB
(57)
Since 8, o decrease monotonically, we have we can show that
de? di
2 -((2 +
d
d
+ am
+21 le. Thus
d1
d1 dr
||3|lcc 2 (62 + 2(
am
)
(58)
d + d2
as that is, that c(t)² grows at least half as fast e(t)². And since 8, o continue to decrease, this inequality will continue to hold thereafter.
we can conclude that dea 2 (c2 [|]]]cc) dt
1 de
2 de
Simplifying the above desired inequality, we get
C > 2 + 2m(1 + ak vi).
||
(59)
c
di Noting that 5 2 1 and m dstd; < 1. and recalling the upper bound e(t)², this reduces to proving
|||||| 2 di + d2
+ 20133111 + ak-va
(60)
35

---

Published as a conference paper at ICLR 2024
Since this occurs at some time 1 I
In and since m 2yam"
1 + k. we get
[1 +
Ink + + ak ak-
||
2
(61)
d1 + d2
Va
The assumed lower bound on va means the sum of the first three terms can be upper bounded by small 1 + o(1) term (say, 9/5) and recalling |||||
2 2/5 it suffices to prove
9 + ak- Va
||||||
(62)
5
= ak Va I 3.
(63)
Taking logs,
2 In va
Va In3,
(64)
Ink
which is clearly satisfied for va 2 1 + k Ink. As argued above, this implies de? dt 2 de?
2 dt by some
time l2 S In
2/am
Consider the time to which this first occurs, whereby c(t2)² growing by least one-half the rate of e(t2)². Here we note that we can derive an upper bound on and at this time using our lemma and the fact that
de s ||B||e,
(65)
de ||
(66)
The solution to this system implies
exp((18(1(2) - exp( exp(-113116)
c(l2) 5
+exp(1181(22) +1)
(67)
vdz + d2
1
exp(1181(t2)
1 +
vdz d2 +1)
(68)
<
exp
22/mm
1+
vd, + d2
+
(69)
((22) 5
exp(1181112)
+
Vdi d2) )n)
1
+
Vdh d2 -11
1
(70)
||
exp
2
1+
Vd, + d2
Vdl + d2 -1
(71)
|| In
2
the exponential term is upper bounded by / giving
Then for a >
[| miin
2]
VIBB
c(12) <
1+
+1
(72)
dd + d2
V |
(73)
2
||
(tz)
<
1+
+
Vdi dg -11
1
(74)
2
2
ddi
dg
VII |||8|
(75)
2
We know that optimization will continue until c²c2 || and also that de? 2 de? Since S c, this implies that 2² || before convergence. Suppose that starting from time 12, é² grows until
36

---

Published as a conference paper at ICLR 2024
time t' by additional amount s. Then have
= e(t')² E(t2)²
(76)
de(1)²
(77)
dt
1 do(t)²
(78)
1/ dt
2(c(t')2 c(t2)²²)
(79)
In other words, c² must have grown by at least half that amount. Since c(t2)² < |||||
4 and therefore
e(t')² < || 4 + s, even if c(t)² is the minimum possible value of & we must have at convergence
11812 This is a quadratic s and solving tells us that we must have s 2 ||lB1]- Therefore, c(t')² ||ll is guaranteed to occur. Noting our derivation of the loss Hessian, this implies the sharpness must reach at least [11 for each dimension of b
= - PI
37

---

Published as a conference paper at ICLR 2024
H ADDITIONAL SAMPLES UNDER VARIOUS ARCHITECTURES/SEEDS
To demonstrate the robustness of our finding we train a ResNet-18, VGG-11, and a Vision Trans- former for 1000 steps with full-batch GD, each with multiple random initializations. For each run, we identify the 24 training examples with the most positive and most negative change in loss from step i step i + 1, for {100. 250, 500, 750). We then display these images along with their label (above) and the network's 's predicted label before and after the gradient step (below). The change in the network's 's predicted labels display clear pattern, where certain training samples cause the network to associate an opposing signal with new class, which the network then overwhelmingly predicts whenever that feature is present.
Consistent with our other experiments, we find that early opposing signals tend to be "simpler", e.g. raw colors, whereas later signals are more nuanced, such as the presence of a particular texture. We also see that the Vision Transformer seems to learn complex features earlier, and that they are less obviously aligned with human perception- this is not surprising since they process inputs in fundamentally different manner than traditional ConvNets.
38

---

Published as a conference paper at ICLR 2024
leil
lng
adceadak
NeJ
Ergg
foo
Hill
deer
dice
NaJ
Is00
bock
hrl
4NY
Adcondak
Jeer
Dog
HJJ
1ug
Jer
Ing
Jky
Jky
Jky
ENy
Jay
ds
6ug
Us
0g
fng
0g
Uk
dg
Jkg
Jky
lel
Ing
3g
Hed
HJ
dk
Jkg
Jg
LK
Jez
Day
(a) Step 100 to 101
(b) Step 250 to 251
dlp
3000
Arc
home
Ang
CH4
1fng
Nrt
dip
C2T
bes
Juy
Jone
skpp
lfng
-
deg Jky
llog
Pask
dip
Cx
Nd
Cx
dip
Jrdaas
Jky
Juy
Aky
lng
feg Arphar
Ing
Ing
Liyha
day
kmn2
bnck
arieviil
bese
Psat
fng
ling
Inc
fng
Ing
ing
Ing
siphac
rick
pack
zszevhil
drg
frre
Ince
Inee
plac
(c) Step 500 to 501
(d) Step 750 to 751
Figure 41: (ResNet-18, seed 1) Images with the most positive (top rows) and most negative (bottom 3 rows) change to training loss after steps 100, 250, 500, and 750. Each image has the true label (above) and the predicted label before and after the gradient update (below).
39

---

Published as a conference paper at ICLR 2024
seplew
Akp
Jer
Akp
bal deg
Pst
zicevhil
pst
Hed
Dst
zeceviils
Hel
PAL
Aky
fng
dkp
Jky
CH
meee
C21
dec
Ing
frgg
Jky
Jaz
C21
ENr
C21
C21
In
dec
fng
dlp
4kp
Jan
frgg
CF
lnge
Aeg
buck
frgg
frog
Inck
(a) Step 100 to 101
(b) Step 250 to 251
SEcevlile
deg
dkp
frog
dk
sighe:
sszeviils
secevhile
arcevhik
4kp
dip
dap
dip
pst
dip
ds
Hel
ling
CH
Ndl
fire
frn
Hed
Jky
Jky
19
Jky
dk
mn
mrn
Cr
mire
Not
fny
FIEE
LK
dk
lrg
Jky
Jay
Iai-bel
deg
fine
Pal
Jrghas
C11
fng
dkg
deer
CF
frng
0i;
Nit
Alribas
Agg
dip-ttigg
(c) Step 500 to 501
(d) Step 750 to 751
Figure 42: (ResNet-18, seed 2) Images with the most positive (top rows) and most negative (bottom 3 rows) change to training loss after steps 100, 250, 500, and 750. Each image has the true label (above) and the predicted label before and after the gradient update (below).
40

---

Published as a conference paper at ICLR 2024
sipkee
Jyy
Est
day
fng
frgg
dk
dkp
Akp
TxT
atpke
191
fny
LWI
LK
#ky
4kp
Uky
aipke
Jky
frg
Jky
led
#k
dk
bx
Tal
back
ski
FAA
dk
frg
rhal
bak
boc
EAA
Hed
bog
Hed
led
J
bac
A0p
dk
dky
Hel
frg
Ing
dock
(a) Step 100 to 101
(b) Step 250 to 251
trck
aryhon
lrg
lerd
stp
bene
41z
dap
skop
sky
Mxp
Enrr
bona
dag
Net
der
10ck
freg
bock
Akp
Mkp
Jag
lrck
1el mne
Arpbee
Cor
der
Iee
Ligha
back
bock
Mkp
Arpkue
CH
Arpbee
513
cup
mg
Ine
The
fng
Bot-Joor
mne
mg
Arplee
Aiplina
Net
met
Cip
Faak
mme
xiceviiia
me
Ing
Thng
deg
dip
cip
(c) Step 500 to 501
(d) Step 750 to 751
Figure 43: (ResNet-18, seed 3) Images with the most positive (top rows) and most negative (bottom 3 rows) change to training loss after steps 100, 250, 500, and 750. Each image has the true label (above) and the predicted label before and after the gradient update (below).
41

---

Published as a conference paper at ICLR 2024
sip
aryle
skp
skp
aeplax
skp
ski
eiplamo
veck
Opplee
Eog 1ig Mpp
Ieg asplae
skgg skp
Frg
Mkp
sxpla
spplax
Yock
veck
meci
Appkac
arylec
Akp
sxpla
seplax
meck
Jkp
Inc
Fick
taak
Inc
Eng
Eag
Eag
Iog
tick
tr5ck
INCC
<kg
fag
Eag
Ing
Ing
mack
track
INCC
Eag
Eag
Bag
Eag
2kg
2ng
(a) Step 100 to 101
(b) Step 250 to 251
dkp
dip
Descnobie ansorebll seseroble
CHI
C21
C21
421
elp
Jrrbee
4x
N1
CT
db
T22
mct
4b
mct
ab
pst
sicoviil
db
4b
met
46
Jb
pstk
siceviile aeceobik
aecovill
mit
db
4b
m2t
db
Abp
db
(c) Step 500 to 501
(d) Step 750 to 751
Figure 44: (VGG-11, seed 1) Images with the most positive (top 3 rows) and most negative (bottom 3 rows) change to training loss after steps 100, 250, 500, and 750. Each image has the true label (above) and the predicted label before and after the gradient update (below).
42

---

Published as a conference paper at ICLR 2024
firg
fnig
frog
Frig
fres
Frig
duer
Jeer
fng
Jeer
deci
Jker
dec
Fuy
Fng
dko
free
lng
day
live
Fiig
fryg
Jeer
Fyy
buy
der
byy
byy
oke
de
dka
lng
frg
JNY
Jur
fny
Fiyg
Hed
bug
bay
Jeer
dec
deb
aipkee
aighte
siplex:
aeglemc
skplas:
sepfam
+
skp
skpp
skp
skpp
atpkee
arghte
stgliee
siples:
seglesc
siplas:
supfom
xkp
skp
skp
aipkee
sirgha
stglee
aggane
siples:
seglemc
siplax:
siglew
+l
App
skp
skp
(a) Step 100 to 101
(b) Step 250 to 251
dip
Nit
Nid
Nd
Net
Nil
tny
trig
frng
trng
freg
mgg
TRCCHENGE
Exg-etic frgg
Jqpuss
Net
hel
Fay
mmng
mg
Emrt
O1p
90
fng
frz
fne
fng
Inge
Ing
Ing
Juck
bs
bad
skp
trcck
Fack
Fack
Teak
Iog
bsck
back
back
brck
brck
trcc
plnc
Teak
arcevlil
zizeville
4x
hone
buc
back
bak
back
besc
Jone
trcc
(c) Step 500 to 501
(d) Step 750 to 751
Figure 45: (VGG-11, seed 2) Images with the most positive (top rows) and most negative (bottom 3 rows) change to training loss after steps 100, 250, 500, and 750. Each image has the true label (above) and the predicted label before and after the gradient update (below).
43

---

Published as a conference paper at ICLR 2024
fri
frig
Axce
deer
Jaer
Jecr
Jaer
Jeer
Akz
dkp
ski
skp
skpp
skpp
skyp
skyp
Lxv
dect
Jxy
frgg
Jaer
Jeer
dkp
Jkp
skp
Mkp
sky
Mxp
Lxy
Ing
Ixy
Jee
01c
dkp
dip
dkp
dkp
atykee
mitt
nxt
psck
back
tack
trck
atykee
digles
mxt
pst
tack
tiac
track
Prg-hog
atykee
sigles
sepleme
pst
tack
trcc
(a) Step 100 to 101
(b) Step 250 to 251
atribas
atrgiase
Zona
hona
1i---n Jirbee
Airibas
dip
hone
hww
hew
bona
hone
Bona
hore
hew
Jenu
hew
hona
hew
Bona
db
Jb
sebenchll
Abp
15xt
mic
di
sesenoble
bag
Abp
Eve
Abp
Max Maneela
mst
mct
Akp
(c) Step 500 to 501
(d) Step 750 to 751
Figure 46: (VGG-11, seed 3) Images with the most positive (top rows) and most negative (bottom 3 rows) change to training loss after steps 100, 250, 500, and 750. Each image has the true label (above) and the predicted label before and after the gradient update (below).
44

---

Published as a conference paper at ICLR 2024
bow
214E
dec
Ogg
dec
de
6:
Ril-iky
X---ily
arpbes
pst
pat
214
dkc
Dgg
dtc
dec
B ratk
X-i-ily 6g
Dat
dic
Drg
dec
dgg
hrw
hrw
6t
Ap-iby
pack
pat
mict
met
mitt
Japoas
airibas
met
met
mit
mst
mntt
(kp-o
pat
mitt
(a) Step 100 to 101
(b) Step 250 to 251
Jag
Jay
Jay
Jky
Jay
wobese uatarbbic catenokic unbmebii
Teak
dky
Jky
4g
Jky
sabasebiic
Jky
uoniic skacki
hadl
fyy
by
bug
byy
byy
byy
by
Fyy
bag
bug
byy
by
byy
by
lig
MACESASI
Adcendak:
by
bag
bn
bug
byy
by
byy
byg
11ng
(c) Step 500 to 501
(d) Step 750 to 751
Figure 47: (VIT, seed 1) Images with the most positive (top 3 rows) and most negative (bottom 3 rows) change to training loss after steps 100, 250, 500, and 750. Each image has the true label (above) and the predicted label before and after the gradient update (below).
45

---

Published as a conference paper at ICLR 2024
2002
hon
2one
Zone
hoee
3tg-ig kone
CR cpp
Ch-ica hor
bip--kg Fone
2X-10 dex
cxp-Aag Ioee
hone
lene
heee
Iene
IncL
Isne
Eev
arylaz
den
stplee
dem
der
Isoe
met
met
appkec
des
den
de.
net
SENTOONEE AENITOEE
applae
hone
siple
stylax
net
met
(a) Step 100 to 101
(b) Step 250 to 251
trak
track
track
day
Jkg
Aay
track
Jky
dkg
of
Aog
taac
deg
deg
dog
4eg
412
me
Nri
Nrt
Nrt
Net
Nrt
Nrt
diz
dip
dip
tack
dep
cxp
Nrt
Net
Net
Net
Net
Net
Ai:
Olp
dip
clp
cxp
Nri
Nrl
Net
Net
Net
Net
Nrt
Nrt
dip
dip
dip
dip
tack
dep
exp
(c) Step 500 to 501
(d) Step 750 to 751
Figure 48: (VIT, seed 2) Images with the most positive (top 3 rows) and most negative (bottom 3 rows) change to training loss after steps 100, 250, 500, and 750. Each image has the true label (above) and the predicted label before and after the gradient update (below).
46

---

Published as a conference paper at ICLR 2024
lirg
Ing
lng
Ing
Dag
fuy
VICE
VIIEE
7144
VTICE
VTACE
TL42
Ann
- Aay
Aow mik
FKCe-bel
Dng
Ing
A
Org
0g
mict
SECE
TECE
Mky
Akg
Ing
dog
drg
lng
0gg
05
214t
mict
TE4E
cip
ukp
1tp
Ap
Hpp
JBp
Akp
hore
hed
hore
boea
hwe
Iona
hee
dep-okk
cip
1tp
ap
12p
/bp
ab
bed
bona
hom
Iona
hew
dr
cap
ukpp
xxp
Jap
Akp
<1p
hore
hons
Ioeu
hww
Iona
deg
bed
tnck-pliuc
(a) Step 100 to 101
(b) Step 250 to 251
aroesill excevilii
sesenoble SENTraNE
WHITIN
zebenchiz
DENERONEE
me
hewa
fize
how
Eng
Dx1
Exg
Ext
Ine
Fine
hrw
hrt
Enge
hoee
hn
Ine
fine
fne
fire
fng
hrt
Eng
di
frg
Eng
Eag
bag
but
(c) Step 500 to 501
(d) Step 750 to 751
Figure 49: (ViT, seed 3) Images with the most positive (top 3 rows) and most negative (bottom 3 rows) change to training loss after steps 100, 250, 500, and 750. Each image has the true label (above) and the predicted label before and after the gradient update (below).
47


Reviews:
[{"review_id": "HCqzo6nvxq", "reviewer": "Reviewer_VsCH", "summary": "This paper studies the influence of samples w/ large, opposing features, and provides explanations for several prior observations.", "questions": "1. In EoS, vibrations are observed along the maximal eigen-direction. Could the authors explain how the presence of outliers impacts this phenomenon?\n\n2. In [1], the authors proposed the concept of multi-view features, which seems to be similar to the Outliers with Opposing Signals in this work. Could the author provide some comparisons?\n\n[1] Allen-Zhu and Li. Towards understanding ensemble, knowledge distillation and self-distillation in deep learning. (ICLR 2023)", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper explores the Outliers with Opposing Signals on the training process of neural networks. \nThe authors support their findings with extensive experiments and bolster their claims through the use of toy models. \nAs applications, the authors also provide new understanding of Progressive Sharpning and EoS.", "weaknesses": "Please see \"Questions\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6t08Wvkxkr", "reviewer": "Reviewer_5nbF", "summary": "The paper aims to understand how optimization is affected by a \"opposing signals\" --- subset of signals that can be associated with different classes depending on the rest of the signals ---  from small groups. For example, skies appear in backgrounds of planes most often, but they also appear in backgrounds of boats. The main claim of the paper is that such signals affect the progress of optimization by forcing the model to approach a basin in loss landscape that balances the opposing the forces. The paper shows experimenting track instability.", "questions": "See weaknesses.", "limitations": "", "rating": 3, "confidence": 2, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "The paper seems to identify an interesting phenomenon where the loss landscape that models end up in can be affected by opposing signals from small groups in data. There are some interesting plots in the paper that show case how the model depends on certain features (like the sky color).", "weaknesses": "There is too much informal language and the paper does not consider other confounding factors in leading to the instability. Specifically, the paper uses \"per-step largest loss change\" to define groups. Of course these groups will have large gradient norm because they have the largest errors in them. \n\n1. There are words like like genuine, correct, and random noise used but without formal definition. The authors acknowledge this but without clarity on what these are, the proposed phenomenon is too vague. You can define features are the groups of data that the feature helps correctly classify,  which becomes a formal definition.\n\n2. The synthetic experiment has a non-linear ground truth (absolute value of x_0), is that what contributes to the reduction in sharpness? \n\n3. What about the roles of batchsize, learning rate, lr schedulers, other kinds of regularization?\n\n\nOne of my issues with the writing is loose language. For example, why call $x_0$ unimportant? Clearly it helps predict the output, just on a subset of data. The idea of opposing signals seems very similar to have a \"shortcut\". The text in section 3 seems tedious because the authors use language the seems to be intentionally removed from terminology in shortcut learning  or distribution shift, but they use an example that is commonly called shortcut learning (model rely on skies to classify planes because many planes have sky in the background).\n\n\nFinally, the main claim is \"opposing signals affect shaprness like so ... \". Then, there should be a single plot that shows this exactly. I could find one such experiment. Why not directly measure sharpness, at least for the first few layers? It seems like this was done in \"Additional findings\", but there is not plot and supporting text. It would be great if there was a plot that showed, training instability occur exactly when sharpness rises up, and then controlling opposing signals via some robust loss or remove some samples both removes the sharpness and training instability. \n\n\nOverall, the idea is interesting but the experiments need to be more than a collection of simple post-hoc checks to show that the cause of optimization is certain groups. One direction, maybe is to randomly change the backgrounds of the supposed outlier groups (upto rho fraction) and show the sharpness behavior is avoided.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YR6LByQSyo", "reviewer": "Reviewer_u2Ni", "summary": "This work discovers a phenomenon throughout the gradient descent training dynamics of deep neural networks: several paired groups of outlier samples emerge in the training set, where the samples of each group in a pair have a strong opposing signal to the other pair, i.e. their corresponding gradients are large and point at opposite directions. The authors hypothesize that this observation could be the reason behind the \"edge of stability\" and some other phenomena in deep learning optimization.", "questions": "* As mentioned above, it might help the readers have a better understanding if there is a visualization of the distribution of the change in loss for training samples, highlighting the frequency of the outliers in comparison with the entire training set.\n\n* The statement of Theorem 3.2 mentions that \"sharpness will increase linearly in $\\Vert \\beta \\Vert$ until some time $t_2$\". Can an exact formula (or at least more explicit upper/lower bounds) be given for sharpness as a function of time? I believe this should be possible due to having explicit formulae for the weights during training for the linear model. Additionally, is it possible to highlight the role of the problem parameters including $k, \\alpha$, and $\\Vert \\beta \\Vert$, and their relationship to the observations for deep networks?\n\n* The example of opposing signals in text does not show the training error of the opposing outlier groups throughout training. Is it possible to recreate Figure 1 for text data?\n\n* As a minor comment, the sharpness is typically referred to as the \"largest eigenvalue of the loss\", while it could be more accurately described as the \"largest eigenvalue of the Hessian of the loss\".", "limitations": "", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* To my knowledge, the observation is novel and can have significant impacts on our understanding of the optimization dynamics of deep neural networks.\n\n* Many connections are established with other observations in the recent literature.\n\n* There are experiments on a variety of setups to ensure the observations are consistent across different settings.", "weaknesses": "* My main concern with the current state of the submission is that the notion of outlier training samples does not seem to be precise enough and might require further clarification. For example, the authors show that the loss of almost half of the training samples may go up during a single iteration, but this change is more significant for a smaller number of outlier samples.  It is not clear what fraction of the training samples are outliers and more generally what the distribution of the loss change over all samples is.\n\n* The relationship between the theoretical statements and the empirical observations of the paper is not entirely clear to me. The theory seems concerned with analyzing the sharpness throughout training with *gradient flow* which monotonically decreases the training loss, while a core part of the observation is to justify the spikes in the training loss that occur due to *large step size gradient descent*.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WcQoRUnDoe", "reviewer": "Reviewer_scXv", "summary": "This paper identifies a new phenomenon in neural network training, where some training examples exhibit opposite effects in training dynamics. Such opposite effects are signals that guide the network to learn specific features. These opposite signals influence the training dynamics significantly, sometimes producing loss spikes. Such opposite signals lead to sharpening, which happens in early layers.\n\nThere are numerous consequences of the opposite effects: (1) it allows neural networks to pick up finer features progressively (1) Batch normalization can stabilize the training dynamics (2) Adam and gradient clipping helps to mitigate the negative effects of training", "questions": "1. Figure 1 suggests that at different phases of training, neural nets pick up very different features: initially features that rely on global information, and later features that rely on localized or detailed information. What is the reason for this? Do we believe that \"global information\" has stronger signal strength so it is learned first?\n2. With opposing signals, do neural networks produce differentiated neurons, each learning a different signal? Perhaps some examination of learned neurons/representations will tell.\n3. The authors mentioned 'In fact, in many cases, these features perfectly encapsulate the classic statistical conundrum of “correlation vs. causation”'. If a feature is irrelevant to a target task, why \"do meaningfully correlate with it\"? Why does a neural net learn features that are irrelevant to a classification task? \n4. The authors mentioned that sharpness is found in the early MLP layers and the embedding layer. What about the self-attention components (considering that they are the most important part of transformers)?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I think this paper presents an interesting (though potentially controversial) method of studying neural networks. Overall it contains a consistent and plausible story of what optimizing neural networks is doing internally, thus providing a mechanistic explanation to many empirical phenomena. While I find that many notions and claims are vague, I tend to believe that there are many valuable insights based on the extensive experiments and examinations of the literature.\n\nHere are some key strengths:\n1. The phenomenon of opposing signals is plausible and seems to be a fundamental cause of many existing techniques and artifacts about neural nets.\n2. The paper presents some heuristics that connect optimization with generalization, which has the potential to inspire new research\n3. There are various types of experiments, ranging from simple MLPs to ResNets and Transformers.", "weaknesses": "An obvious weakness is the vagueness that persists in the entire paper, but this seems to be a deliberate choice by the authors. As a consequence, I find many claims confusing to understand precisely, but overall this paper gives me a better understanding of training dynamics. So I would not object to this style as much as some readers do using a scientifically more rigorous criterion. Despite the weaknesses, I am inclined to give a favorable score.\n\nSome other weaknesses:\n1.  The writing can be organized in a better way. There are many claims that are buried in long paragraphs. I would suggest that the authors number and list their claims/hypothesis/conjectures clearly, and then provide explanations for each of the claims. Even if they don't have conclusive evidence, a clear statement should be presented clearly so that future work can cite the statement unambiguously.\n2. Currently there are no clear quantitative measurements. I do suggest that the authors provide one or two measurements to quantify \"opposing signals\" so that later work can examine their claims more precisely.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)