The Directionality of Optimization Trajectories in Neural Networks
Authors: Sidak Pal Singh
Bobby He
Thomas Hofmann
Bernhard Schölkopf
Venue: ICLR 2025 Poster (2025)
Keywords: optimization
trajectory
redundancy
LLMs
neural networks
Primary area: optimization

Abstract: The regularity or implicit bias in neural network optimization has been typically studied via the parameter norms or the landscape curvature, often overlooking the trajectory leading to these parameters. However, properties of the trajectory --- particularly its directionality --- capture critical aspects of how gradient descent navigates the landscape to converge to a solution. In this work, we introduce the notion of a Trajectory Map and derive natural complexity measures that highlight the directional characteristics of optimization trajectories. Our comprehensive analysis across vision and language modeling tasks reveals that (a) the trajectory's directionality at the macro-level saturates by the initial phase of training, wherein weight decay and momentum play a crucial but understated role; and (b) in subsequent training, trajectory directionality manifests in micro-level behaviors, such as oscillations, for which we also provide a theoretical analysis. This implies that neural optimization trajectories have, overall, a more linear form than zig-zaggy, as evident by high directional similarity, especially towards the end. To further hone this point, we show that when the trajectory direction gathers such an inertia, optimization proceeds largely unaltered even if the network is severely decapacitated (by freezing >99% of the parameters), --- thereby demonstrating the potential for significant computational and resource savings without compromising performance.

Published as a conference paper at ICLR 2025
THE DIRECTIONALITY OF OPTIMIZATION TRAJECTO- RIES IN NEURAL NETWORKS
Sidak Pal Singh* * ETH Zürich & MPI-IS Tübingen
Bobby He ETH Zürich
Thomas Hofmann ETH Zürich
Bernhard Schölkopf
MPI-IS Tübingen & ETH Zürich
ABSTRACT
The regularity implicit bias in neural network optimization has been typically studied via the parameter norms or the landscape curvature, often overlooking the trajectory leading these parameters. However, properties of the trajectory particularly its directionality - capture critical aspects of how gradient descent navigates the landscape to converge to solution. In this work, we introduce the notion of a Trajectory Map and derive natural complexity measures that highlight the directional characteristies of optimization trajectories. Our comprehensive analysis across vision and language modeling tasks reveals that (a) the trajectory's s directionality at the macro-level saturates by the initial phase of training, wherein weight decay and momentum play a crucial but understated role; and (b) in sub- sequent training, trajectory directionality manifests in micro-level behaviors, such oscillations, for which we also provide a theoretical analysis. This implies that neural optimization trajectories have, overall, a more linear form than zig-zaggy. as evident by high directional similarity, especially towards the end. To further hone this point, we show that when the trajectory direction gathers such an inertia, op- timization proceeds largely unaltered even if the network is severely decapacitated (by freezing > 99% of the parameters), - thereby demonstrating the potential for significant computational and resource savings without compromising performance.
INTRODUCTION
Loss landscapes imply optimization trajectories, but also vice versa. Given a network architecture and the training task, the loss landscape is the high-dimensional surface whose each point characterizes the fit of the parameters to the task objective. A priori, the loss landscape entails the possible trajectories (or paths) that might be followed by an optimization algorithm, such as stochastic gradient descent (SGD). However, the particular sets of optimization trajectories that are realized depend on the particular optimization choices and hyperparameters, such as the learning rate, momentum, batch size, weight decay, and more. Such is the importance of trajectory, that it could be argued the regions of the landscape that are never encountered or realized in typical optimization trajectories, might as well not be in the landscape all. Essentially, the possible optimization trajectories that are realized by an optimization method determine its effective loss landscape.
Implicit bias of optimization and what it means for trajectories. Consequently, a significant body of literature builds around the principle of implicit bias facilitated by standard optimization algorithms (Gunasekar al.. 2018; Li et al., 2019; 2020; Moroshko et al., 2020; Barrett and Dherin, 2020), which loosely speaking refers to an undesigned and oftentimes beneficial simplicity that arises in optimized trajectories. This preferential access of realized landscapes should be contrasted with the established non-convexity of global neural network (NN) landscapes (Dauphin et al., 2014; Choromanska et al., 2015; Kawaguchi, 2016; Li et al., 2018), and suggests that implicit bias may lend formal and reasonable support to the empirical success of massively over wer-parameterized NNs. As a result, one might expect to see signs and hallmarks of regularity in the sequence of steps that comprise realized NN loss landscape trajectories. In other words, the following questions, which form the basis of this work, naturally arise:
Correspondence contact sidappal.comm

---

Published as a conference paper at ICLR 2025
What, if any, directional structure exists in neural trajectories? Do these paths have a lot of zigzags and bends, reaching the solution winding and coiling. or are they straight and direct? How do these properties evolve as optimization progresses?
Mapping the Trajectory Properties. More precisely, we explore and develop key qualitative as well as quantitative indicators (hallmarks) about the directional complexityrregulaiiy of the optimization trajectory. Towards this end, we analyze and compare multiple intermediate checkpoints amongst themselves, across different scenarios and large-scale case studies. qualitative hallmark, which we call the "Trajectory Map', conveys the directional (diiariyyy of the parameters and visually depicts the nature of optimization within and across various stages of training, i.c. at a pan-trajectory level. Our quantitative hallmarks are functions of these trajectory maps, measuring various notions of angles, over and above the sequence of steps in the trajectory.
Benefits of the Trajectory View. We believe the trajectory view has several advantages over other measures to study NN training landscapes, such as loss- loss-based linear interpolation (Goodfellow et al., 2014; Frankle et al., 2020) hidden representation based metrics (Kornblith et al., 2019; Nguyen et al., 2022; Lange et al., 2023). For example, it: (a) brings in a level of architecture agnosticity and helps unlock shared insights onto features of optimization, (b) contains an intrinsic independence to loss and data that necessitates no explicit inference over additional data samples, (c) allows analyzing and prognosing the developing solution strategy on-the-fly -fly instead of waiting until convergence, and (d) provides potential hints at the bottlenecks and redundancies in the optimization procedure.
Our contributions are:
(1) We propose the novel perspective of trajectory maps, and showcase how it uncovers new insights into the nature of directional evolution in parameter space during neural network training.
(2) We use this to uncover that optimization trajectories possess, in general, highly directional similarity, but simultaneously there reside oscillations at micro-levels - which we study theoretically.
(3) We show how model scale in LLMs has regularizing effect on the directional complexity of trajectories, and that directional measures have trend which is more predictive of performance than traditional norm-based complexity measures.
(4) We provide evidence that after the trajectory direction saturates, a significant amount of the network capacity can be frozen without compromising performance while making training efficient.
2
METHODOLOGY
Matrix representation of Trajectory. Let assume the optimization trajectory consists of set T of points (0.),-o- each denoting the (flattened) parameters of the network encountered at some step and which live in the parameter space 0 IR", i.e., Tc S 0. This set of points need not contain the entire set of points visited in the course of optimization but instead can represent a subset of points, possibly sampled at an interval of & points. will be convenient organize this set of points, which define the trajectory, in the form of matrix, O [R(++1)xp, whose first dimension T + makes explicit the inclusion of the initialization 0y.
Trajectory Map Analyzing the matrix O on its own, might get cumbersome as the size of modern networks ranges in millions and billions of parameters. Hence, we will resort to looking at functions
of the kernel matrix K = 00* T which would be a square matrix of shape = T + 1. Further, it will also be helpful to isolate and analyze the directional aspect of the trajectory, for which we will normalize the set of points by their norm, and in effect, consider the set Í = (0./1(0
with the respective matrix é. As a result, the ensuing kernel matrix θθ" which we will refer to as C will contain the cosine similarities between every pair of points in the trajectory, i.e.
(C)s cos-sim(0,, = (0., / 0, 2
Hereafter, we will refer to C as the Trajectory Map (TM). We remark that, although not necessary, here we are essentially considering linear kernels, which we will see provide a variety of insights by themselves. The TM will be our qualitative hallmark of choice for analyzing optimization trajectories.
Quantitative Hallmarks. Besides visualizing the TM as the qualitative hallmark, we will consider the following set of indicators for quantitatively hallmarking the optimization trajectories.
2

---

Published as a conference paper at ICLR 2025
(i) Mean Directional Similarity (MDS): We take the cosine similarity averaged over the entire trajectory map, i.e., over every pair of points in the trajectory. This can be written as, w == 72 1 C n,, where 1 = (1... 1)T R denotes the vector of all ones and n = [T] is the cardinality of the trajectory. By using the form of the matrix C discussed before, we can further
θ n. Now, it becomes apparent that MDS essentially projects all the trajectory points onto the unit sphere, computes their average and finally takes the squared norm.
rewrite MDS as,
To get better sense of MDS, we can consider its two possible extremes: (a) all the parameter unit-vectors cancel out, yielding a value of O. For instance, this would happen in the scenario when the points in the trajectory are exactly following a circular orbit around the origin; or, (b) when each the parameters point in the same direction, implying that the trajectory is simply a linear path, with w' = 1. Knowing the nature these two extremes, desirable in an ideal trajectory which leads up to a generalizing solution. Thus, hitling its sweef spot would be the target, or where that is unknown, at least avoiding these extremes.
-0,0)
20,
40,,
expect neither to be
Figure 1: Illustration of two trajectories and angular measures.
(ii) Angular Measures: One such key measure would be the angle between consecutive (net) updates, i.e., Z(0 - or. 0, - 0r-1), which will measure the extent of directional movement along the trajectory. Next, we can consider cone with the vertex
origin and compute the apex angle there, (0). This measure will give an idea of the amount of directional movement at global scale.
(iii) Norm-based Measures: Besides, for baselines, we also include measures such as: parameter norms |0.112, distance from initialization 00, - 00l12, norm between consecutive points |0.+1 0.112-
Remark 1. For extremely large neural networks, building the underlying kernel matrices can start In principle, there is rich body of work in kernel methods that has focused on developing efficient approximations (Davis al., 2014; Chen al., 2021). We do not resort to such approximations for the sake of accuracy.
becoming eesource-eppensee
Remark While beyond the scope of the main text, the appendix also catalogs results for other vari- ants of angular/norm-based measures, well as a relativized version of trajectory map (Section D).
3 GAINING INSIGHTS VIA TRAJECTORY EXEMPLARS
Setup for obtaining the trajectory exemplars. To begin with, in order to obtain a better grasp of the directionality of trajectories, let contrast the trajectory map obtained with optimal choice of hyperparameters against those with sub-optimal ones. As an exemplar of a desirable trajectory map, we employ the standard SGD-based training recipe' for ResNet50 on ImageNet that achieves top-1 accuracy of ~ 76%. To compare against sub-optimal trajectory maps, we analyze the effects of disabling* momentum and weight decay one by one, since empirically they lead significant deterioration in performance. The resulting trajectory maps, built from checkpoints taken at each epoch, are shown in Figure 2.
Observations. We can make out that the trajectory maps for the sub-optimal hyperparameter choices develop a darker hue much more quickly into training, indicating a significantly higher directional similarity. To the extent that, when put on the same scale as in Figure 2, the rightmost (worst hyperparameter choice) trajectory map shows almost no directional variation as compared to the leftmost (optimal hyperparameter choice) map. Further, for the leftmost trajectory map, the
Namely, this consists of training for 90 epochs with a learning rate n 0.1 (decayed multiplicatively by a factor of 0.1 at epochs 30 and 60), momentum μ = 0.9, batch size 256, and weight decay 0.0001. II the Appendix F, we also present the directional effects of hyper-parameters such learning rate and batch size, well recent regularizers like Sharpness-Aware Minimization (Foret al., 2021).
3

---

Published as a conference paper at ICLR 2025
Optimal Hyperparameter Choice
Sub-Optimal Hyperparameter Choices
2-81.4-1.00
2-2A--0:00
p=02.1=0
Figure 2: Trajectory Maps of ResNet50 trained ImageNet with optimal and sub-optimal (in the order of disabling momentum μ = 0, weight decay 0, or both) hyperparameter choices. The MDS values are, 0.764, 0.901, 0.931, 0.979 respectively. A darker hue represents higher cosine similarity.
directional movement shows a phase- trend?, with its last phase 60 90 epochs having a similar behaviour the entire rightmost plot in the figure above. We emphasize that the onset of this dark hue doesn't It mean the network has converged, as evident from Figure 3(b) which shows the trajectory length covered over time. Rather, only the directional movement reaches a saturation by the end of the initial phase (which is 30 epochs) shown in Figure 3(a).
Qualitatively different solutions. More than just the deterioration of the performance (~ 8%), we conclude that this toggling of the hyperparameters leads to solutions that qualitatively different terms of the directionality of their trajectories. In particular, the presence of weight decay or momentum encourages more directional exploration, while in their absence, the trajectory latches to solution which is nearby in a directional sense.
Thopetory Legght
(a) L(E,, 0p)
6 02-1112
Figure 3: Evolution of the angle traced at origin by the trajectory and its path length over training across different momentum and weight decay settings. The left figure suggests that around epoch 30, marked by the gray dotted line, the directional movement saturates, while based on the right figure, we see that the trajectory still goes further. Set-up: ResNet50 on ImageNet.
Directional Redundancy, Another visible aspect from the trajectory maps is that, in general, neural optimization trajectories seem to possess high mean directionality score (MDS), with 0.764 being the least amongst these exemplars. To contextualize this better, note that the parameters live in 25.6 million dimensional space. This strongly suggests that network optimization trajectories do not merely comprise random points" in high-dimensions whose cosine similarity might go to zero, but rather the sequence of points encountered in trajectory must be highly structured to yield such directionally redundant trajectories. In the Appendix F, we also present a wider set of results, namely, with adaptive optimizers like AdamW as well as more dataasts/architecctres such as Vision Tranformers on ImageNet and VGG16 on CIFAR10, and for different amounts of label noise all of which support this finding of directional redundancy.
A word of caution. However, the above experiment also suggests that extreme values of MDS (> 0.9) should be avoided This can also be understood from the fact that trajectory maps here also comprise
'Wiile phases can be spotted distinetly. we look closely, there seems to another phase transition neighbouring the initialization and the subsequent couple epochs, giving rise to a thin horizontal and vertical sliver of relatively lighter huc. Empirically, we find in this transient phase, the distance to the solution rapidly decreases. In Appendix C, we analyze the relative trajectory map and MDS for random walk/Brownian motion, and comparison we
(expectedly) the trajectory maps of neural networks are more directionally redundant. Besidess, certain amount of directional exploration is crucial evident from our analysis of the grokking (Power al., 2022) phenomenon discussed in Appendix I.
4

---

Published as a conference paper at ICLR 2025
the random initialization, and hence trajectory with the maximum possible MDS value of 1 would have as its end point a scaled version of the random initialization.
Section 3 Key Takeaways:
Optimization trajectories have, overall, a high directional similarity (MDS > 0.75 across cases),
Disabling momentum and weight decay reduces the directional movement in the trajectories.
Directional movement saturates by the early phase, despite the trajectory going longer than that.
UNDERSTANDING DIRECTIONAL MOVEMENT
In this section, our aim is investigate the nature of directional movement in further depth, and in particular, focus on the below two questions:
Question How exactly does the directional nature of trajectories manifest in the late phase of Training, when we know from Figure 3(a) that the overall directionality does not change significantly?
tempting first guess that perhaps the trajectories are just linear in this phase. But, if that were the case, we would be just using line-search (Zhang and Hager, 2004; Vaswani et al., 2019) to train neural networks and skip this otherwise slow second phase of training.
Initial Thoughts.
Question 2: How do weight decay and momentum promote directional movement and lower MDS?
Initial Thoughts. The current results might even seem to run against the intuitive pictures of momentum and weight decay. For instance, intuitively, the use of momentum should strengthen the previous gradient directions and lead to increased directional similarity, but we find the opposite. Similarly, in the absence of weight decay, the network is not constrained to remain in a ball around the origin and, in principle, there should be more license to explore in the landscape.
What are these intuitive impressions not taking into account?
4.1 DIRECTIONAL MOVEMENT DIFFERENT LEVELS AND OSCILLATIONS
Augl: a4 Sane
Updatte nocTs
Thaining Epochs (5)
Troining Epochs (4)
(a) L(0 o., O2
12-1)
(b) 10
0.112
Figure 4: (a): Oscillatory nature of updates. (b): Beyond the first phase (after 30 epochs). the updates reside at micro-scale with smaller update norms. The different curves show the effect of momentum and weight decay hyperparameters. The setup same as before: ResNet50 on ImageNet.
We find that the late phase of training is not so straightforward. Far from co-linear updates, it marked by significant noise and oscillation in the (stochastic) gradient descent updates. The answer to this conundram then lies in the fact that such noisy directional movement resides at "micro" -lvve but does not surface up to the 'macro' -leve..
Answer 1.
To see this, let us first take a general overview of the Figure 4, without looking at each of the individual curves. We see Figure 4(a) that for a significant part of the training. the angle between successive update directions? is obtuse (gray horizontal line marks 90%). Apart from short intervals
We use this macro-mict micro distinction in terms of the angle traced at origin by the trajectory, see Figure 3(a). "These updates should be thought of as "net" or 'aggregate' updates, since they are at an per-epoch granularity for tractability reasons, and not every single step taken by the optimizer. Empirically the current granularity is still rich enough in that the presented trends persist even if go 2x to 5x coarser.
5

---

Published as a conference paper at ICLR 2025
Figure 5: (Left):: Directional movement due to oscillations results in obtuse angles; (Right): Added directional movement due to weight decay. Here. the downward vector represents the pull towards the origin (O) due to weight decay, while the rightward vector the force due to the loss.
2a 0,40
4ha R,8 Beal
near learning rate decay, this obtuse angle holds almost throughout the late phase. However, the extent which this angle is obtuse decreases over the phases and further, Figure 4(b) reveals that beyond the early phase (marked by the vertical gray line), these oscillatory updates have much smaller norm. Consequently, the late-phase oscillations reside micro-scale and therefore leave the macro-level directionality unaffected.
Answer 2. Interestingly, when look closely at Figure 4(a), we find that the angle between updates is more obtuse when momentum is turned on versus when off (compare the solid lines versus dashed-dotted lines). Also, rather visibly, the angle tends be larger when weight decay enabled additionally (compare the solid dark blue and turquoise lines), , suggesting that weight decay and momentum are closely intertwined. Crucially, this observation points to how there is more direction movement when these hyperparameters are enabled and thereby decreasing MDS.
Furthermore, weight decay by itself has added directional effects, which we illustrate through a simple physics-based intuition in the Figure 5, right. In particular, we can think of the loss gradient pulling the network parameters rightwards, while the force exerted by weight decay tries to pull the network downwards. The joint presence of these forces lends more directionality to the trajectory, and is accentuated with increasing weight decay (downward force). See Appendices E.2 and E.4 for more.
4.2 THEORETICAL MODELLING OF THE UNDERLYING MECHANISM
To further understand the mechanism underlying the oscillatory nature and the intertwined role of momentum and weight decay, we turn to the simplest and oft-employed model of a quadratic problem.
Lemma 1. Given quadratic problem with l regularization of strength a > O, namely,
mingerd 10 MM _M = with M E Raxd symmetric with eigenvalues À S 2... > >a the angle between successive steps A = 0: 0 A++1 θc+1 O,, when using gradient descent with one-step momentum (μ 0) and learning rates nhe M1+1, can be upper and lower bounded as follows:
(D, A.r) & men+r (1 me(u + a + Aa)) 11.
( A4+1) R moM+1(1 ne(u+ + a + A1)) + a)
The proof, in Appendix A, inherently considers the solution at 0 0, but if that is not the case, we can substitute it in the objective and our derived bounds would scale in the squared distance to the solution, i.e. | 0" 1 Besides, in the above proof, consider one-step momentum, which inherently means resetting the momentum after every 2 steps. This is done for convenience, as our main purpose is to anyways gain insights into the phenomenon and not provide its ultimate proof.
Turning to the bounds themselves, notice that if the learning rate T) 3 1/(A1 + f a), the lower bound will turn negative and will be multiplied by factor of (>, + a)) On the other hand, although the first term of the upper bound might still be positive, importantly, it is scaled by factor of (Aa a)² a² for matrices M which are close degenerate (Aa 0).
Low-rank Hessian and Edge of Stability (EoS). In NNs, the Hessian of the loss with respect to the parameters will play the role of the matrix M, since we can assume a second-order Taylor series will hold across the two steps. But it is also known through prior empirical work that the Hessian is significantly degenerate (Sagun et al., 2017), which has also been proven rigorously for deep linear fully- cully-connected and convolutional networks (Singh et al., 2021). Furthermore, this requirement on the learning rate 7)t is actually looser than the adaptivity of the largest eigenvalue of the Hessian to the learning rate À ~ 2/11, as shown in the recent work on Edge of Stability (Cohen et al., 2022).
*The smaller update norms and the micro-level directionality in the late phase do not simply arise due to lower learning rates, but also closely related to the loss of plasticity (Lyle al., 2023; Dohare al., 2024).

---

Published as a conference paper at ICLR 2025
Explaining the Obtuse angles. Owing these facts, will have that (M(M) 0, and which further implies that the upper bound on the inner-praed aener-product between the updates will be approximately zero (as typically the regularization strength and the lower-bound will be large in absolute value but negative. Therefore, this explains how the angles between consecutive epochs be obtuse. More broadly, the obtuse angle indeed implies that there are oscillations, especially along the direction of the largest Hessian eigenvector. Further, from Lemma we see that the magnitude the inner-product of the updates scales proportion Mt-1- Hence, way to dampen the oscillations is to decrease the learning rate, and as can be seen in Figure 4(a), the learning rate decay epochs 30 and 60 followed right after with the angles turning from obtuse to acute. Lastly, here in the constraint on the learning rate (the additive terms and μ),, we can also see momentum and weight decay go hand-in hand-in-hande ia-hand, each accentuating the effect of the other.
also small, e.g., 0.0001 in the ResNet setting),
Section Key Takeaways:
Training close EoS causes obtuse angles between updates, leading to directional exploration.
Weight decay and momentum closely intertwined, and further enhance the oscillations. The stae- oscillations/bouncing reside at micro-scale, as evident from smaller update norms (Figure 4(b)) and angles between consecutive epochs (Figure 4(a)).
5 LLMS AND DIRECTIONAL HALLMARKS OF TRAJECTORIES
Having better understood the nature of directional movement, we would like to know how general- izable are findings. So, given the increasing relevance of Large Language Models (LLMs), in this section, we study whether language modelling tasks result in a similar trajectory structure as exhibited in vision tasks. Besides, in connection LLMs, scaling comes up as natural question. Hence, would like to investigate how scale affects the directionality of trajectories: does it provide more direc- tions for learning and increase the complexity of trajectories or instead it has regularizing effect?
100m
1.4b
12b
Chockpaint index
Figure 6: Trajectory Maps of Pythia GPT-NeoX mod- orders of
22K a8и S4K.
88
066
els across scales trained on Pile. The corresponding MDS w 0.678, 0.759, 0.815.
1088
0.4
1224
100
1226
142
Checkpoint index
Thanks to Pythia's (Biderman et al., 2023) publicly released model checkpoints over training, for GPT-NeoX (Black et al., 2022) models ranging in sizes from Million (M) to 12 Billion (B) we can provide answers to the above questions. For tractability. we select every fourth available checkpoint and consider models of sizes: 14M, 70M, 160M, 410M, 1.4B, 2.8B, 6.9B, 12B. The results for a shortlist of these experiments can be found in Figure (for more, see Figure 69). First up. these results establish that even the LLM trajectory maps have high directional redundancy across scales, suggesting widespread applicability of our key finding.
(a) Apex Angle L(,,, 0%)
Origin
|0:|
(c)
||0: 0g
Figure 7: Directional complexity measures correlate better with performance than norm-based ones.
Detailed Observations. Elaborating further, in Figure 6, we find a tiny dark square grid in the upper-left comer, which delineates precisely the learning rate warmup phase. Next, the trajectory
7

---

Published as a conference paper at ICLR 2025
map for the 160M model (and smaller) seems to have discemible substructure, but which becomes more homogeneous with increasing model scale. In general, increasing scale lends an intense dark hue to the trajectory maps, suggesting rising directional similarity. Even the horizontal and vertical slivers (see the first few rows and columns) corresponding to the warmup phase start to assume a higher cosine similarity with scale. Overall, this seems to suggest that the larger models are biased towards directionally regular optimization trajectories.
Directional measures correlate with performance better than norm-based measures. In fact, Figure 7(a) shows that the apex angle made at the origin by the trajectory monotonically decreases with increasing scale. This is in contrast to norm-based complexity measures such as overall parameter norm (Gunasekar et al., 2018) and distance from initialization (Nagarajan and Kolter, 2019) which seem to be poorly correlated to performance, as visible Figures 7(b) and 7(c). Norm-based measures fail to capture the monotonic trend, and even have the largest complexity value for the pest-peeforming model.
Why do parameters become aligned with scale? A Theoretical Argument. We prove, in Appendix B, that this surprising finding about the progressive increase in cosine similarity with scale has relatively simple explanation, at least in the case of the large-width limit of deep networks. The gist of our argument is that in the large width limit, any parameter updates that lead to stable feature updates must necessarily yield updated parameters that are identically aligned with their initialisation. While this is not so surprising for lazy learning regimes like the Neural Tangent Kernl (Jacot et al., 2020) where feature learning does not occur, more surprising aspect of our finding is that this is necessarily true also feature learning regimes, such as pP (Yang et al., 2022).
Section 5 Key Takeaways:
LLMs up to 12B parameters exhibit like patterns of directionally redundant trajectory maps.
Apex angle traced at origin monotonically decreases with increasing model scale, as opposed to traditional Norm-based measures, thus showing promise as useful complexity measure.
6 LEVERAGING DIRECTIONAL REDUNDANCY FOR EFFICIENT TRAINING
Our trajectory map analyses have revealed significant directional redundancy in the optimization trajectories of broad range of neural networks, which is especially prominent later into training. While we have seen that the macro-level directional movement saturates early (Section 3), micro-level oscillations continue to persist in the late phases (Section 4). We thus hypothesize that once this directional saturation is achieved, the full cupacity of a network might not be required for training. If true, this would motivate optimization hybrids for efficient training.
How to best decapacitate the network? As the primary desiderata, we want to (i) retain the least number of parameters possible, (ii) ensure that the capacity is removed in a structured manner to real- ize speed-ups (unlike, unstructured sparsity), (iii) intervene minimally in the implementation workflow. Clearly, our desiderata is stringent; but it allows testing the above hypothesis more compellingly.
Teajectory Maps ResNet20, CIRAR10
Trining just Scalers
Chacipoti Iodox
Codee Stilley
Full-Network
Accurscy
Cheskgott
Indes
Figure 8: Training only the BN scalar parameters suffices after the directional movement saturates. The trajectory map of the original, full network, training is shown on the left and, in the bar chart the right, the bars denoting the performance achieved by training BN parameters from a particular epoch. 'Baseline' denotes the network accuracy just before switching to optimizing the BN parameters only.
8

---

Published as a conference paper at ICLR 2025
Approach: Training only Normalization Layers. This meets our desiderata since (i) normalization layers have typically < 1% of the parameters, (ii) all other layer types are frozen and do not need gradient updates, thus ensuring speed-up is realizable, (iii) this is essentially Also, (batch/layer/groopp) normalization layers are ubiquitous in modern architectures. Further, this re- purposing of normalization layers was inspired by Frankle al. (2021) who empirically demonstrated the cetter-than-expecee performance when training just the scalar parameters in batch-normalization (BN) layers from initialization, well theoretical expressivity results (Burkholz, 2024).
line change in code.
Experimental Setup. We experiment with ResNet20 on CIFARIO using SGD over 160 epochs, freezing non-Bi layers at different points in training. and then using only the 1,376 scalar parameters in the BN layers for the remaining training duration. Figure 8 presents the results alongside the trajectory map of the full network training.
Observations. (i) Training just BN layers from initialization, as claimed in Frankle et al. (2021), results in network with a test aceuracy of 54.8%, which although interesting falls considerably short of the 91.2% test accuracy obtained by training the entire network. In way, from the horizontal strip of trajectory map around we see that at this stage the directional movement is yet to saturate. (ii) However soon after, from around 40 epochs, where the trajectory map develops dark hue (and the cosine similarity to the final parameters is ~ 0.9), training BN parameters alone leads to within 2% of the full-network accuracy. (iii) From around 80-th epoch, this gets to within 1%. and completely matches thereafter. Notably, this feat is achieved by training only 0.5% of the overall parameters, and this fares even better than training all the parameters in the bulkier last layer shown in Figure 77.
Similar findings on ImageNet. Likewise, in our experiments with ResNet50 on ImageNet as in Figure 2, we found that although training BN layers right from initialization gets top-1 accuracy of just ~ 6.4%, but this decapacitated training from epoch 30 gets to within ~ 10% of the full-network's accuracy and from epoch 60 within about 2% of This is again a striking result, considering only 0.18% of the parameters are being trained.
Benefits for Efficient Training. Besides the runtime saved in backward pass, this also leads to savings in the GPU memory consumption as the optimization buffers (like the momentum buffer or those used for preconditioning adaptive methods) now need to be of significantly smaller size (typically < 1%, i.e., 99% savings), which could, in turn, make larger batch sizes feasible.
Broader Implications. The above results pointedly demonstrate that the above hypothesis holds through the strong desiderata fairly well. More broadly, we envision that this proof-of-concept can be readily adapted into hybrid optimization scheme, and possibly with more flexible desiderata that possibly allow for interleaved training with other parameters or equip more parameters to start with thereby contributing to non-trivial cost savings. While our computational resources restrict us vision models, anticipate many of these insights will carry over to LLMs, given the observed similarities in their trajectory maps. We encourage the broader research community to explore the decapacitation idea as means optimize LLM training and spark further innovation in this respect.
Section 6 Key Takeaways:
When directional movement saturates, neural networks can be severely decapacitated and trained only through parameters in normalization layers.
Our results CIFAR10 and ImageNet show that this comes with little compromise in performance, while making training much more efficient.
7 RELATED WORK
Directional Redundancy and Trajectory Structure. Prior work (Ji and Telgarsky, 2020) has theoretically noted a notion of directional convergence, wherein the parameters of simple networks and classifiers converge quickly, in terms of their direction. Likewise (Merrill et al., 2020) have observed that cosine similarity between subsequent parameter checkpoints during T5 (Raffel et al., 2023) pre-training rapidly approaches one. In a similar vein, SiilleeEscuret et al. (2023) find that gradients tend to have moderate but positive alignment with the solution direction, all throughout training. Further, in a recent work (Dherin and Rosca, 2024), propose the existence of regions called "corridors" in (full-batch) gradient descent training, whereby the trajectory behaves like line segments.

---

Published as a conference paper at ICLR 2025
All of these can be seen diverse facets of the core phenomenon of redundancy in the directional movement, which we elaborate in depth here. In comparison, our work lays out a comprehensive analysis of the directional aspects of trajectories, which we study through trajectory maps and related directional measures, and where we uncover different levels of directional movement and their mechanisms thus contributing significant nuance over existing literature.
Implicit Effects of Hyperparameters. Implicit bias (Gunasekar et al., 2018; Li et al., 2019; 2020; Moroshko et al., 2020; Barrett and Dherin, 2020) has emerged as one of the main contenders for explaining the success of deep neural networks. This principle has also inspired several works which seek to uncover the implicit effects that might be latent in the regular working of hyperparameters. To name few, Andriushchenko et al. (2023) for instance suggest a loss stabilization mechanism behind weight decay, while (Liu et al., 2023) attempt to characterize the implicit bias of large learning rates in terms of resulting in a flatter solution, and Jelassi and Li (2022); Cao et al. (2023) explain the implicit bias of momentum and batch normalization with regards to margin. In contrast, we showed the implicit effects of momentum and weight decay from trajectory perspective, i.e., on the increased directional movement, as well as how these two hyperparameters interact with each other.
Decapacitation. This might be reminiscent of the lottery-ticket hypothesis (Frankle and Carbin, 2019), with a subtle but key distinction that the lottery-tickets are concerned with sub-networks (certain neurons are set to zero) while here the network structure remains the same and the capacity is stripped by freezing the parameters. Also, the mechanism of finding lottery-tickets is retrospective in nature; while here we present a mechanism decapacitate that depends the current network parameters. However, it would be worth investigating the connection further, especially given their shared reliance on the early phase of training albeit through directional saturation versus linear mode connectivity (Frankle et al., 2020). Besides, a similar exploration of tuning normalization parameters has been tried for fine-tuning (Zhao al., 2023), although not in the context of (pre-)training which is our focus here.
8 DISCUSSION
Summary. We observe that optimization trajectories in neural networks exhibit a rich directional structure, across wide range of architectures, datasets, optimizers, and hyperparameters. This is firstly visible through the trajectory maps, which generally have high mean dimensionality scores. Next, as the angular measures reveal, the directional movement persists throughout training, but resides at varying levels, with the macro-level behaviour in the early training phase and micro-level oscillations subsequently. We utilize this to come up with and demonstrate network decapacitation, whereby once the directional saturation is reached, freezing significant network capacity leaves the performance unaltered thus showcasing the potential for efficient training of neural networks.
Future Work. The two most interesting directions of future work are as follows:
(A). Testing the Generalization Potential of Directional Measures. We have seen in Sections 3 and 5, that directional measures are correlated with generalization. In particular, they appear to exhibit U-shape trend, where the extreme values should clearly be avoided. A dedicated study focusing exclusively on the generalization potential of directional measures, and comparing these findings to flatness-based analyses, would be highly insightful. Intuitively, flatter solutions will contain several low curvature directions in their vicinity. which would align well with the discussion here.
(B), Analysis of Layerwise dynamics. Another interesting aspect of trajectory maps is that they can be easily extended to hone into directional dynamics at layerwise level. Our initial analysis into GPTNeoX models discussed in Section 5 shows that the layerwise trajectory maps inhabit a rich struc- ture, as shown in Figure 75. In particular, for models with less than billion parameters, the Q,K.V V trajectory maps show significant heterogeneity in the timescales of directional convergence, with middle layers converging the last directionally. Strikingly, the Q.K.V V dynamics homogenize across depth with higher scale. Morcover, these intriguing observations suggest utilizing trajectory maps for (data-free) mechanistic understanding, complementing (Elhage al., 2021; Grosse et al., 2023).
Conclusion. Overall, we have merely seratched the surface of this trajectory perspective into understanding optimization behaviour in neural networks. We hope this work will bring further nuance in these areas and contribute towards hybrid optimization schemes that can exploit the showcased directional redundancy.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
We would like to thank Benoit Dherin and Tiago Pimentel for reading a draft of the paper and providing useful comments. Also, we thank Weronika Ormaniec for catching typographical mistakes and making useful remarks. Sidak Pal Singh would also like to acknowledge the financial support from Max Planck ETH Center for Learning Systems.
REFERENCES
Suriya Gunasekar, Jason Lee, Daniel Soudry, and Nathan Srebro. Characterizing implicit bias terms of optimization geometry. In International Conference on Muchine Learning, pages 1832-1841. PMLR, 2018.
Yuanzhi Li, Tengyu Ma, and Hongyang Zhang. Algorithmic regularization in over-parameterized matrix sensing and neural networks with quadratic activations, 2019.
Yuanzhi Li, Colin Wei, and Tengyu Ma. Towards explaining the regularization effect of initial large learning rate in training neural networks, 2020.
Edward Moroshko, Suriya Gunasekar, Blake Woodworth, Jason D. Lee, Nathan Srebro, and Daniel Soudry. Implicit bias in deep linear classification: Initialization scale vs training accuracy, 2020.
David GT Barrett and Benoit Dherin. 22Xi:20099.11622 2020.
Implicit gradient regularization. arXiv preprint
Yann N Dauphin, Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, Surya Ganguli, and Yoshua Bengio. Identifying and attacking the saddle point problem in high-dimensional non-convex optimization. Advances in neural information processing systems, 27, 2014.
Anna Choromanska, Mikael Henaff. Michael Mathieu, Gérard Ben Arous, and Yann LeCun. The loss surfaces of multilayer networks. In Artificial intelligence and statistics, pages 192-204. PMLR, 2015.
Kenji Kawaguchi. Deep learning without poor local minima. Advances in neural information processing systems, 29, 2016.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. Advances in neural information processing systems, 31, 2018.
Ian J Goodfellow, Oriol Vinyals, and Andrew M Saxe. Qualitatively characterizing neural network optimization problems. arXiv preprint 1412::44 2014.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In International Conference Machine Learning, pages 3259-3269 PMLR, 2020.
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 3519-3529, PMLR, 09-15 Jun 2019. URL https: //proceedings..o mlr ppress/ppe/////cee...ee... v97 ht77//ornblihhh htt.lll...........m
Thao Nguyen, Maithra Raghu, and Simon Kornblith. On the origins of the block structure phe- nomenon in neural network representations. arXiv preprint arXiv:2202.07168 2022.
Richard D Lange, Devin Kwok, Jordan Kyle Matelsky, Xinyue Wang, David Rolnick, and Konrad Kording, Deep networks as paths on the manifold of neural representations. In Timothy Doster, Tegan Emerson, Henry Kvinge, Nina Miolane, Mathilde Papillon, Bastian Rieck, and Sophia Sanborn, editors, Proceedings of 2nd Annual Workshop on Topology, Algebra, and Geometry in Machine Learning (TAG-ML), volume 221 of Proceedings of Machine Learning Research. pages 102-133. PMLR, 28 Jul 2023. URL https: / http://dooeeeeeeeeeeeeeeeeececee mlr. wwwpsseeeaaae v221/ lange233 html.
11

---

Published as a conference paper at ICLR 2025
Damek Davis, Jonathan Balzer, and Stefano Soatto. Asymmetric sparse kernel approximations for large-scale visual search. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2107-2114, 2014.
Qi Chen, Bing Zhao, Haidong Wang, Minggin Li, Chuanjie Liu, Zengzhong Li, Mao Yang. and Jingdong Wang. Spann: Highly-efficient billion-scale approximate nearest neighbor search, 2021.
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization, 2021.
Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: General- ization beyond overfitting on small algorithmic datasets, 2022.
Hongchao Zhang and William W Hager. A nonmonotone line search technique and its application to unconstrained optimization. SIAM journal on Optimization, 14(4):1043-1056, 2004.
Sharan Vaswani, Aaron Mishkin, Issam Laradji, Mark Schmidt, Gauthier Gidel, and Simon Lacoste- Julien. Painless stochastic gradient: Interpolation, line-search, and convergence rates. Advances in neural information processing systems, 32, 2019.
Clare Lyle, Zeyu Zheng, Evgenii Nikishin, Bernardo Avila Pires, Razvan Pascanu, and Will Dabney. Understanding plasticity in neural networks. In International Conference on Machine Learning, pages 23190-23211. PMLR, 2023.
Shibhansh Dohare, Fernando Kernandez-Garcia. Qingfeng Lan, Parash Rahman, A Rupam Mah- mood, and Richard S Sutton. Loss of plasticity in deep continual learning. Nature, 632(8026): 768-774, 2024.
Levent Sagun, Utku Evei, V Ugur Guney, Yann Dauphin, and Leon Bottou. Empirical analysis of the hessian of pver-parametrized neural networks. arXiv preprint crXiv:/706.04455 2017.
Sidak Pal Singh, Gregor Bachmann, and Thomas Hofmann. Analytic insights into structure and rank of neural network hessian maps. Advances in Neural Information Processing Systems, 34: 23914-23922 2021.
Jeremy M. Cohen, Simran Kaur, Yuanzhi Li, J. Zico Kolter, and Ameet Talwalkar. Gradient descent on neural networks typically occurs at the edge of stability, 2022.
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O' O'Brien, Fric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pages 2397-2430. PMLR, 2023.
Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, et al. Gpt-neox-200 An open-source autoregressive language model. arXiv preprint 2wXiv:2204.06745, 2022.
Vaishnavh Nagarajan and J. Zico Kolter. Generalization in deep networks: The role of distance from initialization, 2019. URL htthttts//axxiiiii org/ abs/1901. 01672.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks, 2020.
Greg Yang, Edward J.
Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tensor programs v: Tuning large neural networks via zero-shot hyperparameter transfer, 2022.
Jonathan Frankle, David J. Schwab, and Ari S. Morcos. Training batchnorm and only batchnorm: On the expressive power of random features in cnns, 2021.
Rebekka Burkholz. Batch normalization is sufficient for universal function approximation in CNNs. In The Twelfth International Conference on Learning Representations, 2024. URL https: //oppereevi n forum? d=w0Y f
12

---

Published as a conference paper at ICLR 2025
Ziwei Ji and Matus Telgarsky. Directional convergence and alignment in deep learning, Advances in Neural Information Processing Systems, 33:17176- 2020.
William Merrill, Vivek Ramanujan, Yoav Goldberg, Roy Schwartz, and Noah Smith. Effects of parameter norm growth during transformer training: Inductive bias from gradient descent. arXiv preprint arXiv:2010.09697 2020.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer, 2023.
Charles Guille-Eseuret, Hiroki Naganuma, Kilian Fatras, and Ioannis Mitliagkas. No wrong turns: The simple geometry of neural networks optimization paths. arXiv preprint arXiv:2306. 2023.
Benoit Dherin and Mihaela Rosca. Corridor geometry in gradient-basel optimization. arXiv preprint arXiv:2402.088/88 2024.
Maksym Andriushchenko, Francesco D' Angelo, Aditya Varre, and Nicolas Flammarion. Why do we need weight decay in modern deep learning?, 2023.
Chunrui Liu, Wei Huang, and Richard Yi Da Xu. Implicit bias of deep learning in the large learning rate phase: A data separability perspective. Applied Sciences, 13(6):3961, 2023.
Samy Jelassi and Yuanzhi Li. Towards understanding how momentum improves generalization in deep learning, 2022. URL https:////// openreview. net/ forum? fdm11fo66
Yuan Cao, Difan Zou, Yuanzhi Li, and Quanquan Gu. The implicit bias of batch normalization in linear models and two-layer linear convolutional neural networks. In The Thirty Sixth Annual Conference on Learning Theory, pages 5699-5753. PMLR, 2023.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks, 2019. URL https::///// arxiv. .rrg/ hasp//b///// /tab//880k 1803. 03635.
Bingehen Zhao, Haoqin Tu, Chen Wei, Jieru Mei, and Cihang Xie. Tuning layernorm in attention: Towards efficient multi-modal I1m finetuning. arXiv preprint arXiv:2312.11422 2023.
Nelson Elhhge, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, et al. A mathematical framework for transformer circuits. Transformer Circuits Thread, 1:1, 2021.
Roger Grosse, Juhan Bae, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, Evan Hubinger, Kamilé Lukovsiūté, Karina Nguyen, Nicholas Joseph, Sam McCandlish, Jared Kaplan, and Samuel R. Bowman. Studying large language model generalization with influence functions, 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pages 1026-1034, 2015.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.
Lenaic Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differentiable programming. 2020.
Tanishq Kumar, Blake Bordelon, Samuel J. Gershman, and Cengiz Pehlevan. Grokking as the transition from lazy to rich training dynamics, 2024.
13

---

Published as a conference paper at ICLR 2025
Part I
Appendix
Table of Contents
A Omitted Proofs
15
B Why cosine similarities increase with scale?
16
C Comparing Gradient Trajectories with Random Walks
17
C.I Experimental Results
17
C.2 Expected MDS values for Random Walk
19
D Relative Trajectory Maps
21
E Detailed Experimental Results
21
E.I ResNet50: Switching off the hyperparameters
21
E.2 Additional Discussion on Effects of Momentum
24
E,3 ResNet50: Weight Decay, AdamW
24
E,4
Additional Discussion on Weight Decay
24
F
Directional Effects Other Key Settings
27
VIT: Weight Decay, Adam
................................
27
F2
ResNet50: Weight Decay, SGD
................................
30
F3
ResNet50: Sharpness Aware Minimization analysis
................................
33
ResNet50: Momentum Analysis, LR 0.1, WD 0.0001
................................
36
VGG: Momentum Analysis, 0.1, 0.0001
39
F6
Momentum Analysis, 0.1, W
40
F7
VGG: Momentum Analysis, 0.01, WD 0.0001
................................
42
F8
VGG: Momentum Analysis, LR 0.01, WDO
................................
44
F9
VGG16 Batch Size Analysis
46
F.10
Trajectory Maps in the presence of label noise
48
G GPT-NeoX Trajectory Analysis
50
H Tryjetose Maps
52
H.l Q.K.V Trajectory Maps across Scales
54
I Trajectory Maps for Grokking
55
J Putting Directional Redundancy to Test
56
Comparison with (discretized) Trajectory Map J.1
56
K Effect of different learning rate schedules
57
K.1 Trajectory Maps
57
K.2 Angle between successive epochs
58
L Effect of Trajectory Granularity
59
L.1 Trajectory Maps with finer granularity
59
L.2 Angle between successive Epochs
59
L.3 Fine-Grained view into the first epoch
61
Angle between successive Epochs for very large batch size training
61
14

---

Published as a conference paper at ICLR 2025
A OMITTED PROOFS
Lemma 2. Given a quadratic problem with & regularization of strength a > 0, namely,
millERR 10 M0 + = with M E Raxd symmetric with eigenvalues X > .... > Xa. the angle
between successive steps D. 0, 0, D+t1 0 0,, when using gradient descent with one-step momentum (p 0) and learning rates nt. 1t+1, can be upper and lower bounded as follows:
L A.41) & 7 (1 melf + a + Aa)) a=)210,-111
(Ai, A.+1) M01+ 11 m=(u+ a + (1) + aa) 10-11
Proof. Given function f()) 0
M0 + f 0 the gradient 0 will be Vf(0) (M + aT)0.
Then at the first optimization step, we do
0, =
+ al)0;-1
The particular update being A := 0, 0,-1 = - (M + aI)0,-1. The next update is similar, but now also have to factor in the momentum,
P 0: M++1 (V)(0:) MM aI)o:-1)
Aet1 = 0s+1 0: 1|++1 (M al)0: 1u) (M aI]0:-1)
-- (M + aI)0 7/ (M + al) um (M al)0,-1) --]+1 ((1 jum om) M) (M a1)0,-
Now, let us evaluate the inner-product (A,, Aet1),
(1,,4+1)) 7em-10,1 [ aI) ((1 pml m MM (M 0
Now without loss of generality we can consider Z be a diagonal matrix, as Z is symmetric since M is symmetric, we can consider its spectral decomposition Z = UDUT and project 0:-1 onto its cigenvectors contained U. With this the matrices the middle are diagonal and we can commute them, which yields us the following matrix:
(1 pr 160 - - a)
Z diag
(1 pMP 7,a (>a a)²
where, we have denoted the eigenvalues of M as A > Aa-
Since the inner product of the updates a quadratic form, we can upper and lower bound based on the maximum and minimum eigenvalues of Z, thus giving:
MeTk+ Amin (o)nnnZZZ(00.. (Z)]
& mell+ Amax (Z)
Because of the above form of eigenvalues of Z (diagonal matrices have their eigenvalues as their diagonal entries), we will have:
Amax (Z) (1- pm 7/ T)d + a²² and Amin (Z) (1- -m) 7:a (A) + O
15

---

Published as a conference paper at ICLR 2025
B WHY COSINE SIMILARITIES INCREASE WITH SCALE?
Note, we assume that the majority of the parameter norm lies in the square hidden matrices, and not the input or output layers. Moreover, we use o, O,0 denote standard mathematical notation with regards to scaling in the limit width n c0. For vectors, this notation is entry-wise.
Suppose we have hidden layer with input zn IR for width n1, that is acted by (without loss of generality) a square matrix Wo E R=x= to give:
ho WozO
We suppose x has (() entries, as is the case with standard mntialistioon/ppaannetttsssoosss He et al. (2015). We suppose Wo has i.i.d elements with initialisation that is O(1/vT) in order ensure that each element the features h has entries (1).
Now, if take gradient update with learning rate 77 on some downstream loss L that depends on h (and not or æ)), we get:
W1 = Wo rjch 20
where dh OL R=*! is our feature derivative. 8h
Then if have new input æ (wlog 21 zo), we have new features:
h1 x Wi ho nndhh Ip z0
For features to be stable (i.e. 0(1)) after the update, need nndh to be O(1,, because 2n So
0(1) by assumption on NB: if nnjdh = o(1) we have no feature learning (ie NTK regime Jacot et al. (2018)), and nundh = (1) we have feature learning (ic jP Yang et al. (2022)).
In any case, 1/d/ (1/nn) entry-wise, which means that W1 Wo -rdhh ag has ((1/n) entries, again by assumption on the scale of elements of æo.
But because Wo 0(1/vn)) the initialisation will elementwise-dominal the O(1/n) update for the first training step (and more training steps follows by induction). As result, the update WT - Wo will always be order of at least /n smaller than the initialisation, and hence the new parameters Wr will be exactly aligned with the initialisation Wo for all T the large width limit, i.e. the cosine similarities will be 1.
16

---

Published as a conference paper at ICLR 2025
C COMPARING GRADIENT TRAJECTORIES WITH RANDOM WALKS
The structure that we observe in trajectory maps following gradient trajectories raises the question of we would observe similar structure in random walk.
If we have T timesteps or epochs, with parameter space E IRP and "learning rate" schedule (17, (7)))=1> can consider random walk with updates:
0, - 0: ind. N(O, Tp)
which to say that time step 1, each parameter coordinate in the parameter vector is updated independently with Gaussian of variance 1/ū, and the updates are independent across different time steps.
Then, if 0' denotes single parameter coordinate for dimension < for two time steps < I, we have:
(0,, ~ NOO (H, ))
where H. =- nh is the cumulative squared learning rate from t' 1 to t'
Then, by the strong law of large numbers we have for the large parameter space - oc limit:
1. 1 Σ (0:)2 6. H.
110.12
(0)) a.s. He
1 0 0.) = Σ
P
000 a. H.
p
and by the property of composing almost sure limits, we also have almost sure convergence in the cosine similarity:
(0.. 0,> |0 0
a.e. H.
H. V H,
VH,H.
in the large parameter space limit, which we use as an approximation to give analytic formulas for the trajectory map and MDS that we can compare gradient trajectories.
One thing note, is that this cosine similarity V H. becomes invariant to the scale of the learning rates and instead it is the relative rate of decay in the learning rate schedule that matters.
C.I EXPERIMENTAL RESULTS
The first thing to note is that our empirical simulation of the random walk matches the theoretical limit described in the section above, for finite parameter count (such as 10, 000). Next, comparing the these relative trajectory maps with those for ResNet50 (Figure 10) we find that the latter reveal much more directional redundancy component their trajectories opposed to random walks. This further lends support to the thesis that optimization trajectories ensued when training neural networks highly structured and have significant directional redundancy.
An additional thing to note is that the above relative trajectory map for random walks covers the setting of decreasing the step size to mirror how the optimization procedure is setup for ResNet50. In the case of no such step size decay, the analytic and empirical versions of the relative trajectory map depicted in the Figure 12.
17

---

Published as a conference paper at ICLR 2025
Analytic Browaian Motion
Samalated Brownisn Motion (3-10000)
Charkpaint Index
04
133
92
Checkpoint Index
Checkpoint Index
Figure 9: With step size decay: Relative Trajectory Map for a Random Walk/Brownian motion in both analytic and empirically simulated settings.
B.U.A 0.0023
3.2003
--0.A
Figure 10: Relative Trajectory Maps, with respect to initialization, of ResNet50 models for different amounts momentum and weight decay.
mom 0.9.wd 0.0001
mom 0.wd 0.0001
mom 0.9.wd
mom Ouwdi 0
A
64
Lheckpaint
52
52
Checkpoint Index
Figure 11: Relative Trajectory Maps, with respect initialization, of ResNet50 models when subtracting the trajectory map of Random Walk (shown in Figure 9) with corresponding learning rate schedule. As before, this is carried out different amounts of momentum and weight decay.
Brownian Motion
Sim Brownien (3-10000)
100
Chac-pponn Index
07
04
Checkpoint Index
Checkpoint Index
Figure 12: No Step size decay: Relative Trajectory Map for Random Walk/Brownian motion in both analytic and empirically simulated settings.
18

---

Published as a conference paper at ICLR 2025
C.2 EXPECTED MDS VALUES FOR RANDOM WALK
Effectively, to get the MDS, we need to compute the expected mean entry of the matrix whose (8, () entry contains the cosine similarity between O, and O, in the described random walk, we'll start by noting the following:
In the large parameter space limit (p- +0), the cosine similarity between 0₂ and O₂ converges almost surely to
min Hs, Ha)
COs 0,,t
mnxxHH,, Hs)
where Hu = -e-1 nh is the cumulative squared learning rate up to time u.
For simplicity, let's consider the case where the learning rate is constant: Tte 1y for all t. Then, H₃ = b17", and the cosine similarity simplifies to
min (s, t) max(s,
cos 02.
The expected mean entry of the matrix is the average of all these cosine similarities:
T
min(s, t) max(s, t)
Mean
72
#=1
Due to the symmetry of the cosine similarity, we can write this as:
T 7-1
T
T
Σ
Mean
7
Mean
S+ +
T2
where
T-1 T S
Σ
To compute S, we approximate the sum by an integral for large T:
S
dl ds
s-1 1
Calculating the integral:
8 dt = 2s 22 -
((s) =
J/ V
T
T
T
S= 12 3112) ds 1/2
1/2 ds -
s ds
J--1
J==1
J==1
Computing the integrals:
T
1/2 ds
2
(T32 1), s=1
sds
1 (T2 - 1)
3
2
s=1
Plugging back:
2
(T3/2 (T//2- 11) - 11)
(2 -1)
S = 17
3
S 1/2 - 11 -TT2 - -1)
S 3
( 1112 - (T2 - 1)
19

---

Published as a conference paper at ICLR 2025
4
4
S
-72 T2
-
T
3
3
- T 71/2 +1
4
3
Now, under the large T assumption, we have as the MDS:
MDS
73 S
+
2 7
7
T2/2
+
1
1 T
72
T
3
3
2
8
2
1
MDS
T
(I)
3
T2
1
1
As T J0, the terms involving
and
vanish, leaving:
T
73/2¹
T
2 MDS
3
20

---

Published as a conference paper at ICLR 2025
D RELATIVE TRAJECTORY MAPS
In some cases, it might be useful to analyze the trajectory relative to some point 0- as the origin, so there we will instead consider the of points T, {0, 0, Or and correspondingly organize of the points of the trajectory, then will
it in the matrix + E (2 When is itself omit the row of zeros and shape the matrix as RTxx, A natural point from where to contextualize the trajectory would be the initialization q and this relative trajectory will then be denoted Oo (where the subscript 0 is not to be confused for the usual origin O, namely, 0o 0).
Trajectory Map. Analyzing the matrix θ or θ, on its own, might get cumbersome as the size of modern networks ranges in millions and billions of parameters. Hence, we will resort to looking at functions of the kernel matrix K θ0* which would be square matrix of shape T depending if the
T 1, or
for * O, the relative kernel matrix K, o of shape T point 0 is a part of the trajectory or not. Further, it will also helpful to isolate and analyze the directional aspect of the trajectory, for which will normalize the set of points by their norm, and in 0.-0. 17 with the respective matrix θ As a result, the ensuing effect, consider the set T- 10.-0.12
kernel matrix θ θ, which we will refer to as C, (or C Co for the usual origin O), will contain the relative cosine similarities between every pair of points in the trajectory. So, (C,)iy is,
( 0, 0 0,)
cos-sim (0: - 0,,0 - 0,) =
0, 0 0, 1
We will refer to C as the Trajectory Map (TM) and C, (for t + O) as the Relative Trajectory Map (RTM).
E DETAILED EXPERIMENTAL RESULTS
E.I RESNET5O: SWITCHING OFF THE HYPERPARAMETERS
2-000-
Figure 13: Trajectory Maps of ResNet50 models across different amounts of momentum and weight decay
The relative trajectory maps can be found in Figure 10.
21

---

Published as a conference paper at ICLR 2025
143
(a) 2(1 o2, 0c)
(b) L(0.41 O2, Or 0%)
12h4-64 -.altc 1-00
Rpecko
(c) ((0++h 0e, 0g kx-k), for Ao
(d) ((0, 0o, Or 0v)
CEO
(0) 2(9++1 c., 0r 0o)
(f) Apex Angle at Initialization L(e. 0o)
0o. 0
20m 91480 01008
Marn
CROOK
Ebeeder
(g) Apex Angle at Origin Z(0,, 0o)
Figure 14: Angular measures of the Trajectory for ResNet50 trained on ImageNet
22

---

Published as a conference paper at ICLR 2025
ECATT
(a) |0=|12
(b) 10.4
0r|12
(c) 10c
00l
Figure 15: Norm-based measures of the Trajectory for ResNet50 trained ImageNet
Train Accunacy over Stapa
Tint Ausurass over Epochs
Thain LANK
Tot Accunnce
E.A trer
X0001
30000
20000
30000
4000
Steps
Steps
Epochs
(a) Train Loss
(b) Train Accuracy
(c) Test Accuracy
Figure 16: Loss and accuracy values for Momentum, Weight Decay experiments. The final test accuracy values are 75.96, 72.63, 70.86, 68.73, in the order listed in the figure legend.
Weight Decay together
Weight Decay togother
Pearson: 0:50
Pearson: 0.83
Spearman:
Spearasan
1eet Accurary
74
Tocl. Aceussy
74
71
09
0.50
0.83
2:00
0.55
0.955
0.900
IUS23
D-230
0.005
0LD40
6.445
Moan Directionality Siailarity
(molative)
Diractionality Sieilarity
(a) Correlation of MDS
(b) Correlation of Relative MDS
Figure 17: Correlation plots of MDS and relative MDS with test performance.
23

---

Besides, in Figure 3(a) and 14(f), we find that in the presence of momentum, a larger angle is traced the origin by the trajectory, suggesting initialization is smaller. The latter can also be seen from Figure 15(b), since with momentum, the trajectory moves further away from the initialization. Apart from this, the absence of weight decay, the updates seem to be strengthening with momentum and the parameter norm rises 15(a) as well, least Published as a conference paper at ICLR 2025
E.2 ADDITIONAL DISCUSSION ON EFFECTS OF MOMENTUM
directional exploration, while the angle traced
giving rise to mental picture of trajectory similar to that left purple trajectory in Figure until the training hits EoS.
With weight decay, as there a decrease parameter norm Figure 15(a) alongside the EoS process. as well as due to the presence of larger obtuse angles, expect reasonable affinity with our illustration, where we see the updates oscillating and slowly drifting towards the origin below.
E.3 RESNET50: WEIGHT DECAY, ADAMW
2-081
2-080L
1.4
Figure 18: Trajectory Maps of ResNet50 models across different amounts of weight decay
X+C3
PHO3I
2-0
Figure 19; Relative Trajectory Maps, with respect to initialization, of ResNet50 models across different amounts of weight decay
E.4 ADDITIONAL DISCUSSION ON WEIGITT DECAY
Let us turn to weight decay alone and understand its directional effect. We have already noticed the increase in mean directional similarity (MDS) when weight decay is disabled for ResNet50 trained with SGD on ImageNet. In fact, we find a similar effect with an adaptive optimizer, like AdamW (Loshchilov and Hutter, 2017) the trajectory maps for which Figure E.3. Here, we used regularization constants from À = 0 until the first value where we witness decrease test performance, which this case was 1. Specifically, we analyze the weight decay coefficients in À E {1, 0.1, 0.01, 0.001, 0). The corresponding MDS come out to be, 0.731, 0.679, 0.844, 0.882, 0.885. We notice that, as before, increasing weight decay leads to a heightened directional exploration, or lower MDS; except À = being the seeming anomaly.
shown in
But find that this can be remedied simply by looking the relative trajectory maps (Fig- ure E.3), and computing the relative MDS, i.e., is 0.985, 0.807, 0.862, 0.897, 0.900 for À 0.001, 0 respectively. This occurs since such a high weight decay À 1, causes this particular network to underfit (train/test top-1 accuracy the rest the networks improves, more or less, as expected with weight decay, and in particular, achieve accuracies of 75.45%. 73.38%, 71.03%, 71.41%.
1, 0.1, 0.01, 0.0,0
54.63%, 50.52%). The performance for
Having reaffirmed our results extensively about the directional exploration due to weight decay, we understand it through simple physics-based intuition, shown in the Figure 5. In particular, we can think of the loss gradient pulling the network parameters rightwards, while the force exerted by weight decay tries pull the network downwards. The relative strengths of these 'focces'
24

---

Published as a conference paper at ICLR 2025
have been represented by the lengths of the two vector arrows. We notice that as the weight decay strength is increased, from the left subfigure to the right, the angle traced at the origin (O) also increases. This explains how weight decay contribute towards directional exploration.
113
130
(1h
<(0c+1 O2, 0t)
(b) <(1++1
Oe, Or Os)
(c) 2( - 0c 0: _: k) for lt -1
(d) L(0: 0 Or 0v)
55
(c) / (
O,, Or 00)
(f) Apex Angle at Initialization L(, 0o)
0o.
3010 1448- 070ck (0.00
(g) Apex Angle at Origin (0,,, 0o)
Figure 20: Angular measures of the Trajectory for ResNet50 trained on ImageNet
25

---

Published as a conference paper at ICLR 2025
Mengle
Sonde
(a) |00||l2
(b) 10.4x 0:112
(c) 10. 0ol
Figure 21: corm-based measures of the Trajectory for ResNet50 trained on ImageNet
Truin oour Staps
Train Accuracy
Accuracy
Tot Accunay
Thain
30000
Train Loss
(b) Train Accuracy
(c) Test Accuracy
Figure 22: Loss and accuracy values for Weight Decay experiments. The final test accuracy values are 50.54, 75.45, 73.38, 71.03, 71.41, in the order listed in the figure legend.
Duffersnt Weight Decay (ndamW)
Difforent Weight (Adam!h)
Pearson: 0.54
Paacsoo: - 3.99 Spearman:
Speattman:
75
Tost Accurary
Test Accurasy
72
0.73
0.75
0,90
0.35
00822
0.54
1:80
0.3
Meon Directionality Similarity
(celative) Mean Directionality Similarity
Correlation of MDS
(b) Correlation of Relative MDS
Figure 23: Correlation plots of MDS and relative MDS with test performance. We omit the case where the weight decay coefficient is set to the extremely large value of 1 such that the network severely underfits (train accuracy < 40%).
26

---

Published as a conference paper at ICLR 2025
F DIRECTIONAL EFFECTS IN OTHER KEY SETTINGS
In addition to the momentum and weight decay, there are other crucial hyperparameters, such as learning rate and batch size, whose directional effects warrant mention. We carry out additional experiments in these settings, and from where, the key findings are that the learning rate, as it would be easy to guess, indeed encourages directional exploration leading to low MDS scores. But, somewhat more interestingly. we find that increasing the batch size also helps further exploration and thereby decreases the MDS scores. The trajectory maps can found in the Figure 64 While we encourage the curious reader to have look at the Appendix F.9, we find that with increased batch size, the angle between the updates as well as the angle between the update and the current location become increasingly obtuse, and thus making room for wider directional exploration. In contrast, for smaller batch sizes these angles are closer to 90%. We hypothesize that a similar mutual interaction, observed with weight decay and momentum, also occurs with batch size is considered. detailed analysis, however, remains outside the current scope.
Lastly, also experimented with Sharpness- Minimization (Foret et al., 2021) (SAM), where we found that a higher value of the SAM regularization coefficient leads to a slightly increased directional similarity, which could potentially be related to SAM directing optimisation flatter basins wherein the individual points are more directionally alike and have higher cosine similarities. The detailed results can be found in the Appendix F.3.
Other Settings and Datasets. As a final remark for this section, we would like to emphasize that similar results for weight decay as well as momentum, can be found under different hyperparameter settings in the supplementary material. In particular, hallmarks for multiple values of learning rate, weight decay, and momentum for VGG on CIFARIO as well other values for momentum and weight decay in the case of ResNet50 trained with SGD, and even Vision Transformer trained with Adam on ImageNet across varying weight decay, but these have to be omitted here due space constraints.
analyze the qualitative and quantitative
F.1
VIT: WEIGHT DECAY, ADAMW
A.0.01
0.0011
30
30-
Clocc kpine Lodox
40
Checkpoint index
Figure 24: Trajectory Maps of ViT models across different amounts of weight decay
0.01
0001
36
Chedipoin Endex
Figure 25: Relative Trajectory Maps, with respect to initialization, of ViT models across different amounts of weight decay
27

---

Published as a conference paper at ICLR 2025
Rogh
2014
Spacis
Rpodo 10
(a) /00+1 0e, 0c)
(b) / (0.+1 02, Or 0o)
Widts
0.31
Z(0,+ O2, 02 kc_ for l:
(d) ((, 1o. OT 00)
Weugl
Wicgl DNC0
15
Spark
(0) <(1++1 P₁, 0r 0u)
(f) Apex Angle at Initialization L(,,
0.. 0 0a)
10m0 3a3k et OREB: 100 03
C300)
(g) Apex Angle at Origin L(0,, 8o)
Figure 26: Angular measures of the Trajectory for VIT trained on the ImageNet dataset
28

---

Published as a conference paper at ICLR 2025
Wide
(a) ||0:112
(b) 10.4
0c112
(c)
10.
00|2
Figure 27: Norm-based measures of the Trajectory for ViT trained on the ImageNet dataset
Train
Teat Actunacy over Epoclas
00003
Tet AccuINES
Drigle beog
0300
3,003
0
20
Stege
Tpochs
(a) Train Loss
(b) Test Accuracy
Figure 28: Loss and accuracy values for Weight Decay experiments. The final test accuracy values 75.81, 75.36, 73.26. in the order listed in the figure legend. The figures do contain the trends for both 0.01 and 0.0001, but they often lie on top of each other, so zooming in might be needed to see them separately.
Differunt Woight Denav values (WT):
Different Weight Decay values (VIT)
Spearman:
Spearman: 1.000
75.5
Test Accurscy
Toet Accurecy
75.0
73.0
76.0
74.0
0.535
0.300
5.855
Du300
2,835
03860
0.813
0.15
0.520
0.855
0.230
6,555
0.240
Meen Direclionelily Similarily
(relative) Meen Directionelily Similarily
Correlation of MDS
(b) Correlation of Relative MDS
Figure 29: Correlation plots of MDS and relative MDS with test performance.
29

---

Published as a conference paper at ICLR 2025
F.2 RESNET50: WEIGHT DECAY, SGD
0.001
0. 0001
30
30
Cleckpiit Index
30
50
60
60
90
Chockpoant indes
Figure 30: Trajectory Maps of ResNet50 models across different amounts of weight decay
A 0.001
- 0001
39.
20-
Chickptn:: mibex
Checkpoon Irdes
Figure 31: Relative Trajectory Maps, with respect initialization, of ResNet50 models across different amounts of weight decay
30

---

Published as a conference paper at ICLR 2025
Work
Rpodo 10
(a) ((0++ 0i, 0c)
(b) /(0++1 0c, Or 0o)
Weigh
Wisth
0.01
2001
Pp
(c) L(0 O1, O2 Os 02-k) k. for 1:
(d) L(,, 0o. Or 0o)
Eoar
0
Witti Dour
Sparh
(0) <(1++1 P₁, 0r 0u)
(f) Apex Angle at Initialization L(e,
0.. 0 0o)
13mh baak ORUB 00
(g) Apex Angle at Origin L(0,, 8o)
Figure 32: Angular measures of the Trajectory for ResNet50 trained on ImageNet
31

---

Published as a conference paper at ICLR 2025
100
Kooght
(a) |10:112
(b) 10e+x
0:l12
(c) |0, 001
Figure 33: Norm-based measures of the Trajectory for ResNet50 trained on ImageNet
Truin Lesa our Staps
Actunacy Staps
Test Accuracy over Epochs
Thin LOON
Tor. AouuEsc
Doigl
Mikde Domn
000000
20000
102001
Stags
S0egss
Apoette
(a) Train Loss
(b) Train Accuracy
(c) Test Accuracy
Figure 34: Loss and accuracy values for Weight Decay experiments. The final test accuracy values 66.42, 75.96, 70.86, in the order listed in the figure legend.
Different Wright Decay valums (SGD)
Diffecent Weight values (SGD)
Pearson: 0.23
Pearzon: 0.59 Spearzann:
Spearman:
74
Tost Accor Accorary
Test Accuracy
70
70
65
0.70
0.75
0.50
0.85
0.50
2:85
0.93
0.58
6:55
0.900
0.50
0.96
0.50
Directionality Similarity
(relative) Directionality Similarity
(a) Correlation of MDS
(b) Correlation of Relative MDS
Figure 35: Correlation plots of MDS and relative MDS with test performance.
32

---

Published as a conference paper at ICLR 2025
F.3 RESNET50: SHARPNESS AWARE MINIMIZATION ANALYSIS
Figure 36: Trajectory Maps of ResNet50 models across different values of SAM regularization coefficient
Figure 37: Relative Trajectory Maps, with respect initialization, of ResNer50 models across different values of SAM regularization coefficient
33

---

Published as a conference paper at ICLR 2025
147
101
Specaa
Rpodo 10
(a) ((0++1 0,, 0c)
(b) (0.+1
02, Or 0o)
25% -A 1189
Pp
(c) 2(0 Or, 02 -2-k)) k. for 1:
(d) (0, 0o. OT 0o)
(0) <(1++1 P₁, Or 0u)
(f) Apex Angle at Initialization L(e,
0.. 0 0d)
10hc Saga ORE 00
SULE
(g) Apex Angle at Origin L(ee, 8o)
Figure 38: Angular measures of the Trajectory for ResNet50 trained on ImageNet
34

---

Published as a conference paper at ICLR 2025
125
(a) |10:12
10
0c112
(c) 10
0011
Figure 39: Norm-based measures of the Trajectory for ResNet50 trained ImageNet
Trais Loss over Staps
Accuracy
Toal AccurICS
Strps
Tpochs
(a) Train Loss
(b) Train Accuracy
(c) Test Accuracy
Figure 40: Loss and accuracy values for SAM experiments.
The final test accuracy values for
= 0.5, 0.1, 0.05, are 72.72, 76.5, 76.3, 75.96 respectively.
Shorpnnss Awars Mininisation strengtha
Sharpness Aware Minimisation aEwngtth
Pearson:
Speartan: -020
Test Accuracy
Test Accuracy
70
03700 0.75
0.750
0.800
0,903
Directionalily Similarity
Directionality Similarity
(a) Correlation of MDS
(b) Correlation of Relative MDS
Figure 41: Correlation plots of MDS and relative MDS with test performance.
35

---

Published as a conference paper at ICLR 2025
F.4 RESNET50: MOMENTUM ANALYSIS, LR 0.1, WD 0.0001
J 0.
0.6
20
Cleclppit todes
20
40
50
60
10-
60
60
Chockapont indis
Figure 42: Trajectory Maps of ResNet50 models across different amounts of momentum
0.9
0.6
Checkppint Lndox
Chockpintt indes
Figure 43: Relative Trajectory Maps, with respect initialization, of ResNet50 models across different amounts of momentum
36

---

Published as a conference paper at ICLR 2025
143
000
Roode
(a) <(0++1 Beg 0c)
(b) <(0 Bc, 0o)
Spoda
(c) <(e++ Oc, 0. O2 kc- for d:
(d) < Do. OT 000
(c) Z(1 O,, Or 0o)
( Apex Angle Initialization 2 (0c 0o)
0,
(g) Apex Angle at Origin L(e,, Co)
Figure 44: Angular measures of the Trajectory for ResNet50 trained on ImageNet
37

---

Published as a conference paper at ICLR 2025
(a) |10:112
10.+.
0.112
(c)
|0
00|2
Figure 45: Norm-based measures of the Trajectory for ResNet50 trained ImageNet
over Staps
Train Accunaes Stepa
Toa. Aucuraay
Than
100000
(a) Train Loss
(b) Train Accuracy
(c) Test Accuracy
Figure 46: Loss and accuracy values for Weight Decay experiments. The final test accuracy values are 75.96, 74.98, 72.63, in the order listed in the figure legend.
Diffeemet Momentam valuns
Different Mommmon volros
70.0
Pearsmn--
Spearmatn: +1.00
Spear
0.50
75.5
Tbst Ancuracy
75.0
Test Accuricy
75.0
74.0
74.0
713
72.5
725
3.55
6.78
0.30
0.82
0.86
0.00
0.00
0.00
0.02
0.963
0.546
c.0s8
0.800
Directionality Similarity
(relative) Directionality Similarity
(a) Correlation of MDS
(b) Correlation of Relative MDS
Figure 47: Correlation plots of MDS and relative MDS with test performance.
38

---

Published as a conference paper at ICLR 2025
F.5 VGG: MOMENTUM ANALYSIS, LR 0.1, WD 0.0001
p 09
- 0.5
P-0
Chelsphit tedes
143
133-
Chockpoat indee
Figure 48: Trajectory Maps of VGG16 models different amounts of momentum
0.9
0.5
30
Checkpiint tndex
40-
148-
03-
Figure 49: Relative Trajectory Maps, with respect initialization, of VGG16 models across different amounts of momentum
39

---

Published as a conference paper at ICLR 2025
130
100
(a) <(Ec+1 De, 0c)
(b) <(0++1 Oe, 0o)
Spoda
Rpedko 00
(c) 2(2 Or A: kc_k), for l
(d) ((). 0,, Or .b
Rp
Z(0c+1 O,, OT 0o)
( Apex Angle Initialization 2 L(, 0o)
0o.
3400 30-0 02022
(g) Apex Angle at Origin L(,,, Go)
Figure 50: Angular measures of the Trajectory for VGG16 models trained on CIFAR10.
F.6 VGG: MOMENTUM ANALYSIS, LR 0.1, WD
40

---

Published as a conference paper at ICLR 2025
Zpnda
(a) |10:|12
(b) [0c+s 02112
(c)
10
0o 2
Figure 51: Norm-based measures of the Trajectory for VGG16 models trained on CIFARIO.
0
p=03
y=
20-
Clodlpon bokes
169-
Chackpoot indes
Figure 52: Trajectory Maps of VGG16 models across different amounts of momentum
20-
Chickpent mades
00-
Checkpoon Indes
Figure 53: Relative Trajectory Maps, with respect to initialization, of VGG16 models across different amounts of momentum
41

---

Published as a conference paper at ICLR 2025
Te
103
143
Spacial
Rpedo 10
(a) (00+1 Oi, 0c)
(b) / (0.+1
02, Or 0o)
<(e++ k 0e 0
k) 8.-k)
for l
(d) / 0,, 0p)
Epecke 60
2goda 251
(c) ((0++1 0r, Or 0u)
(f) Apex Angle at Initialization L(0, 0o)
0o.
2012 A:40 Oregs
25.0 122 30
Apex Angle at Origin (0c, 0o)
Figure 54: Angular measures the Trajectory for VGG16 models trained on CIFAR10.
F.7 VGG: MOMENTUM ANALYSIS, LR 0.01, WD 0.0001
42

---

Published as a conference paper at ICLR 2025
(a) ||0:|22
(b) 10c+s
Orl12
(c) 10:
00l2
Figure 55: Norm- based measures of the Trajectory for VGG16 models trained on CIFARIO.
0.9
0.5
30-
Chedlpmm babox
160-
130-
180
Checkpiin indix
Figure 56: Trajectory Maps of VGG16 models across different amounts of momentum
20-
Chickpent mades
22
Checkpont Indes
Figure 57: Relative Trajectory Maps, with respect to initialization, of VGG16 models across different amounts of momentum
43

---

Published as a conference paper at ICLR 2025
6.A1
08. 1.
1
Opocia
Rpeda 10
(a) (00+1 0i, 0c)
(b) / (0.+1 02, Or 0o)
Z(0++ Or, O2 Os 02-k) for 1:
(d) ((, 1o. Or 0o)
32
120
1.2
(0) L(E 6₁, Or 0u)
(f) Apex Angle at Initialization L(e,
0.. 0 0o)
1000 013b 0R00
(g) Apex Angle at Origin L(Oe, 0o)
Figure 58: Angular measures of the Trajectory for VGG16 models trained on CIFARIO.
F.8 VGG: MOMENTUM ANALYSIS, LR 0.01, WD 0
44

---

Published as a conference paper at ICLR 2025
Poeth:1 30
(a) |0:||2
(b) 10.4x 0.112
(c) 10. 0ol2
Figure 59: Norm-based measures of the Trajectory for VGG16 models trained on CIFARIO.
0.9
0
20-
Checkpint tidex
135-
Figure 60: Trajectory Maps VGG16 models across different amounts of momentum
--
p 0.5
Cleckpsitt Indee
10
Chockpoitt indea
Figure 61: Relative Trajectory Maps, with respect to initialization, of VGG16 models across different amounts of momentum
45

---

Published as a conference paper at ICLR 2025
19h
Epecta
Rpedo
(a) ((0+1 Oi, 0c)
(b) (0.+1 02, 0r 0o)
Ppatn
L(0 O1, 02 O2 02.k) k). for 1:
(d) ((, 1o. Or 0o)
120
(0) L(E 6₁, Or 0u)
(f) Apex Angle at Initialization L(e,
0.. 0 0o)
(g) Apex Angle at Origin L(Oe, 0o)
Figure 62: Angular measures of the Trajectory for VGG16 models trained on CIFARIO.
F.9 VGGI6 BATCH SIZE ANALYSIS
46

---

Published as a conference paper at ICLR 2025
Zonda 20
Spoda
Zonka 20
(a) |00:|2
(b) 10.4x
0.112
(c) 10. 0ol2
Figure 63: Norm-based measures of the Trajectory for VGG16 models trained on CIFAR10.
Figure 64: Trajectory Maps VGG16 models across different batch sizes. The learning rates have been scaled in proportion the batch size, and the training schedule was adjusted to sure an equal number of steps (and not simply epochs) for all the runs. We also adjusted the learning schedule drop learning rates corresponding number of steps across the experi- ments. The respective MDS values are = 753, 0.723, 0.660, 0.619 and the test accuracies are 91.63%, 91. 82%, 92. 92,44% 92.39%.
8-123
B-225
0--112
8- 3234
Figure 65: Relative Trajectory Maps, with respect to initialization, of VGG16 models across different
batch sizes
47

---

Published as a conference paper at ICLR 2025
103
100
(a) <(0++1 Beg 0c)
(b) <(ee+1 Pc, 0a)
Datch
1360
Spoda 20
(c) ((e++ Oe, 0. O2 )), for d:
(d) <(0, Do. OT 000)
Dec
12
(c) L(1 O., Or 0a)
Apex Angle Initialization L(0e 0, 0o)
Ager Saah CREA
18.00
Ca
(g) Apex Angle at Origin L(e,, 0o)
Figure 66: Angular measures of the Trajectory for VGG16 models trained on CIFARIO.
F.10 TRAJECTORY MAPS IN THE PRESENCE OF LABEL NOISE
We observe that with increasing label noise, the network is required to undergo more directional explo- ration to find a solution that can interpolate the training set. The MDS scores decrease monotonically with increasing label noise.
48

---

Published as a conference paper at ICLR 2025
(a) |10:112
(b) 10c4A 0:l12
(c) 10.
00l
Figure 67: Norm-based measures of the Trajectory for VGG16 models trained on CIFARIO.
080
000
420
59
"35
(a) Label Noise 0.4
(b) Label Noise 0.7
(c) Label Noise 1.0
Figure 68: Trajectory maps when CNN is trained on CIFARIO with different amounts of label noise, i.c. what fraction of samples have been assigned random labels.
49

---

Published as a conference paper at ICLR 2025
G GPT-NEOX TRAJECTORY ANALYSIS
14M
70M
100M
410M
10
86
226
70C
83K-
Cherkpannt index
122
342K
Oziino Staderyy
20
1324
1.2K
140%
1.48
2,88
6.9
126
66.
238-
395
545
1082
1298
206
12000
100
Cbeckwoint
Figure 69: Trajectory Maps of Pythia GPT-NeoX models across two orders of model scales trained on Pile. The corresponding MDS values are w 0.650, 0.672, 0.678, 0.726, 0. 759, 0.786, 0.818, 0.815.
3.0
6K-
22K
36K
0.8
100K
122K
Ginbwrtyy
Chockpeint
6,98
12B
23K
122K
223
1272
22
L4X
22K
100K
122K
228
7/
p0K
100%
1224
Checkpoint
Figure 70: Relative Trajectory Maps, with respect to initialization, of Pythia GPT-NeoX models across two orders of model scales.
500
SC1
(a) ||0=||a
(b) 10
0,11
(c) |0,
0o
Figure 71: Nose-based measures of the Trajectory for GPT-NeoX trained on the Pile dataset
50

---

Published as a conference paper at ICLR 2025
12
Spocia
Rpedo 10
(a) /(0++1 01, 0c)
(b) / 201+1 0c, Or 0o)
2N -A--- 1280
Seak
Reeku
(c) 2(0,+ Or, 02 Os 0e_ k). for 1:
(d) L(0, 1o OT 0o)
1.31
30
(0) <(0 O₁, Or 0u)
(f) Apex Angle at Initialization L(e,
0.. 0 0o)
11244
(g) Apex Angle at Origin L(ee, 8o)
Figure 72: Angular measures of the Trajectory for GPT-NeoX trained the Pile dataset
51

---

Published as a conference paper at ICLR 2025
H
LAYERWISE-TR.JECTORY MAPS
(a) query-key-value, bias
(b) query query-key value weight
(c) dense, bias
Dardgsenbo boo
(d) dense, weight
(c) dense- 4h-to-h, bias
(I) 4h-to-4 weight
Figure 73: Layerwise Trajectory Maps, grouped by layer type, for the 14M GPT-NeoX model trained on the Pile dataset.
52

---

Published as a conference paper at ICLR 2025
(a) dense-h-to-4h, bias
(b) dense-h-to-4h, weight
(c) embed-in. in, weight (d) embed-out, weight (e)
final-layer norm,
layer---moo-- (1)
final- Linl-layer-norm
bias
weight
input-layernorm, bias
(h) input-layernorm, weight
post-altention-lap layyttionnnayymoomm,. bias
ii) post-altention-laerrnon weight
Figure 74: Layerwise Trajectory Maps, grouped by layer type, for the 14M GPT-NeoX model trained on the Pile dataset.
53

---

Published as a conference paper at ICLR 2025
H.1 Q.K,V TRAJECTORY MAPS ACROSS SCALES
Coeat SRRAOAAAA
(a) 14 M
leanulie
Egll
gio
kynuulige
COoDA SELANT
(b) 160 M
Invin.igge
Tvees Llxcs
krees 15
lyves
Iement gir
CHEE DEAACO
Chacqpot BOO
1.4
(d) 12
Figure 75: Trajectory maps of Q.K.V layers become homogenized over increasing scale.
54

---

Published as a conference paper at ICLR 2025
1 TRAJECTORY MAPS FOR GROKKING
In grokking (Power et al., 2022), we have that the performance on test samples significantly lags behind the training performance. Below, we look the trajectory maps in this setting, considering the experimental setup of https: : wwwph/.....//eeyyoooee//o/ooooo.
We can observe
204
784
264
08
0.8
01
101
0.
511
0.
84
1001
0.6
066
04
104
0.5
1404
05
118
1364
044
044
1364
1804
(a) Upto about 1500 steps
(b) Upto about 2000 steps
374
510
100
775
0.
03
04
04
6161
(c) Upto about 4000 steps
(d) Upto the end of training.
Figure 76: Trajectory maps during the course of learning. Grokking (Power et al.. 2022), or sudden increase in test accuracy while training accuracy is already at a ceiling, occurs where the trajectory map also shows transition point.
in Figure 76 that:
Upto about 1500 steps: Everything pitch blue. No directional exploration, test accuracy remains, more less, random.
Upto about 2000 steps: Some directional movement starts to happen, and some initial signs of improvement test performance.
Upto about 4000 steps: Transition point for directional exploration. Test performance visibly improves.
We think that without (appropriate) directional exploration. the training converges to a //tp
like solutions. Moreover, we believe that being 'lazy' in the direc- tional sense is highly intertwined with being 'lazy' in the sense of feature learning (Chizat et al., 2020). Besides, the above experiments show that the resemblance with the lazy regime is more than an analogy. Kumar et al. (2024) have shown that grokking can be seen as the transition from the lazy the non-lazy (rich) training regime. In particular, the test accuracy first shows a marked growth is also the part where the directional exploration starts happen.
find that the precise part of the training, where
55

---

Published as a conference paper at ICLR 2025
J PUTTING DIRECTIONAL REDUNDANCY TO TEST
Trajectory Map: ResNet20, CIFAR10
Training BN vs Readout
BatchNorm only
Toxt Lager
20
20
88-
Checkpoint Index
Cosine Similarity
0.7
641
0.6
100
M48
100
120
0.4-
140
03-
Chockpoint Index
Test accuracy
Figure 77: Comparison of training only batch norm parameters with training entire last layer (readout) parameters.
The presented results, and that in the main section, have been averaged over 3 seeds. The standard deviation is never more than 0.05 (for the batchnorm training, even less), and hence it is difficult to make it out in the plots and has been omitted.
J.1 COMPARISON WITH (DISCRETIZED) TRAJECTORY MAP
Training just Scalars
BatchNem only
Badiln
0.3
0.5
Casine Similarity
0.11
100
Full-Network
0.49
120
Accuracy
140
28
80
100
120
140
Test accuracy
Epochs
Figure 78: Trajectory Map and the corresponding decapacitation test via tuning only batch-norm parameters. Here we discretize the trajectory map to 10 color levels to facilitate comparison.
56

---

Published as a conference paper at ICLR 2025
K EFFECT OF DIFFERENT LEARNING RATE SCHEDULES
In order to ensure that our broader trajectory map results are not sensitive the particular choice stepwise learning rate schedule, decay, and polynomial decay with power (2). We run this comparison both without and with an initial warmup for epochs to the base learning rate of 0.1. Besides, facilitate easier comparison, we discretize the plots to 10 color levels. These resulting trajectory maps are illustrated below in Figures 80, 81, and the learning schedules plus the train/test loss depicted in the Figure 83.
study for three other learning rate schedules: cosine, linear
Leaening Rone Sched, les
111
1740
20803
2003
(a) Learning Rate Schedules
(b) Train Loss
(c) Test Loss
Figure 79; Visualization of different learning rate schedules and the accompanying train and test loss curves (in the non-warmup case of Figure 80).
K.1 TRAJECTORY MAPS
Without Warmup
polynomial
stepwise
10
Epoch Index
20-
20
20-
-38
Similancy
Epoch Index
Figure 80: Comparison of trajectory maps for ResNet50 trained ImageNet across different learning rate schedules.
K.1.1
WITH WARMUP FOR 5 EPOCHS
cosine
linear
polynomial !2)
stepwise
Epoch Index
Cosine
80
20
80
25
Epoch Index
Figure 81: Comparison of trajectory maps for ResNer50 trained on ImageNet across different learning rate schedules.
Importantly, find that the larger nature of the trajectory map is highly similar across the different schedules, despite minor variations. This demonstrates that our claims about directional similarity robust to the choice of learning rate schedules.
57

---

Published as a conference paper at ICLR 2025
K.2 ANGLE BETWEEN SUCCESSIVE EPOCHS
We also investigate the effect of different learning rate schedules on the angles formed between successive epochs, and, in particular, how it varies in the presence absence of momentum and weight decay. Said differently, this is analogous to the setting of Figure 4(a), but where we swap out stepwise learning rate schedule for cosine, linear, and polynomial (degree schedules. These results are located in the Figure 82.
angle botweon succospiee epochs (t dogrees)
Angle bed www.eeeennn suscensiie eppoctt deggrengen
100
munb lambda-le- 1e-4 mumb. lambdee1ee- 2e-4 mum0.a, lambdaamo mumd, lambdaam
60-
lambdi-lad
mu-0 0.90.. lambda-O
30-
50-
mumd, lambda-0
Epochs
(a) Cosine
(b) Linear
Angle between surcessive degrees)
90
mumo Sambida=
mumo, lambda-Ie-i Iambidaui
musi, lambowind
20
Epochs
(c) Polynomial (degree 2)
Figure 82: Effect of different learning rate schedules on the angle between successive epochs. We find that as in Figure 4(a), significant part of the training shows obtuse angles between the updates (as marked by the part above the dashed gray lines), and which tend be higher in the setting of having both momentum and weight decay, like seen previously.
58

---

Published as a conference paper at ICLR 2025
L EFFECT OF TRAJECTORY GRANULARITY
L.1 TRAJECTORY MAPS WITH FINER GRANULARITY
We run our prototypical ResNet50 experiments 2x and 4x granularities, each of which amounts to having 2 and 4 evenly spaced checkpoints for every epoch. This results in an overall (including initialization) 181 and 361 checkpoints, as opposed to 91 checkpoints considered previously. The below-mentioned comparisons show that our results at the granularity of 1x produce highly similar trends as seen with the finer granularity.
The above holds both for the visualizations of trajectory maps as well as angular hallmarks such as angle between successive checkpoints.
1x granularity
2x granularity
4x granularity
13
10-
10-
0.8
20
20
Cosine Similarity
30-
30-
0.6
40
10
30
50
70
80
90
10
20
30
40
50
60
70 80
90
10
20
30
40
70
90
Epoch Index
(a) Trajectory Maps with color levels
granularity
2x granularity
granularity
0.75
50
50-
50
0.49
60
50
0.30
70
70
70
00
80
90
30
s0
70
90
0.00
Epoch Index
(b) Trajectory Maps with 20 color levels
Figure 83: Visualization of different learning rate schedules and the accompanying train and test loss curves (in the non-wame non-warmup case of Figure 80).
I.2 ANGLE BETWEEN SUCCESSIVE EPOCHS
59

---

Published as a conference paper at ICLR 2025
Angle: betweeen successfve checkpoints
Comparison across granularities
Granufertty
100
80-
60-
20
40
50
80
Epochs
Figure 84: Direct comparison of building trajectory maps that are (4x) finer granularity than previously used. Hyperparameter settings are identical between the two, i.c., momentum is 0.9 and weight decay is 0.0001.
Angle between puccessvv chocipoms dogrocs
110
100
Angle bettweeca stccessdivee spoctaan
180
muel tambeta=1esc
musd. bembdaaieed
muin a. lemiddsad
murd, lambdend
26
Training Epochs (f)
Eoochs
(a) granularity
(b) granularity
Figure 85; Effects of Momentum and Weight Decay at different granularities. The right figure is the same as the one in the main text. 1x granularity refers to a checkpoint every epoch; while 4x granularity would mean checkpoint every quarter of an epoch.
60

---

Published as a conference paper at ICLR 2025
L.3
FINE-GRAINE VIEW INTO THE FIRST EPOCH
100
500
0.6
betwoon (dagredd
40-
1800
0.94
2700
0.2
0.0
0.88
4500
0.08
900
30000
2000
2002
4200
5000
Sheme
(a) Trajectory Map of the first Epoch
(b) Angle between each of the adjacent steps
Figure 86: Trajectory map and Angle between checkpoints take at fine intervals during the first epoch. The first epoch contains 5004 steps, and for the left subfigure, we consider checkpoints after every 18 steps which suffices here since still finer granularity will make the already high resolution only marginally more high resolution. Hence the trajectory map comprises of total of 279 checkpoints (including the initialization). As far as the right subfigure is concerned, here we process every single step yield detailed view of the steps in the first epoch. The particular setting shown are for our prototypical setting with momentum 0.9 and weight decay 0.0001.
M ANGLE BETWEEN SUCCESSIVE EPOCHS FOR VERY LARGE BATCH SIZE TRAINING
Angle weteeee surrassivve sporhs degloss)
110
Angle hersoven mocornttie [In
mued.s, lambda- 1e-4 mum lamisdamlad
60
mu-0.s lambda-D lamoda
50
20
10
80
80
Training Epochs (f)
(a) Batch Size: 16384
Batch Size: 256
Figure 87: Angle between successive epochs for networks trained with different batch sizes.
61


Reviews:
[{"review_id": "fqmCXUzOlM", "reviewer": "Reviewer_idmu", "summary": "The paper presents several tools for visualizing, quantifying, and comparing neural network training trajectories. The optimization paths of neural networks are represented by a Trajectory Map (TM), which consists of the cosine similarities between parameters in checkpoint pairs of a neural network over training time. The angle between adjacent samples of the trajectory is also evaluated. Theoretically, this angle is found to be bounded by the Hessian eigenvalues so that in most training settings it is either obtuse (negative cosine) or orthogonal (0 cosine). Unlike parameter distances, the angular similarity of parameters with their initializations are also found to reduce as models get larger - a result which is used to predict relative performance between models. Finally, the \"decapitation hypothesis\" is introduced with supporting empirical evidence, which hypothesizes that the majority of parameters can be frozen without impacting performance, once a network's training trajectory becomes fixed in cosine similarity relative to the initialization.", "questions": "On a related note to the aforementioned weakness, neural networks are known to be much less stable during the early training period. What does the TM and angles between adjacent samples look like when sampled at a higher frequency in the first few epochs (e.g. every iteration of epoch 1)?\n\nHow much of the observations related to phases of training (figure 2 left, figure 4) are due to discontinuities in the learning rate schedule? For instance, what if a linear decay or cosine annealing schedule is used instead?\n\nThere is a similarity between the TM and notions of representational similarity which also consider distances and angles among the cosine similarities of activations (e.g. [1] Lange et al. 2023) . Could the authors discuss if either the methods of representational similarity could be applied to TM, or if there is an empirical or theoretical connection between a network's trajectory in parameter space vs representation space?\n\n[1] Lange, R. D., Kwok, D., Matelsky, J. K., Wang, X., Rolnick, D., & Kording, K. (2023, September). Deep Networks as Paths on the Manifold of Neural Representations. In Topological, Algebraic and Geometric Learning Workshops 2023 (pp. 102-133). PMLR.", "limitations": "", "rating": 10, "confidence": 2, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "Originality: the paper neatly ties together several previous works related to phases and trajectories of optimization. Although some of the findings confirm results from other work (e.g. larger networks are closer to initialization), other findings are novel and surprising (e.g. momentum and weight decay increasing rather than decreasing variability in the optimization paths).\n\nQuality: the paper is overall very thorough and addresses almost all of the pertinent concerns that a reader may raise (see Weaknesses for issues). Although the decapitation hypothesis is not very well elaborated on, this is fine since the paper already covers a large amount of ground.\n\nClarity: the paper is very clearly laid out and neatly presented, with helpful color choices to enhance the figures and accompanying text. The writing has a succinct, engaging style, and is enjoyable to read. Having said that, there is some excess complexity in some sentence structures (e.g. contribution 1) and vocabulary choices (e.g. \"epochly\") which are difficult to understand.\n\nSignificance: the paper gives a nice disambiguation between distance travelled in parameter space during training, versus the straightness of the trajectory. The proposed measure of directionality in section 5 seems to be a useful metric for model capacity and training progress. Finally, the decapitation hypothesis does open up a lot of possible methodological work in the future, and potentially a new perspective on existing empirical methods for reducing training costs (e.g. LoRA).", "weaknesses": "Appendix C is important in that it looks at the baseline where the trajectory is a random walk, since naturally one would expect trajectories to become more similar over time due to learning rate decay and having travelled further from the origin. In general, it would be more informative if *all similarity values, including in the main text* were reported as increases/decreases over the random walk baseline (with decaying step size), and plots showed such relative differences instead of absolute similarities. Specifically:\n- It is not obvious from figures 9-10 how much added similarity/dissimilarity is in training trajectories versus random walks - for instance, is the random walk with decay more dissimilar than training trajectories at all points in time, or do they invert in order of similarity at some point?\n- What is the non-relative TM for random walks (to compare with figure 2)?\n- Similiarly, what are the expected MDSs of random walks for a given step size schedule?\n- Are real (empirically determined) step sizes used to derive/simulate the random walk TMs in figure 9? This seems to be the fairest method, and thus it would also be necessary to consider all combinations of momentum/weight decay and their effect on step size.\n\nThe per-epoch sampling has an unclear interaction with the analyses of adjacent steps (i.e. figure 4), because the choice of how often to sample the network could change the measured results. For example, it could be that no momentum/no weight decay actually has larger angles when measured between successive iterations, but the sum of such steps over the epochs results in smaller angles. For the sake of thoroughness, it would be nice to know how results such as figure 4a depend on the sampling rate.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PqJu4N6aWg", "reviewer": "Reviewer_2KNQ", "summary": "This paper introduces a novel approach to understanding neural network training trajectories by examining their directionality, particularly through the concept of a \"TM: Trajectory Map.\" This approach sheds light on the consistency and detailed behaviors of optimization paths throughout training. The authors propose that once directional movement saturates, the model’s capacity can be effectively reduced (\"decapacitation\"), potentially improving computational efficiency. The study presents evidence of directional redundancy within optimization trajectories across both image classification tasks and large language models.", "questions": "- Details on Accuracy and Loss: Could specific values for train loss, test loss, and accuracy be provided for each experiment?\n- Correlation Between Mean Directional Similarity (MDS) and Performance: What is the observed correlation between MDS and test accuracy or loss? Detailing this would clarify the effectiveness of MDS as a performance predictor.\n- Learning Rate Schedule Diversity: In the appendix, including a comparison of Trajectory Maps under different learning rate schedules (e.g., constant, cosine decay, step decay) would be beneficial, as the results suggest that Trajectory Maps may vary depending on the schedule used.\n\n#### Comments\n- Missing Figure References: Figures referenced on lines L1090, L1120, and L1126 are absent.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- Novelty: The Trajectory Map provides a fresh perspective on neural network training by visualizing and quantifying directional movement, potentially contributing to the development of more efficient training techniques.\n- Empirical Validation: Experiments span multiple architectures and hyperparameter configurations, supporting the proposed approach.\nApplicability to Efficiency: The concept of \"decapacitation,\" where computation resources can be reduced once directionality saturates, is promising for the design of more resource-efficient training strategies.", "weaknesses": "- Dependency on Specific Configurations: The initial results for ResNet50 appear heavily dependent on a specific learning rate decay schedule. It remains unclear whether author's findings can be replicated across different learning rate schedules (e.g., constant, cosine decay), suggesting additional experiments would be valuable to verify generalizability.\n- Correlation Between MDS and Performance: The proposed Mean Directional Similarity (MDS) metric lacks clear quantitative correlation with test accuracy or loss, which weakens the evidence of its effectiveness. Further analysis to clarify this relationship would strengthen the contribution.\n- See questions section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bfnIfIErxg", "reviewer": "Reviewer_eAxJ", "summary": "This paper flattens the neural network parameters into a vector, then analyzes the evolution of that vector over the course of training. They show across a few architectures and domains that there is large directional stability, affected by hyperparameter choices. They also note that such stability can be leveraged to finetune fewer parameters, in keeping with prior literature.", "questions": "Please see Weaknesses.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The choice of quantity to analyze is quite simple, and it's nice to have a fleshed out analysis of such a quantity\n- It was a good idea to include the analysis of an additional domain (Pythia) so as to make the arguments firmer beyond the overparametrized setting (can get training accuracy to 0) that happens in image classification", "weaknesses": "## Overall\n\n1. I think the motivation for why to study this quantity in the introduction is a bit sparse. As someone who works in the field I think this is an interesting quantity to look at, but I think a reader from elsewhere may not feel similarly and the paper does not make that easier. For a work on a similar topic that I think makes a better case, see https://arxiv.org/abs/2106.16004\n2. We as a field obviously don't have a complete understanding of NN optimization, so I think it's a bit unfair to judge this work by saying \"it's not clear why this analysis connects to things we care about\" as it might be useful in the future, or for correlating with other observations, still I think the paper could make a better attempt to connect to many existing works on the topic to further the scientific dialogue. I'll try to detail this below.\n3. I think the experiments could use a few more hyperparameter settings, it's a bit difficult to make claims about the importance or role of hyperparameters like momentum or weight decay with only two settings (and no sense of how they connect to performance, see next point).\n4. In a paper ostensibly introducing itself about the loss landscape, it was a bit frustrating not to find a single loss curve. If the network fails to fit the loss, then that discussion should occur in concert with increased directional stability. I think it's difficult to understand these things in isolation and I wish the paper made it easier to do so.\n\n## Specific\n\n### Section 1\n1. Implicit bias should be defined in the introduction, at least in a short sentence.\n2. The introduction to the motivating question feels very fragmented.\n3. \"Benefits of the Trajectory View...\" - I think it would be helpful to explicitly contrast this with prior metrics others have considered to make the case stronger.\n4. \"Decapacitation Hypothesis\" - I think this is an odd name, it would imply to me that networks lose capacity, but the actual results later in the paper suggest that networks have sufficiently learned the important features such that you don't need every parameter any more, not that you never needed every parameter. Also I believe that without a discussion of prior work on this topic, giving this a new name is not good hygiene. See https://arxiv.org/abs/2312.11420\n\n### Section 2\n1. I think the motivation for why to flatten the parameters is not very clear. Especially as prior work like muP has explicitly noticed different evolution for different parameters (e.g. bias vs. matrix) given the structure of the network, it would be nice to make an explicit argument contrasting with existing work why this is a good idea. It would also be useful to discuss linear interpolation works like https://arxiv.org/abs/2106.16004 and many works cited in that paper to bolster the case.\n\n### Section 3\n1. \"epochly\" -> intermediate?\n2. I feel that convergence in direction should probably be considered as convergence given that CE loss can never be minimized because of the exponential.\n3. Training loss curves for Fig 2 + 3 are missing: is directional similarity mostly a result of the model failing to learn well at all?\n4. Directional movement saturates exactly when learning rate is decayed, so it'd be nice to have a hyperparameter setting without this additional confounding factor...\n5. It might be nice to rescale the trajectory maps to be on separate color scales. For example in the case with no weight decay, the increased similarity could be due to the fact that WD removes superfluous weights, but without a lot of the initialization sticks around that isn't used much for the actual prediction task. This effect could drown out any signal in the trajectory map.\n\n### Section 4\n1. The \"Initial Thoughts\" are confounded by the lack of training loss to correlate with the plots. It could be that the model is stuck at a local minimum.\n2. Certainly in Fig. 4 the updates are oscillating, but this might be expected for (relatively) small batch sizes as the examples are likely very different. It would be interesting to know how the overall gradient oscillates.\n3. In 4.2 it's very unclear to me why this is the right model to consider for NN as so much is simplified here. Certainly the stable Hessian assumption seems unfounded in the early stage of training given the trajectory differences shown in this work, without a measurement that it's stable... It's fine to try to simplify to basics but I think the presentation should be amended.\n4. Also as far as I understand, EoS is for gradient descent, while here we look at practical SGD, this seems like a huge difference no?\n5. Would be good to discuss related work in this section on linear interpolation, see previous links.\n\n### Section 5\n1. \"Directional measures predict...\" feels really too strong a claim for the existing evidence. I would want to see performance for many different hyperparameter choices vs. the apex angle. Besides there is already work on practical generalization measures (https://arxiv.org/abs/1912.02178) and it would be nice to talk about how the stronger ones there connect.\n\n### Section 6\n1. There is already work on tuning just normalization layers in the finetuning setting (https://arxiv.org/abs/2312.11420) that should be referenced and discussed here. I also don't love the naming of this phenomenon as I feel it could be stated simply as what you did.\n2. It would be nice to have experiments on Pythia here as well.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pU02dxk0Aa", "reviewer": "Reviewer_BhS2", "summary": "This paper introduces the concept of the trajectory map to study the dynamics of neural network training. Specifically, the trajectory map is a matrix where each element in $i$-th row and $j$-th column represents the cosine similarity between the model parameters at $i$-th iteration and the model parameters at $j$-th iteration.\nThrough visualization and quantitative analysis, the study uncovers patterns such as directional redundancy as reflected in the trajectory map. Additionally, the empirical influence of momentum and weight decay on the trajectory map is verified. Finally, the paper suggests utilizing directional redundancy to enhance the efficiency of neural network training.", "questions": "(1) What are the shared and outlined observations between Figure 2 and Figure 6? Despite the influence introduced by the learning rate schedule in Figure 6, for points along the diagonal, the cosine similarity exhibits a monotonically decreasing trend in both the horizontal and vertical directions, generally declining from 1 to 0 as observed from the upper-left point of the diagonal. While this observation is straightforward but introduces nothing new.\n\n(2) Line383 to line388 and corresponding Figure 7. \n\n(2.A) I believe that the observed trends are linked to variations in model size rather than performance across different metrics.\n(2.B) However, I am unclear on the significance of this comparison beyond the linear relationship between the apex angle and model size. Could you elaborate further on the insights provided by Figure 7.a in this regard?\nFurthermore, Figures 7.b and 7.c \"fail to capture the expected monotonic trend,\" but the implications of this \"failure\" are unclear, other than that these metrics cannot be used as reliable predictors for model size.\n\n(3) The experiments with large language models (LLMs) in Section 5 utilize Adam as the optimizer instead of SGD. Could you provide further insights on how the choice of different optimizers, such as Adam versus SGD, influences the training process and outcomes, particularly in relation to the dynamics of adaptive learning?\n\n(4) Section 5 key takeaways (line398 - line402). \n\n(4.A) What does the complexity measure refer to in this context? Is it related to the complexity of the neural loss landscapes, the architecture of the model, or something else? Could you clarify how this measure is utilized to assess the optimization process?\n(4.B) While I agree that the complexity metric reveals a correlation with model size, claims regarding other correlations seem unsupported or underexplored in the current analysis.", "limitations": "", "rating": 3, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "(1) This work provides a relatively thorough validation of the proposed trajectory map concept, specifically for analyzing the training trajectory. The concept of the trajectory map is new and transforms extremely high-dimensional information into a more manageable and trackable form, making it suitable for visualization and enabling the meaningful application of statistical tools for further insights.\n\n(2) The identified observations and corresponding patterns may offer valuable insights into the dynamics of neural network training, potentially revealing useful information about the optimization process and the network’s behavior during different phases of training.", "weaknesses": "(1) The introduction of customized properties in this work leads to findings relevant within those tailored domains. This work introduces new aspects of neural network training but does not build strong connections with the existing understanding of the optimization of neural networks. Thus, the new definitions and results in this work are less useful in motivating the development of new optimization methods or helping understand current optimization methods.\n\n(2) The primary finding of directional redundancy is questionable. First, projecting high-dimensional information into lower-dimensional space may result in significant information loss, meaning that redundancy observed in low-dimensional representations may not necessarily imply that iterations during the later stages of training are redundant or unnecessary. Thus, the leveraging of directional redundancy (in section 6) lacks concrete theoretical support and sufficient assessment.\n\n(3) Section 6 offers a relatively weak argument for the practical application of directional redundancy, specifically the claim that \"once directional saturation is achieved, the full capacity of a network might no longer be required for further training.\" While the image on the right side of Figure 8 presents straightforward patterns, these patterns do not clearly align with the left side of the figure, and they seem more contingent on the percentage of iterations", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)