Approaching Deep Learning through the Spectral Dynamics of Weights

## Abstract

Abstract We propose an empirical approach centered on the spectral dynamics of weights—the behavior of singular values and vectors during optimization—to unify and clarify several phenomena in deep learning. We identify a consistent bias in optimization across various experiments, from small-scale “grokking” to large-scale tasks like image classification with ConvNets, image generation with UNets, speech recognition with LSTMs, and language modeling with Transformers. We also demonstrate that weight decay enhances this bias beyond its role as a norm regularizer, even in practical systems. Moreover, we show that these spectral dynamics distinguish memorizing networks from generalizing ones, offering a novel perspective on this longstanding conundrum. Additionally, we leverage spectral dynamics to explore the emergence of well-performing sparse subnetworks (lottery tickets) and the structure of the loss surface through linear mode connectivity. Our findings suggest that spectral dynamics provide a coherent framework to better understand the behavior of neural networks across diverse settings.

## 1 Introduction

Interest in neural networks has exploded in the past decade. Capabilities are rapidly improving, and deployment is ever-increasing. Yet, although issues with these technologies now have social repercussions (Bender et al., ; Bommasani et al., ), many fundamental questions regarding their behavior remain unanswered.

Figure: (a) Schematic
Refer to caption: x1.png

For instance, despite extensive research, we still lack a complete understanding of the implicit biases (Neyshabur et al., ) of neural networks trained via stochastic optimization. Even basic questions regarding the role of regularization like weight decay (Hanson & Pratt, ; Krogh & Hertz, ; Zhang et al., ) have only partial answers (Van Laarhoven, ; Andriushchenko et al., ; Yaras et al., ). Perhaps most vexing, we lack a complete explanation for how neural networks generalize, despite having the capacity to perfectly memorize the training data (Zhang et al., ). Such an explanation may allow us to design better algorithms, however a lack of understanding makes the deployment of neural networks vulnerable to uninterpretable errors across fields (Szegedy et al., ; Ilyas et al., ; Hendrycks et al., ; Zou et al., ).

Although theoretical explanations have been put forward to make analysis tractable, these studies often rely on special settings like deep linear networks (Arora et al., , ) or infinite-width systems (Jacot et al., ), and arguments may rely on unsubstantiated or impractical assumptions like near zero initialization. On the empirical side, a growing body of work in interpretability has attempted to reverse-engineer neural networks (Rahaman et al., ; Barak et al., ; Nanda et al., ), but given the difficulty of the tasks, the systems of interest have been very small-scale, and the methodology for analysis quite bespoke and difficult to scale. A third category of work aims at understanding empirical behavior from a higher level (Zhang et al., ; Huh et al., ; Yu & Wu, ), but while these works often study larger-scale systems, they often focus on more abstract objects like the gram matrix (Huh et al., ) or NTK (Fort et al., ), and thus do not have the granularity and predictive power of the previous two categories.

To bridge these gaps, we propose a task-agnostic, unifying perspective of many disparate phenomena in deep learning across many different practical tasks and architectures, including image classification with ConvNets, image generation with UNets, speech recognition with LSTMs and language modeling with Transformers. Through extensive experiments, we examine the dynamics of singular values and singular vectors of weight matrices and the spectral dynamics of weights. We show that such dynamics underlie many distinct phenomena and appear intimately tied to generalization. We are motivated to study these dynamics specifically as optimization is the fundamental process driving deep learning (Nagarajan & Kolter, ; Zhang et al., ), the matrix parameters form the core identity of any neural network, and the singular value decomposition is a fundamental way to view any matrix. We detail our specific contributions in the following paragraphs.

As a test bed for understanding generalization, Power et al. () introduce the “grokking” phenomenon, where a small-scale model initially minimizes the training loss but performs poorly on validation data, then with much more training suddenly minimizes the validation loss. In particular, Nanda et al. () showed that in simple modular arithmetic tasks, the specific solution learned by optimization could be reverse-engineered from the weight matrices. Although this description is precise, in Section 3, we notice a task-agnostic view of grokking, observing that the drop in validation loss during grokking coincides with the simultaneous discovery of low-rank solutions across all weight matrices in the network. The connection between rank and generalization might intuitively be through Occam’s Razor. Furthermore, echoing existing works (Lyu et al., ; Liu et al., ), we find that weight decay clearly affects grokking and rank minimization: neither behavior occurs as strongly without it. Having said that, we demonstrate that increasing the training data can partially compensate for the absence of weight decay in terms of generalization and rank minimization, and in all cases, we see a correlation between low-rank matrices and generalization. Thus, we find that examining spectral dynamics provides a task-agnostic view of grokking.

Though this suggests a connection between rank and generalization, grokking is typically studied on synthetic tasks, very small-scale models like single-layer Transformers or small MLPs, and requires very particular hyperparameter settings (Power et al., ; Nanda et al., ; Gromov, ; Kumar et al., ). If our perspective is to be useful, it needs to scale to larger systems. Thus, we turn to common empirical tasks drawn from the literature like image classification, image generation, speech recognition and language modeling as well as varied and larger networks like VGG (Simonyan & Zisserman, ), UNet (Ronneberger et al., ), LSTM (Hochreiter & Schmidhuber, ) and Transformers (Vaswani et al., ).

In Section 4, we demonstrate that the spectral dynamics are biased toward effective rank minimization across various practical neural networks in complex settings. Although this behavior echoes theoretical predictions in the deep linear setting, we find that the behavior of networks disagrees with a common theoretical assumption about low-rank dynamics: alignment of singular vectors in consecutive layers (Saxe et al., ; Arora et al., , ; Milanesi et al., ). Thus, the rank minimization mechanism may differ from what the theory describes. It is notable too that our hyperparameter settings are drawn from existing literature, thus the trend toward rank minimization coincides with well-generalizing networks across settings.

Still, in these more practical settings, we do not see such an obvious rank minimization as in grokking, nor is there a sudden transition from memorization to generalization. One particularly notable ingredient for grokking was a very high level of weight decay. Weight decay has a long history as a regularizer explicitly penalizing parameter norm, which can be used for norm-based generalization bounds (Bartlett, ), but these bounds do not seem to explain the success of practical systems (Nagarajan & Kolter, ; Jiang et al., ). Recently, others have suggested an implicit effect of weight decay on parameter rank in theoretical or small-scale empirical studies (Galanti et al., ; Timor et al., ) which may be connected to generalization (Razin & Cohen, ). As such, we explore this effect of weight decay in practical settings.

In Section 5, we empirically connect rank minimization to weight decay, showing that weight decay promotes rank minimization across architectures and tasks. In addition, in some cases it also appears to promote singular vector alignment in consecutive weights despite the nonlinearities between layers, which indicates further compression of the model. Although weight decay explicitly penalizes norm, studying spectral dynamics allows us to observe the implicit effect on rank. Such an effect can help in understanding generalization as norm regularization is insufficient.

Given the suggestive connection between rank and generalization, we turn to the classic memorization experiments of Zhang et al. (). Zhang et al. () demonstrated that even small networks can memorize random labels. Hence, any arguments about generalization need to take into account the structure of the data and the optimization process. In Section 6, we show that training with random labels leads to high-rank solutions, while rank with true labels is much lower. We also find that while random labels do not align consecutive layers, true labels do, which is surprising given the non-linearities between layers. Through spectral dynamics, we see a clear difference between the optimization of generalizing and memorizing networks, which provides a foothold in the climb toward better theoretical understanding.

Our results suggest that viewing neural networks through the lens of spectral dynamics can shed light on several generalization-related phenomena, but we suspect there are broader connections. In the literature, many curious and unexplained phenomena regarding neural networks exist. We take two as case studies. First, the lottery ticket hypothesis (LTH) (Frankle & Carbin, ), which has found the existence of sparse sub-networks with similar performance. Such a phenomenon provides evidence that, despite ever-increasing parameter counts and energy costs, efficient smaller networks already exist. Understanding the source of such efficiency may help us alleviate deployment costs. Second, linear mode connectivity (LMC) (Nagarajan & Kolter, ; Frankle et al., ; Neyshabur et al., ), which finds that models sharing a portion of the optimization trajectory can be averaged together in weight-space to yield a stronger model (Wortsman et al., ; Ramesh et al., ). This phenomenon indicates that, after some training, the loss surface is quite convex in a subspace, even though the optimization problem is theoretically extremely nonconvex. As any finetuning from pre-trained models stays in this convex space (Neyshabur et al., ; Li et al., ; Sadrtdinov et al., ), an explanation for what underlies model-averaging would help to clarify the role of pretraining, would shed light on the commonly-used low-rank adaptation (Hu et al., ), and could lead to better optimization.

In Section 7, we find that global magnitude pruning, a standard procedure for finding lottery tickets, preserves top singular vectors and acts like a low-rank pruning. We also see that the ability to interpolate between models in LMC strongly correlates with sharing top singular vectors. With these results, we note that the two phenomena can be seen as aspects of the spectral dynamics of weights and bring them under the umbrella of prior sections.

To summarize the discussion above, by studying the spectral dynamics of weights, we find:

- •
Grokking is intimately linked to rank minimization;
- •
Rank minimization is a general phenomenon in more complex tasks;
- •
Weight decay acts implicitly as a low-rank regularizer;
- •
Generalizing solutions have a lower rank than memorizing ones; and
- •
Top singular vectors are preserved when performing magnitude pruning while linearly interpolating between connected modes.

All of these phenomena and effects have previously been studied in isolation to varying degrees, but by approaching deep learning through spectral dynamics, we aim at a common language for neural networks. Code for all experiments is released at [https://github.com/dyunis/spectral_dynamics](https://github.com/dyunis/spectral_dynamics).

## 2 Related Work

### 2.1 Grokking

Power et al. () first noticed a surprising phenomenon they called “grokking” where models quickly fit the training data on toy tasks, then after a long period of training, very quickly generalize on the validation data. Later, others found that this phenomenon can occur in a relaxed fashion (Thilak et al., ) on very simple models and different datasets (Liu et al., ; Gromov, ; Kumar et al., ; Xu et al., ) and that weight decay seems critical to cause it (Lyu et al., ; Liu et al., ; Tan & Huang, ). Some posit a transition from the kernel (Jacot et al., ) to rich (Atanasov et al., ) regime explains grokking (Kumar et al., ; Mohamadi et al., ). There is also empirical evidence for a connection between double descent and grokking (Davies et al., ) the discovery of a sparse solution (Merrill et al., ), the simplification of decision boundaries (Humayun et al., ) and the leading indicator of loss oscillation (Thilak et al., ; Notsawo Jr et al., ). None of these works have explicitly examined the connection with rank, which we do in Section 3, and provides a common framework through which to view many of these results.

### 2.2 Singular Value Dynamics

Prior work on deep linear networks (Arora et al., ; Milanesi et al., ) suggests that rank minimization may describe implicit regularization in deep matrix factorization better than simple matrix norms. See (Arora et al., ) (Appendix A) for a detailed argument. However, a critical assumption used in these works is “balanced initialization.” This means that for consecutive matrices $W_{i}$ and $W_{i+1}$ in the product matrix $\prod_{j}W_{j}$, we have $W^{\top}_{i+1}W_{i+1}=W_{i}W_{i}^{\top}$ at initialization. Decomposing these matrices with SVDs and leveraging orthogonality, this simplifies to $V_{i+1}\Sigma_{i+1}^{2}V_{i+1}^{\top}=U_{i}\Sigma_{i}^{2}U_{i}^{\top}$ where $U_{i}$ and $V_{i+1}$ are orthogonal matrices. Since these are orthogonal decompositions of the same matrix, their diagonals must be equivalent, allowing for the permutation of elements with the same value. This leads to $U_{i}=V_{i+1}O$ up to signs, where $O$ is a block diagonal permutation matrix that may permute the rows of equivalent diagonal elements. Notably, if all diagonal elements are distinct and $U_{i}$ and $V_{i+1}$ are square matrices, then $U_{i}=V_{i+1}$ up to signs.

Under the balanced initialization assumption, all product matrices will be aligned. Consequently, the product of the diagonals will evolve in a closed-form manner, with larger singular values growing faster than smaller ones. As shown by (Arora et al., ), this translates to rank-minimizing behavior with increasing depth in the matrix products. This formula is also empirically validated for linear matrix factorization problems. Similar results have been derived for tensor products and other structured settings (Saxe et al., ; Yaras et al., ). More generally, (Ji & Telgarsky, ) show that for deep linear networks with infinite training alignment between layers will happen. In Section 4, we explore how these conclusions and assumptions hold for much larger, practical neural networks that are far from linear.

### 2.3 Low-Rank Properties

Another line of research focuses on more general low-rank biases. Early work explored norms as an implicit bias (Gunasekar et al., ). Theoretical analyses reveal that norms or closed-form functions of weights might be insufficient to explain implicit regularization, but they do not necessarily contradict the possibility of rank minimization (Razin & Cohen, ; Vardi & Shamir, ). Numerous studies investigate low-rank biases in various matrices, including the Jacobian (Pennington et al., ), weight matrices (Le & Jegelka, ; Martin & Mahoney, , ; Frei et al., ; Ongie & Willett, ), Gram matrix (Huh et al., ), and features (Yu & Wu, ; Feng et al., ). Additionally, research suggests that dynamics influence the decay of rank (Li et al., ; Chen et al., ; Wang & Jacot, ). Some works establish connections between weight decay and rank minimization in idealized settings (Ziyin et al., ; Galanti et al., ; Zangrando et al., ; Ergen & Pilanci, ; Parhi & Nowak, ; Shenouda et al., ). We are particularly interested in how far these connections extend in practice. In Section 5, we present empirical evidence that sometimes disagrees with, but also expands, arguments from theory and small-scale systems to much larger ones.

## 3 Grokking and Rank Minimization

Motivated by theoretical work that proposes connections between rank and generalization (Razin & Cohen, ) weight decay and rank (Galanti et al., ; Timor et al., ; Yaras et al., ; Zangrando et al., ), and the importance of weight decay for grokking (Power et al., ; Lyu et al., ; Liu et al., ) in simple settings, we evaluate the potential connection between rank and grokking in neural networks. Examining grokking through the lens of rank (and more generally spectral dynamics) offers a complementary perspective on grokking with other descriptions such as fourier decomposition (Nanda et al., ), the simplification of linear decision boundaries (Humayun et al., ), the connection to double descent (Davies et al., ), and the discovery of a sparse solution (Merrill et al., ).

Figure: (a) Error
Refer to caption: extracted/5805818/figs/grokking/weight_decay_0.0-error.png

We largely follow the setting of Nanda et al. (), optimizing a single-layer Transformer for modular addition (details in Appendix A). Inspired by work in the deep linear case (Saxe et al., ; Arora et al., ; Milanesi et al., ; Yaras et al., ), we track the evolution of singular values for individual weight matrices. To gain a high-level overview of all parameter evolutions, we compute the (normalized) effective rank of a matrix $W$ (Roy & Vetterli, ) with rank $R$ as

$$ $\displaystyle\text{EffRank}(W):=-\sum_{i=1}^{R}\frac{\sigma_{i}}{\sum_{j} \sigma_{j}}\log\frac{\sigma_{i}}{\sum_{j}\sigma_{j}}\enspace,$ (1) $\displaystyle\text{NormEffRank}(W):=\frac{\text{EffRank}(W)}{R}\enspace,$ (2) $$

where $\sigma_{i}$’s are the singular values of matrix $W$ and $\text{EffRank}(W)$ is the entropy of the normalized singular value distribution. As the probability mass concentrates, the effective rank decreases. We plot $\text{NormEffRank}(W)$ to compare across layers and time.

In addition, inspired by the assumptions of balancedness made by prior work (Arora et al., , ), we examine the alignment of consecutive weight matrices in the Transformer. To examine and quantify this alignment between consecutive matrices in a network at training time $t$, i.e.,

$$ $\displaystyle W_{i}$ $\displaystyle=\sum_{j=1}^{R}\sigma_{j}(t)u_{j}(t)v_{j}(t)^{\top}\enspace,$ $\displaystyle W_{i+1}$ $\displaystyle=\sum_{k=1}^{R}\sigma^{\prime}_{k}(t)u^{\prime}_{k}(t){v^{\prime} _{k}}(t)^{\top}\enspace,$ $$

we compute,

$$ $A(t)_{jk}=\lvert\langle u_{j}(t),v^{\prime}_{k}(t)\rangle\rvert,$ (3) $$

where the absolute value is taken to ignore sign flips in the SVD computation. We then plot the diagonal of this matrix $A(t)_{ii}~{}\forall~{}i\leq 100$ over time. For exact details on how alignment is computed for different architectures and layers that are more complex than the fully connected case, please see Appendix A.

In Figure 2, we see a tight alignment: the sudden drop in validation loss coincides precisely with the onset of low-rank behavior in the singular values. Examining inter-layer alignment during training, we observe that the final low-rank solution gradually emerges from the model’s middle ranks. Conversely, the grokking phenomenon is absent without weight decay, and no low-rank solution seems to develop. We also replicate the setting of Thilak et al. () who show a form of grokking without weight decay, but when plotted on a linear-scale x-axis the generalization appears much less sudden than the setting with weight decay. Additionally, when using 90% of the data and no weight decay, generalization still coincides with effective rank minimization. The familiar reader will note that Nanda et al. () previously showed that the particular solution found in modular addition is a low rank fourier decomposition, so our observations on low rank weights will directly follow. While such a description for modular addition is impressively precise, it may be difficult to obtain for more complex tasks. In following sections we argue that rank minimization is a perspective that can apply in more complex settings when one does not know what to look for in the weights, and it may even be possible to eventually interpret the neural network via the top ranks (Praggastis et al., ).

We also want to highlight a complication between norm and generalization that our results indicate, in dialogue with Liu et al. (). One can see that without weight decay but with 90% of the data, the generalizing solution that is discovered has many singular values larger than 1, and the maximum singular value is also larger than that of the grokked solution, for which most singular values are near zero. Recall the norm of matrix $W$ is given by $\lVert W\rVert_{2}=\sqrt{\sum_{i}\sigma_{i}^{2}}$ where $\sigma_{i}$ are the singular values. Thus, the large-data solution has a much higher norm than the grokked solution. Still, both settings generalize with a certain low-rank behavior. The confounding factor is that, with high weight decay, the smaller singular values disappear, while without, they do not. We currently lack a precise understanding of how low-rank behavior directly relates to generalization, and it might not always hold true – high-rank solutions might also generalize. Nonetheless, from Occam’s Razor’s perspective, a low-rank solution is conceptually “simpler” due to its lower dimensionality, making for a compelling explanation.

## 4 Spectral Dynamics Across Tasks

Inspired by the results on grokking and prior work on deep linear networks, which studies the evolution of the SVD of the weight matrices (Saxe et al., ; Arora et al., , ; Milanesi et al., ; Yaras et al., ), we apply the same analysis to larger, more practical systems. We show that the trends we saw in the analysis of grokking mostly hold true across networks and tasks at a much larger scale, even though our findings do occasionally deviate from theoretical predictions.

### 4.1 Methodology

Our experiments aim to examine reasonably sized neural networks across a variety of tasks. We select models and tasks that are representative of current applications. Specifically, we focus on:

- •
Image classification with CNNs (VGG-16 (Simonyan & Zisserman, )) on CIFAR10 (Krizhevsky, );
- •
Image generation through diffusion with UNets (Ronneberger et al., ) on MNIST (LeCun, );
- •
Speech recognition with LSTMs (Hochreiter & Schmidhuber, ) on LibriSpeech (Panayotov et al., ); and
- •
Language modeling with Transformers (Vaswani et al., ) on Wikitext-103 (Merity et al., ).

Training hundreds of runs for each of the above experiments is computationally expensive, limiting the scale of models we can explore. We primarily adopt hyperparameters from existing literature, with minor modifications for simplicity. This ensures that any correlations observed are likely a reflection of common practices, not introduced bias on our part. We hope that the broad scope of these experiments will allow for a more general perspective on neural network optimization.

The primary evidence in this section comes from computing the SVDs of weight matrices within the models. Consequently, we disregard 1D bias and normalization parameters in our analysis. However, previous research suggests that in some cases these parameters are not crucial for performance (Zhang et al., ; Mohan et al., ; Karras et al., ). Due to the large number of matrices in these models, we present plots of individual layers’ matrix parameters and statistics summarizing behavior across layers for conciseness of presentation. Hundreds of thousands of plots were generated for this study, making it impossible to include them all. Full experimental details, including hyperparameters, are available in Appendix A, where we take hyperparameters from existing settings in the literature.

### 4.2 Effective Rank Minimization

Building on theoretical (Saxe et al., ; Arora et al., ; Milanesi et al., ; Boix-Adserà et al., ; Yaras et al., ) and empirical (Dittmer et al., ; Martin & Mahoney, , ; Boix-Adserà et al., ) findings, we investigate effective rank minimization across parameters in larger models and on a more diverse variety of tasks. Figure 3 reveals a consistent trend: the effective rank of network parameters generally decreases throughout training, regardless of the specific parameter or network architecture. This suggests a progressive “simplification” of the network as training progresses, and echoes our previous findings in the high-data regime on modular addition.

Figure: (a) VGG
Refer to caption: extracted/5805818/figs/vgg/lr_0.1-sv-model.layers.11.conv.weight_sv.png

We further conduct a singular-value pruning experiment to explore the relationship between low-rank behavior and model performance. We prune either the top or bottom half of the singular values for each weight matrix in the network and then evaluate the pruned model at each training step. Intuitively, we expect the top singular values to capture the most critical information for the network’s function. Figure 4 confirms this hypothesis, demonstrating that the pruned parameters, without further training, can closely approximate the full model’s performance. This may not have been the case. In particular, simultaneously pruning lower components across all layers may lead to losing some critical signal that must be passed between layers, or the large number of small magnitude singular values may provide some important noising effect, thus empirical confirmation is necessary. Another way to argue this is that, though each individual matrix is well-approximated in euclidean space by its top ranks, it is unclear the total effect on the neural network function from pruning these matrices simultaneously. We will rely in later sections on this observation that large singular values are more important to the function of the network.

Figure: (a) VGG
Refer to caption: extracted/5805818/figs/vgg/lr_0.1-train-loss.png

### 4.3 Alignment of Singular Vectors Between Layers

Similar to the analysis of grokking, we investigate the alignment between consecutive layers in the larger neural networks considered in this section. We not only employ the alignment matrix defined in Eqn. 3 but also derive and plot a scalar measure for alignment based on the top diagonal entries:

$$ $a(t)=\frac{1}{10}\sum_{i=1}^{10}A(t)_{ii}$ (4) $$

For specific details on calculating this measure in diverse architectures and complex layers (beyond fully connected layers), please refer to Appendix A.

Figure: (a) VGG
Refer to caption: extracted/5805818/figs/vgg/lr_0.1-align-model.layers.11.conv.weight_model.layers.12.conv.weight_align-diag.png

Figure 5 reveals a key finding: the theoretical assumption of balanced initialization, which posits aligned singular value decompositions (SVDs) between weight matrices (Arora et al., ; Saxe et al., ), does not hold true at the start of training in these larger networks. Additionally, unlike the linear case discussed in Du et al. (), the alignment does not appear to remain static throughout training. However, a weak signal of alignment in the top ranks develops. This trend is somewhat reminiscent of the theoretical result provided by Mulayoff & Michaeli () for linear networks under the assumption of whitened input data. Still, the weakness of the observed signal means that existing theoretical models do not fully capture the complexities of real-world neural network training.

## 5 The Effect of Weight Decay

In light of the previously observed evolution of singular values, we investigate a proposed effect of weight decay. Though weight decay explicitly penalizes the norm of weights, there is evidence that complicates the connection between norm and generalization for neural networks (Razin & Cohen, ; Andriushchenko et al., ), meaning we do not have a full understanding as to why weight decay may be useful. Alternatively, some theoretical (Boix-Adserà et al., ; Razin & Cohen, ; Yaras et al., ; Timor et al., ; Ongie & Willett, ; Galanti et al., ; Zangrando et al., ) and empirical works (Galanti et al., ; Boix-Adserà et al., ) propose a connection with the rank of matrices in constrained settings. Still, a comprehensive connection to larger empirical networks has not yet been demonstrated.

Figure: (a) VGG
Refer to caption: extracted/5805818/figs/vgg/weight_decay_0.0-sv-model.layers.11.conv.weight_sv.png

We speculate on the intuition of the mechanism in more practical settings. Notice in its simplest form that weight decay asks for $\text{arg min}_{W}\,\mathcal{L}(W)+\lambda\lVert W\rVert_{F}^{2}$, where $\lVert W\rVert_{F}^{2}=\sum_{i=1}^{R}\sigma_{i}^{2}$ with singular values $\sigma_{i}$ of weight matrix $W$ with rank $R$. We saw that larger singular values of neural networks grow faster (Fig. 3 top row) and that the top singular vectors are much more useful for minimizing task loss than the bottom ones (Fig. 4). Thus, with minor weight decay regularization, one straightforward solution for the network may be to minimize the rank of a given weight matrix while preserving the top singular values to minimize $\mathcal{L}(W)$. Timor et al. () argue a similar effect as if all singular values are less than 1, the norm of activations will shrink with depth, so it will be impossible to pass signals with sufficiently deep networks. Thus it is important for a few singular values to be sufficiently large.

Figure 6 shows that adding weight decay produces this exact low-rank behavior, while too much weight decay leads to complete norm collapse. The exact choice of “too much” varies across architectures and tasks, though it may be due to the use of SGD for VGG (Simonyan & Zisserman, ) instead of AdamW (Loshchilov & Hutter, ) for the rest.

In addition, even more surprisingly, large amounts of weight decay promote a tighter alignment in the top singular vectors of consecutive layers of the Transformer, but not other networks. We see this in Figure 7. This behavior is quite reminiscent of the balancedness condition (Arora et al., , ; Du et al., ), though the Transformer considered here has nonlinearities and much more complex structures. It is curious that the trend is so strong for only this architecture. We also provide additional evidence in Appendix A where Figure 14 shows that the solutions with very high weight decay are still performant, even though they are much lower rank. Though it is difficult to argue as simple a trend as “lower rank equals better generalization” because one does not know the minimal rank necessary for a given task, it is notable that the role of weight decay for improving generalization is tied up with its function as a rank regularizer. In addition, although we lack precise tools to entirely interpret complex models, when there are only a few ranks per matrix it may become possible to extend analysis efforts (Nanda et al., ; Praggastis et al., ) to more complex domains.

Figure: (a) VGG
Refer to caption: extracted/5805818/figs/vgg/weight_decay_0.0-align-model.layers.11.conv.weight_model.layers.12.conv.weight_align-diag.png

Figure: (a) Train Err.
Refer to caption: extracted/5805818/figs/random_labels/random_labels_False-train-err.png

## 6 Spectral Dynamics with Random Labels

Given the observations connecting generalization and rank thus far, and the enlightening view on the implicit effects of weight decay, we are interested in seeing whether the perspective developed sheds any light on the classic random label memorization experiments of Zhang et al. ().

Similar to Zhang et al. (), we train a simple MLP to fit random or true labels on CIFAR10. Please see Appendix A for the details regarding the experimental setup. Zhang et al. () decay the learning rate to zero, and the random label experiments only converge late in training. Consequently, we use a constant learning rate to control this phenomenon. We see in Figure 8 that both cases are able to achieve zero error, though with different singular value evolution and alignment in the middle layer.

Surprisingly, we see that with true labels the inner layers are low rank, while with random labels they are much higher rank. This may be explained by the shared structure in the true classes of the dataset, which manifests in the parameters. Even more surprisingly, we find here that even without weight decay, inner layers align with true labels, while with random labels, this alignment occurs and then disappears with more training. This is particularly intriguing as there are non-linearities that could theoretically separate the network from the linear case, and yet strong alignment occurs despite that. Such alignment has not yet been leveraged by existing theory, and might provide structured assumptions for new understanding. In summary, these results suggest that viewing generalization through the lens of rank may be fruitful.

## 7 Beyond Generalization

We have seen over the course of many experiments that deep models are biased toward low rank, and that there is a tempting connection between rank minimization and generalization. Still, the lens of spectral dynamics can be applied more broadly. In the following subsections, we explore two phenomena: lottery tickets (Frankle & Carbin, ) and linear mode connectivity (Frankle et al., ). Beyond shedding further light on neural networks, these phenomena have implications for more efficient inference and storage, as well as understanding the importance of pretraining (Neyshabur et al., ). We find that lottery tickets are a sparse approximation of final-checkpoint top singular vectors. The ability to linearly interpolate between faraway checkpoints and improve performance coincides strongly with top singular vector sharing between checkpoints. Such observations may form a foundation for a better understanding compression and model averaging (Wortsman et al., ; Ilharco et al., ).

### 7.1 Top Singular Vectors Become Stable Earlier

Figure: (a) VGG
Refer to caption: extracted/5805818/figs/vgg/lr_0.1-sva-model.layers.11.conv.weight_sva-diag.png

Before we explore the phenomena, we first make another observation that will be helpful. As top singular values grow disproportionately large, it would be natural that top singular vectors become stable in direction as the gradients remain small. To demonstrate this, for a given matrix in the network $W_{i}(t)=\sum_{j=1}^{R}\sigma_{j}(t)u_{j}(t)v_{j}(t)^{\top}$ at training time t, we compute

$$ $S(t)_{jk}=\lvert\langle u_{j}(t)v_{j}(t)^{\top},u_{k}(T)v_{k}(T)^{\top}\rangle\rvert,$ (5) $$

where $T$ is the final step of training, and the absolute value is taken to ignore sign flips in the SVD computation. We then plot the diagonal of this matrix $S(t)_{ii}~{}\forall~{}i\leq 100$ over time. We also use a scalar measure of the diagonal to summarize like in the alignment case: $s(t)=\frac{1}{10}\sum_{i}S(t)_{ii}$. In Figure 9, we see that top singular vectors converge in direction earlier than bottom vectors.

### 7.2 Lottery Tickets Preserve Final Top Singular Vectors

As large singular vectors will become stable late in training, we wonder about the connection to magnitude pruning and the lottery ticket hypothesis. Frankle & Carbin () first showed evidence for the lottery ticket hypothesis, the idea that there exist sparse subnetworks of neural networks that can be trained to a comparable performance as the full network, where the sparse mask is computed from the largest magnitude weights of the network at the end of training. Frankle et al. () build further on this hypothesis and notice that, for larger networks, the masking cannot begin at initialization, but rather at some point early in training. Still, the mask must come from the end of training.

The reason for this particular choice of mask may be connected to the dynamics we previously observed. Specifically, at the end of training large singular values are disproportionately larger, so high-magnitude weights may correspond closely to weights in the top singular vectors at the end of training. If magnitude masks were computed at the beginning, the directions that would become the top singular vectors might be prematurely masked as they have not yet stabilized, which may prevent learning on the task.

Here we train an unmasked VGG-16 (Simonyan & Zisserman, ) on CIFAR10, then compute either a random mask, or a global magnitude mask from the end of training, and rewind to an early point (Frankle et al., ) to start sparse retraining. Please see Appendix A for details. In Figure 10, we plot the singular vector agreement (SVA, Eqn. 5) between the final model, masked and unmasked, where we see exactly that magnitude masks preserve the top singular vectors of parameters, and with increasing sparsity fewer directions are preserved. Even though prior work has remarked that it is possible to use low-rank approximations for neural networks (Yu et al., ), and others have explicitly optimized for low-rank lottery tickets (Wang et al., ; Schotthöfer et al., ), we rather are pointing out that the magnitude pruning procedure seems to recover a low-rank approximation.

Figure: (a) Loss
Refer to caption: extracted/5805818/figs/lottery_tickets/mask_type_magnitude-train-loss.png

We also compute the singular vector agreement (SVA) between the masked model trajectory and the original unmasked model trajectory (diagonal of Eqn. 5). We see in Figure 10 that there is no agreement between the bottom singular vectors at all, but there is still loose agreement in the top singular vectors. Thus, it seems the mask allows the dynamics of only the top singular vectors to remain similar, which we know are most important from the pruning analysis in Figure 4.

Preserving top singular vectors by pruning seems like a natural outcome of large matrices, so as a control, we follow exactly the same protocol except we generate the mask randomly with the same layerwise sparsity. We can see in Figure 10 that this results in much lower preservation of top singular vector dynamics, and also performs worse, as in (Frankle et al., ). It would not be surprising that random pruning is worse if simply evaluated at the end of training, but masking is applied quite early in training at epoch 4 of 164 long before convergence, so it’s striking that the network now fails to learn further even though it is far from convergence. We interpret this as evidence that the mask has somehow cut signal flow between layers, so it is now impossible for the network to learn further, while magnitude pruning and rewinding still allows signals to pass that eventually become important.

### 7.3 Spectral Dynamics and Linear Mode Connectivity

We come to the final phenomenon that we seek to describe: linear mode connectivity. Linear mode connectivity (LMC) is the property that one can interpolate linearly between two different minima in weight space and every parameter set along that path performs well, which gives the impression that the loss surface of neural networks is somehow convex despite its theoretical nonconvexity. This was first demonstrated in small networks with the same initialization (Nagarajan & Kolter, ), then expanded to larger networks and connected to lottery tickets (Frankle et al., ; Paul et al., ). Entezari et al. () first conjecture that two arbitrary minima show LMC up to permutation, and demonstrate it in simple models. This was expanded to wide models (Ainsworth et al., ; Jordan et al., ; Qu & Horvath, ), and can be proven in various ways (Kuditipudi et al., ; Brea et al., ; Simsek et al., ; Ferbach et al., ), but it does not hold for standard models (Qu & Horvath, ). LMC has also been exploited for model-averaging and performance gains (Wortsman et al., ; Ilharco et al., ; Rame et al., ). Still despite all of this work, we lack a description for why LMC occurs. In particular: why is there a convex, high dimensional (Yunis et al., ) basin that models find shortly in training (Frankle et al., ), or after pretraining (Neyshabur et al., ; Sadrtdinov et al., )? We do not answer this question in full, but find an interesting view through the singular vectors.

#### 7.3.1 Linear Mode Connectivity Correlates with Top Singular Vector Agreement

As we saw earlier directional convergence of top singular vectors in Figure 9, it suggests the dynamics of those components are more stable, so we might expect mode-connected solutions to share these components. To examine this, we plot agreement between the singular vectors of the weight matrices at either endpoint of branches:

$$ $\displaystyle W^{(1)}(T)$ $\displaystyle=\sum_{j}^{R}\sigma_{j}(T)u_{j}(T)v_{j}(T)^{\top}\enspace,$ $\displaystyle W^{(2)}(T)$ $\displaystyle=\sum_{k}^{R}\sigma^{\prime}_{k}(T)u^{\prime}_{k}(T){v^{\prime}_{ k}}(T)^{\top}\enspace,$ $$

spawned from the same initialization in training. If the branches are split from an initialization on a trunk trajectory $W(t)$, we call $t$ the split point or epoch. We visualize the diagonal of $\lvert\langle u_{j}(T)v_{j}(T)^{\top},u^{\prime}_{k}(T){v^{\prime}_{k}}(T)^{
\top}\rangle\rvert_{jk}$ vs. split epoch, where the absolute value is taken to ignore sign flips in SVD computation.

Figure: (a) VGG
Refer to caption: extracted/5805818/figs/vgg_lmc/lmc-train-loss_barrier.png

Figure: (a) VGG
Refer to caption: extracted/5805818/figs/vgg_lmc/lmc_cross-train-loss_barrier.png

To remind the reader, LMC only occurs after a small amount of training time has passed. Too early and the final models of each branch will show a bump, or barrier, in the loss surface along the linear interpolation (Frankle et al., ). To measure this precisely, we use the definition from Neyshabur et al. (), which is the maximum deviation from a linear interpolation in the loss, an empirical measure for convexity in this linear direction. When this deviation is 0, we consider the checkpoints to exhibit LMC. Please see Appendix A.9 for details on the calculation. Given evidence in Figure 4 that top components are the most important for prediction, and that top components become stable before training has finished, it is plausible that LMC is connected to the stability of top singular vectors in the later portion of training.

This would mean that checkpoints that do not exhibit the LMC property should not share top singular vectors, while checkpoints that do exhibit the LMC property should share top singular vectors. We see in Figure 11 that this is the case across models and tasks, where the alignment between endpoints is much stronger in top singular vectors. We also see no LMC and poor agreement in top components between branches that have initializations from different trunk trajectories, but with the same split epoch $t$ and the same branch data order in Figure 12. Thus, these top directions are not a unique property of the architecture and data, but rather are dependent on initialization. It is notable that concurrent work (Ito et al., ) arrives at a similar conclusion: permutation solvers between optima match top singular vectors. Though the conclusions are similar, their experiments are primarily conducted on smaller scale settings, and only for permutation matching at the end of training. Here we connect these observations to the optimization behavior of networks throughout training.

#### 7.3.2 Perturbing Breaks Linear Mode Connectivity and Singular Vector Agreement Simultaneously

To make the connection between top singular vectors and LMC even tighter, we intervene in the normal training process. If we add random perturbations to destabilize the components that will become the top components long before they have converged, and if singular vector agreement is tied to LMC, we would like to see that final models no longer exhibit the LMC property. Indeed this is the case. In Figure 13, when increasingly large random perturbations are applied, the barrier between final checkpoints increases and the LMC behavior disappears. Please see Appendix A for details. In addition, the previously-strong singular vector agreement disappears simultaneously. Thus it seems this agreement is tied to linear mode connectivity.

We speculate that, due to the results in Figure 4 that show the top half of the SVDs are much more critical for performance, if these components are shared then interpolating will not affect performance much. Rather, interpolation will eliminate the orthogonal bottom components which may only make a minor impact on performance. If however the top components are not shared, then interpolating between two models will remove these components, leading to poor performance in between. Such observations may help in explaining the utility of pretraining (Neyshabur et al., ), weight averaging (Rame et al., ; Wortsman et al., ; Ilharco et al., ) or the use of LoRA (Huh et al., ) to replace full finetuning.

Figure: (a) VGG
Refer to caption: extracted/5805818/figs/vgg_lmc/lmc_pert-train-loss_barrier.png

## 8 Discussion

We provide an empirical perspective to understand deep learning through the lens of SVD dynamics. We first note a tendency toward rank minimization on a small scale in grokking, then expand these findings to practical networks and tasks. In addition we find that weight decay, though it explicitly penalizes norm, implicitly promotes this low-rank bias. We also show through the developed analysis that generalization and memorization differ in the rank and alignment of solutions found by optimization.

We go beyond remarks on generalization and show that magnitude pruning for lottery tickets acts similarly to low-rank pruning, and linear mode connectivity coincides with the sharing of top singular vectors between checkpoints. Low rank models have more efficient inference, and finetuning from pretrained models always operates in the LMC regime (Neyshabur et al., ), thus this understanding can lead to more efficient inference and compression and possibly stronger optimization practices.

While a comprehensive theory for our results remains elusive, these observations can act as a platform for a deeper understanding of deep learning. Notably, the observed spectral dynamics appear consistent across diverse settings, even without restrictive assumptions like balanced initialization, linearity, or small weight scales. This suggests a common underlying mechanism.

On the empirical side, several interesting problems present themselves. Interpretability of neural networks is a growing area of research (Nanda et al., ), and there already exist early efforts to interpret singular vectors of convolutional weights (Praggastis et al., ). There may also be connections to other unexplained phenomena such as double descent (Belkin et al., ; Nakkiran et al., ; Davies et al., ) or adversarial examples (Szegedy et al., ; Ilyas et al., ; Hendrycks et al., ). The solutions to these problems may help us diagnose deployment risks in the wild, and ideally help to design better algorithms. We believe our results contribute another step along this path.

## Acknowledgements

DY would like to thank the following people for helpful discussion at many different stages: Wei Hu for a conversation on connecting LMC to rank dynamics in deep matrix factorization; Jonathan Frankle for very early support and periodic discussions on aiming to understand the source of LMC; Shubham Toshniwal and Shane Settle for support in following this line of work in the early stages; Mikhail Belkin, Tengyu Ma, and Rob Nowak for early conversations and encouragement on studying rank minimization in small-scale cases; Sudarshan Babu, Haochen Wang, Takuma Yoneda, and Xiao Zhang for discussions and enthusiasm on resuming this work when it had been paused; Sam Buchanan for similar enthusiasm on resuming the work and very helpful comments on presentation; Anmol Kabra for suggestions on additional experiments and visualizations to round out the work; and David McAllester for very helpful comments on presentation. DY would also like to thank Adam Bohlander for his work in maintaining the TTIC cluster, which made the many experiments in this paper possible. This material is based upon work supported by the National Science Foundation Graduate Research Fellowship Program under Grant No. 1754881. KKP was supported through the NSF TRIPOD Institute on Data, Economics, Algorithms and Learning (IDEAL) and other awards from DARPA and NSF.

## Appendix A Experimental Details

For all experiments, we use 3 random seeds and average all plots over those 3. This is relatively small, but error bars tend to be very tight, and due to the high volume of runs required for this work we lack the resources to run much more.

In order to compute alignment we consider only pairs of layers that directly feed into each other, and ignore the influence of residual connections so as to cut down on the number of comparisons. Specifics on individual architectures are given below.

### A.1 Image Classification with VGG

We train a VGG-16 (Simonyan & Zisserman, ) on CIFAR-10 (Krizhevsky, ) for 164 epochs, following hyperparameters and learning rate schedule in (Frankle et al., ), but without data augmentation. For the optimizer we use SGD with batch size 128, initial learning rate 0.1 and momentum of 0.9. We also decay the learning rate 3 times by a factor of 10 at epoch 82, epoch 120, and finally at epoch 160. We also use a minor amount of weight decay with coefficient 0.0001.

VGG-16 uses ReLU activations and batch normalization (Ioffe & Szegedy, ), and includes both convolutional and linear layers. For linear layers we simply compute the SVD of the weight matrix. For convolutional layers, the parameters are typically stored as a 4D tensor of shape $(c_{\text{out}},c_{\text{in}},h,w)$ for the output channels, input channels, height and width of the filters respectively. As the filters compute a transformation from each position and input channel to an output channel, we compute the SVD of the flattened tensor $(c_{\text{out}},c_{\text{in}}\cdot h\cdot w)$, which maps all inputs to outputs, similar to Praggastis et al. (). This is not the SVD of the entire transformation of the feature map to the next feature map, but rather the transformation from a set of adjacent positions to a particular position in the next layer. For the individual SV evolution plot, we use the 12th convolutional layer.

In order to compute alignment of bases between consecutive convolutional layers, $V_{i+1}^{\top}U_{i}$ we need to match the dimensionality between $U_{i}$ and $V_{i+1}$. For convolutional layers we are presented with a question as to how to handle the spatial dimensions $h$ and $w$ as naively the input dimension of the next layer will be a factor of $h\cdot w$ larger dimension. We experimented with multiple cases, including aligning at each spatial position individually or averaging over the alignment at all spatial positions, and eventually settled at aligning the output of one layer to the center spatial input of the next layer. That is, for a 3x3 convolution mapping to a following 3x3 convolution, we compute the alignment only for position (1,1) of the next layer. This seemed reasonable to us as on average the edges of the filters showed poorer alignment overall. For the individual alignment plot, we use the alignment between the 11th and 12th convolutional layers at the center spatial position of the 12th convolutional layer.

### A.2 Image Generation with UNets

We train a UNet (Ronneberger et al., ) diffusion model (Sohl-Dickstein et al., ; Ho et al., ) on MNIST (LeCun, ) generation. We take model design and hyperparameters from (Wang & Vastola, ). In particular we use a 4-layer residual UNet and train with AdamW (Loshchilov & Hutter, ) with batch size 128, and learning rate of 0.0003 for 100 epochs. This model uses swish (Ramachandran et al., ) activations and a combination of linear and convolutional, as well as transposed convolutional layers.

Computing SVDs and alignment is similar to the image classification case described above, except in the case of the transposed convolutions where an extra transpose of dimensions is needed as parameters are stored with the shape $(c_{\text{in}},c_{\text{out}},h,w)$. For the individual SV evolution plot, we use the 3rd convolutional layer. For the alignment plot, we use the alignment between the 3rd and 4th convolutional layers at the center spatial position of the 4th convolutional layer.

### A.3 Speech Recognition with LSTMs

We train a bidirectional LSTM (Hochreiter & Schmidhuber, ) for automatic speech recognition on LibriSpeech (Panayotov et al., ). We tune for a simple and well-performing hyperparameter setting. We use AdamW (Loshchilov & Hutter, ) with batch size 32, learning rate 0.0003 and weight decay 0.1 for 50 epochs. We also use a cosine annealing learning rate schedule from 1 to 0 over the entire 50 epochs.

The LSTM only has matrix parameters and biases, so it is straightforward to compute SVDs of the matrices. For individual SV evolution plots, we plot the 3rd layer input parameter. In the case of alignment, we make a number of connections: first down depth for the input parameters, then connecting the previous input parameter to the current hidden parameter in both directions, then connecting the previous hidden parameter to the current input parameter. In particular the LSTM parameters are stored as a stack of 4 matrices in PyTorch, and we find alignment is highest for the ”gate” submatrix, so we choose that for all plots. For the individual layer alignment, we plot alignment between the 3rd and 4th layer input parameters.

### A.4 Language Modeling with Transformers

We train a Transformer (Vaswani et al., ) language model on Wikitext-103 (Merity et al., ). We base hyperparameter choices on the Pythia suite (Biderman et al., ), specifically the 160 million parameter configuration with sinusoidal position embeddings, 12 layers, model dimension 768, 12 attention heads per layer, and hidden dimension 768. We use AdamW (Loshchilov & Hutter, ) with batch size 256, learning rate 0.0006 and weight decay 0.1. We use a context length of 2048 and clip gradients to a maximum norm of 1. We also use a learning rate schedule with a linear warmup and cosine decay to 10% of the learning rate, like Biderman et al. ().

For SVDs, for simplicity we take the SVD of the entire $(3d_{\text{model}},d_{\text{model}})$ parameter that computes queries, keys and values from the hidden dimension inside the attention layer, without splitting into individual heads. This is reasonable as the splitting is done after the fact internally. We also take the SVD of the output parameters, and linear layers of the MLPs, which are 2 dimensional matrices. For the individual SV evolution plot, we plot the SVs of $W_{1}$ of the 8th layer MLP

For alignment, we consider the alignment of $W_{Q}$ and $W_{K}$ matrices, $W_{V}$ and $W_{O}$ matrices, computing alignment between heads individually then averaging over all heads. We also consider the alignment between $W_{O}$ and $W_{1}$ of the MLP block, between $W_{1}$ and $W_{2}$ of the MLP block, and between $W_{2}$ and the next attention layer. For the individual layer alignment, we plot alignment between $W_{1}$ and $W_{2}$ of the 8th layer MLP.

### A.5 Weight Decay Experiments

Figure: (a) VGG
Refer to caption: extracted/5805818/figs/vgg/weight_decay_0.0-train-loss.png

All tasks are trained in exactly the same fashion as mentioned previously, with increasing weight decay in the set $\{0,0.0001,0.001,0.01,0.1,1.0,10.0\}$. For ease of presentation we consider a subset of settings across tasks. In Figure 14 we include trained model performance and pruned model performance to show that, even with high levels of weight decay, models do not entirely break down. More so, the approximation of the pruned model to the full model gets better with higher weight decay.

### A.6 Grokking Experiments

We mostly follow the settings and architecture of Nanda et al. (), except we use sinusoidal positional encodings instead of learned.

For the slingshot case we follow hyperparameter settings in Thilak et al. (), Appendix B except with the 1-layer architecture from Nanda et al. () instead of the 2-layer architecture specified. W perform addition modulo 97. The original grokking plot in Thilak et al. () appears much more dramatic as it log-scales the x-axis, which we do not do here for clarity.

### A.7 Random Label Experiments

We train a 4-layer MLP on CIFAR10 (Krizhevsky, ) with either completely random labels, or the true labels. We use SGD with momentum of 0.9 and constant learning rate of 0.001, and train for 300 epochs to see the entire trend of training. The major difference to the setting of Zhang et al. () is the use of a constant learning rate, as their use of a learning rate schedule might conflate the results.

### A.8 Magnitude Pruning Experiments

We use the same VGG setup as described previously. In this case we train til the end, then compute a global magnitude mask. To do this we flatten all linear and convolutional weights into a single vector, except for the last linear layer, and sort by magnitude. Then we keep the top 5% of weights globally, and reshape back to the layerwise masks. This results in different sparsity levels for different layers, so when generating the random masks, we use the per-layer sparsities that resulted from the global magnitude mask.

To retrain the network, we rewind to epoch 4, then continue training with the mask, always setting other weights and their gradients to 0. We average all results over 3 random seeds.

### A.9 LMC Experiments

We save 5 evenly-spaced checkpoints in the first epoch, as well as at the end of the next 4 epochs for 10 intializations in total. We train 3 trunks, and split 3 branches from each trunk for a total of 9 branches which we average all plots over.

Following Neyshabur et al. (), we compute the barrier between checkpoints as follows: given $W^{(1)}(T)$ and $W^{(2)}(T)$ that were branched from $W(t)$ we compute

$$ $b(t)=(\max_{\alpha\in[0,1]}\mathcal{L}((1-\alpha)W^{(1)}(T)+\alpha W^{(2)}(T)) -((1-\alpha)\mathcal{L}(W^{(1)}(T))+\alpha\mathcal{L}(W^{(2)}(T)))$ (6) $$

when this quantity is 0, we consider the checkpoints to exhibit LMC.

We recompute batch normalization parameters after interpolating for VGG-16, and group normalization parameters for the UNet, as these do not necessarily interpolate well (Frankle et al., ). We also compute singular vector agreement for the same parameter between either branch endpoint.

To plot the singular vector (dis)agreement and LMC between different modes, we make 11 evenly spaced measurements interpolating between branch endpoints that had the same split epoch, and the same branch seed, but different trunk initializations.

### A.10 Perturbed LMC Experiments

We perturb all weights $W$ after the point of dynamics stability where we expect to see LMC at the end of training (epoch 4 is sufficiently late in all cases) using randomly sampled normal perturbations $\epsilon\sim\mathcal{N}(0,I)$ with $\lVert\epsilon\rVert=\eta\lVert W\rVert$ where $\eta\in\{0.0,0.1,0.25,0.5,1.0,2.5\}$. We do not perturb the output layer, as this has a very substantial effect on the optimization. We also do not perturb the input layer for the Transformer as it is too computationally expensive for our resources.

## Appendix B Limitations

There are a few key limitations to our study. As mentioned, we lack the computational resources to run more than 3 random seeds per experiment, though we do find error bars to be quite tight in general (except for the generalization epoch in the grokking experiments). In addition, as discussed we ignore 1D parameters in the neural networks, which may be particularly crucial (especially normalization). In addition, due to computational constraints we do not consider alignment of layers across residual connections as this quickly becomes combinatorial in depth, thus there may be other interesting interactions that we do not observe. Finally, due to computational constraints we are unable to investigate results on larger models than the 12 layer Transformer, which may have different behavior.

## Appendix C Compute Resources

All experiments are performed on an internal cluster with on the order of 100 NVIDIA 2080ti GPUs or newer. All experiments run on a single GPU in less than 8 hours, though it is extremely helpful to parallelize across machines. We estimate that end-to-end it might take a few days on these resources to rerun all of the experiments in this paper. Additionally, the storage requirements for all of the checkpoints will take on the order of 5 terabytes.

## Appendix D Code Sources

We use PyTorch (Paszke et al., ) and NumPy (Harris et al., ) for all experiments and Weights & Biases (Biewald, ) for experiment tracking. We make plots with Matplotlib (Hunter, ) and Seaborn (Waskom, ). We also use HuggingFace Datasets (Lhoest et al., ) for Wikitext-103 (Merity et al., ).

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: This paper is clearly written and I do not have other questions.
soundness: 3
strengths: 1) This paper is clearly written and the idea is easy to follow.
2) The authors connect simple settings (grokking) to complex ones (LSTMs, transformers), and explore tasks from different domains including image, speech and language processing tasks. This shows that the rank minimization is a common property for people to understand deep neural networks.
summary: The authors propose an empirical approach centered on the spectral dynamics of the behavior of weights during optimization and they find that 1) Grokking is intimately linked to rank minimization; 2) Rank minimization is a general phenomenon in more complex tasks; 3) Weight decay acts implicitly as a low-rank regularizer; 4) Generalizing solutions have a lower rank than memorizing ones; and 5) Top singular vectors are preserved when performing magnitude pruning and while linearly interpolating between connected modes. 

Those phenomena provides a coherent framework to understand the behavior of networks across different settings.
weaknesses: As this paper is mainly empirical, it would be better to see more results of different tasks in each domain to give more evidence for the observed properties, as one task in each domain might not be enough to say they are general phenomena.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 2
questions: - What could be the reason for not observing a strong alignment between layers as proposed in the existing theory? Why does the alignment appear and then go away? I found this surprising and would like to hear what the authors think.
- The spectral dynamics are different between training with random vs true labels. But how could this insight be leveraged to distinguish memorization vs generalization between different networks trained on the same task? Would the idea be that the higher rank network is more memorizing?
soundness: 3
strengths: - Spectral analysis of the weights is an informative approach that has yielded several theoretical insights in linear networks and informed empirical algorithms, particularly in recent approaches for low-rank adaptation. In the past, such analyses have been done in isolation on simple setups. So this paper's attempt to extend the scope of the insights into something more general is novel.
- The comprehensive experiments touch upon a wide array of observed phenomena in deep learning and generally support the idea that the top singular subspace is more important for task performance.
- Some of the findings were surprising, and call for further study. For instance, panel d in Figure 2 shows an increasing alignment that moves up the ranks over training, suggesting the evolution of the relevant functional low-rank subspace.
- I also found the differences between random label learning and true label learning to be interesting. This nicely highlights how the task structure implicitly shapes the network over learning.
summary: The paper surveys the spectral dynamics of the weights of deep neural networks in several settings, attempting to draw some general conclusions from the commonalities in the observed dynamics. In particular, the authors examine the singular values of weight matrices in the network, the alignment between singular vectors of consecutive layers, and the effect of weight decay on the dynamics of these quantities. They find that neural network training is biased towards low-rank solutions with only a few singular modes dominating information processing after training, in line with previous theoretical and empirical findings. This effect is shown to be more pronounced with higher weight decay. They also find that the alignment between layers is not quite as strong as prior theoretical work has suggested, and requires further study. The paper also reports interesting connections between spectral dynamics and deep learning phenomena such as grokking, generalization, memorization, lottery tickets, and linear mode connectivity.
weaknesses: I have some critiques concerning the writing, particularly the lack of any solid insights. I elaborate below.
- The paper seems to be largely an exercise in plotting the singular value dynamics of several neural networks. While this is interesting in itself, I found that the paper did not extract any conclusive or striking insights from the empirical study, making it difficult to judge the significance of the work. For example, Figure 2 points out that grokking seems to be correlated with drops in the effective rank. This is interesting, but no concrete mechanistic explanation for this effect is attempted. A similar tone pervades the paper, with several disparate empirical reports but no substantive specific hypothesis or overarching message about the phenomena being observed. I understand that the paper is primarily empirical, but the paper would be much more compelling if the reader is given more details as to the significance of each finding or its implications for practitioners.
- A few concrete insights are indeed pointed out, such as weight decay implicitly promoting a low-rank bias. I appreciated the arguments provided for developing an intuition for this effect (lines 410-420). While interesting, this was not investigated in sufficient depth to clear the bar for acceptance of the paper. Furthermore, the reasoning for looking into this effect (lines 193-197) seems somewhat vague. 
- The lack of a core message is also reflected in the writing. For instance, the abstract mentions a "consistent bias in optimization" (line 13) but doesn't specify what this bias is at all. Similarly, I found the introduction perhaps too long but unspecific. It briefly describes several prior works that could be better placed in the related work section. More importantly, I found it difficult to appreciate what is the main question that is being asked. The closest was the "aim to provide a common language for probing and understanding neural networks" (lines 144-145), but still, such a language had not been specified in detail by the end of the paper. I think the paper could be much improved by addressing just these few aspects and being more specific about the core message of the paper.
- In general, I also found the treatment of prior work a little lacking in details. While I appreciate the comprehensive literature that was cited, I think the paper could benefit from closer tailoring of the references to a more specific core message.

The concerns I have raised can largely be addressed by being more specific about the significance of this work and some rewriting. I look forward to hearing more from the authors.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: Please see the weakness.
soundness: 3
strengths: The paper introduce some situations in deep learning and explain them by the spectral dynamics of weights. The advantages are

 1) The paper establishes a link between ''grokkking'' and rank minimization.

 2) The paper provides different examples of deep learning, including CNN, UNet, LSTM, and transformer.
summary: This paper presents a framework based on spectral dynamics—the evolution of weight singular values and vectors during training—to unify key deep learning behaviors. The authors reveal a consistent optimization bias across diverse tasks, from image classification to language modeling, and demonstrate that weight decay strengthens this bias beyond norm regularization. Spectral dynamics also distinguish between networks that memorize versus generalize, offering insights into effective sparse subnetworks (lottery tickets) and the structure of loss landscapes through linear mode connectivity.
weaknesses: 1) There is no clear description of the influences of spectral dynamics on deep learning. For example, in the line 421, weight decay can produces a low-rank behavior, but how much weight decay is needed or is weight decay related with some performance metrics? Figure 6 shows a phenomenon of adding weight decay. However, there lacks a metrics to show the level of weight decay. "The exact choice of “too much” varies across architectures and tasks." could not explain everything. 

 2) The theoretical foundation is weak, providing limited intuition and isolated examples rather than a cohesive theory. In section 4.1 and 4.2, the authors provide the visual processing of trainings, but there is no explanation and theoretical analysis. The authors should provide more accurate analysis rather than only visualization.

 3) The conclusion of the paper seems to be too simple and not rigorous. It provides some right but not useful insights. In section 6, the authors provide additional connections between spectral dynamics and other phenomena. However, the connections seems to be verfied by previous works, the paper only provides some examples and visualizations of these connections.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: - In top row of Figure 2(a), why is there a large blue shaded region? The caption doesn't introduce what the shade represents. My first guess is the standard deviation but the shade reaches negative values.
- In top row of Figure 2, is the model a one-layer transformer? If so, why are there 6 layers in Figure 2(c) and 50 layers for a 12-layer transformer in Figure 3(d)?
- In line 359, why can't the last layer be pruned?
- In line 395, the authors state that the absence of alignment is reminiscent of Mulayoff & Michaeli. Mulayoff & Michaeli found that the alignment appears with GD and SGD with reasonable stepsizes and may break for GD with a large stepsize. So I'm wondering if absence of alignment in Figure 5 is due to large stepsizes and can be recovered if a smaller stepsize is used. Also, if I'm not mistaken, the assumption of whitened input is not necessary for alignment in linear networks.
soundness: 3
strengths: - This paper is well-written. The phrasing is polished. The literature review is mostly comprehensive. The problem is motivated well with an insightful introduction. I enjoyed reading it.
- The authors use the tool of spectral dynamics to examine a wide variety of tasks and analyze many recent outstanding questions in deep learning.
- Extensive experiments are carried out to provide information for the analysis.
summary: This paper proposes to inspect the evolution of singular values & vectors of weights in training to gain mechanistic interpretability in deep learning. Experiments are conducted on various tasks with various models. Results shed light on grokking, low-rank weights, weight decay, and alignment between adjacent layers.
weaknesses: - The authors reviewed some prior theoretical papers and mentioned that they often cannot explain seemingly relevant phenomena in practice, which I appreciate. But in the same spirit, there are probably cases where this paper's tools cannot explain either. Could the authors give a few such instances or some relevant insights? For instance, in line 118, the authors write that any arguments about generalization cannot be capacity-based alone. Then I wonder whether arguments about generalization can be spectral-dynamics-based alone. Are there cases where the generalizing solution has a higher rank than the memorizing solution? If yes, perhaps the conclusion in line 140 should be revised or stated with more caveats.
- For the architectures studied in this paper, except for MLPs, there is not a very standard definition of alignment off the shelf. As mentioned in the Appendix, the authors tried a few different definitions before settling on the current version. So I am wondering if the results vary much with different ways of evaluating alignment.

Meta-review: The authors studied the evolution of singular values and vectors of weight matrices during training. In particular, the authors observed several interesting empirical phenomenon of low rank behaviour related to grokking, weight decay, and magnitude pruning. 

I must admit the reviews and the scores left this paper very borderline, and the discussions did not seem to make further progress, so this is a very close call. Since a decision must be made, I have to be a bit more opinionated towards this paper. 

I have mentioned this in my discussion with reviewers privately, but I am broadly speaking a bit skeptical of this approach of empirical research: making a lot of interesting observations, but not drawing useful conclusions. This is obviously subjective, but I did see some reviewers echo the same concern. I am genuinely not sure if these interesting observations can be put together in a useful manner to be built upon. 

For example, in Figure 2, there are a lot of information here. I'm pretty confused which part of the plot is helpful for me to understand what's happening. More importantly, as a researcher reading wanting to build on top of this, what are some useful take-aways here for the next project? Does this figure help us understand grokking or low rank behaviour of weights? These are genuine questions of mine, and I hope you don't take the tone as being harsh. 

That being said, there are ways this paper can improve over the borderline for me. As reviewer vA12 has already mentioned, a more conclusive story developed from the observations will be helpful. If there is a helpful interpretation beyond "these singular value plots are interesting", I can see this work being built upon, and therefore would see this in a positive light. 

However, at this point, given a decision must be made, I will recommend reject based on the discussion above and the borderline score.

Decision: Reject