Small Singular Values Matter: A Random Matrix Analysis of  Transformer Models
Authors: Max Staats
Matthias Thamm
Bernd Rosenow
Venue: NeurIPS 2025 poster ()
Keywords: Random matrix theory
RMT
Llama 3
Spectra
singular value decomposition
Primary area: deep_learning
TL;DR: We investigate the importance of small singular values and present a theoretical model that explains their relevance.

Abstract: This work analyzes singular-value spectra of weight matrices in pretrained transformer models to understand how information is stored at both ends of the spectrum.
Using  Random Matrix Theory (RMT) as a zero information hypothesis, we associate agreement with RMT as evidence of randomness and deviations as evidence for learning. 
Surprisingly, we observe pronounced departures from RMT not only among the largest singular values - the usual outliers - but also among the smallest ones. 
A comparison of the associated singular vectors with the eigenvectors of the activation covariance matrices shows that there is considerable overlap wherever  RMT is violated. Thus, significant directions in the data are captured by small singular values and their vectors as well as by the large ones. We confirm this empirically: zeroing out the singular values that deviate from RMT raises language-model perplexity far more than removing values from the bulk, and after fine-tuning the smallest decile can be the third most influential part of the spectrum. To explain how vectors linked to small singular values can carry more information than those linked to larger values, we propose a linear random-matrix model.  Our findings highlight the overlooked importance of the low end of the spectrum and provide theoretical and practical guidance for SVD-based pruning and compression of large language models.

Small Singular Values Matter: A Random Matrix Analysis of Transformer Models
Max Staats
Matthias Thamm Institute for Theoretical Physics Leipzig University thamm@itp.. wnw.uniileeeeee mmni-leizigg de
Center for Scalable Data Analytics and Artificial Intelligence Leipzig University staats@itp.. staats@itp.umi-leiip www-leipzig........... de
Bernd Rosenow
Institute for Theoretical Physics Leipzig University
rosenowGphysik. mmi-leipzig.dd de
Abstract
This work analyzes singular-value spectra of weight matrices in pretrained trans- former models to understand how information is stored at both ends of the spectrum. Using Random Matrix Theory (RMT) zero information hypothesis, we asso- ciate agreement with RMT as evidence of randomness and deviations as evidence for learning. Surprisingly, we observe pronounced departures from RMT not only among the largest singular values the usual outliers but also among the smallest ones. comparison of the associated singular vectors with the eigenvectors the activation covariance matrices shows that there is considerable overlap wher- ever RMT is violated. Thus, significant directions in data are captured by small singular values and their vectors well as by the large ones. We confirm this empirically: zeroing out the singular values that deviate from RMT raises language-nn modell perplexity far more than removing values from the bulk, and after fine-tuning the smallest decile can be the third most influential part of the spectrum. To explain how vectors linked small singular values can carry more information than those linked to larger values, we propose a linear random-matrix model. Our findings highlight the overlooked importance of the low end of the spectrum and provide theoretical and practical guidance for SVD-based pruning and compression of large language models.
Introduction
Large language models (LLMs) have become foundational in deep learning, revolutionizing natural language processing tasks such as translation, text classification, and question answering [27 45| 48
51. Despite the vell-documented success (29), thorough theoretical understanding of their inner workings remains incomplete. Although researchers have investigated various facets of LLMs (37]. fundamental questions persist about how these models encode information and the specific roles of their components.
One promising approach for gaining deeper insights is the application of random matrix theory (RMT), which has proved effective for identifying structural properties and information density in neural networks 3I help determine where information located within models. When networks are randomly initialized, the weight distributions precisely matches RMT predictions. After training, deviations from these predictions reveal how model parameters have adapted.
43 ir In particular, RMT analysis of the spectrum of weight matrices can
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

hhrpg//--eeer/ettt-sss--aneeee Building on these insights, we use RMT to pinpoint regions in LLMs where relevant features are encoded, by identifying deviations from the RMT-predicted spectrum. We study the singular value spectra of weight matrices from three pretrained models: Ber [66] Pythia [ and Llama-88 [4].
We identify the regions lying outside the theoretically predicted Marchenko-Pastur spectrum (30 as areas of feature learning by comparing the corresponding singular vectors with the covariance matrix of the layer activations, finding strong similarity. Interestingly, this phenomenon is not only present for the largest but also for the smallest singular values. This similarity stays consistent across different blocks of the transformer architectures and holds for all three models we examine. When removing groups of singular values (and associated vectors) from these models, performance degrades most significantly for the smallest and largest singular values that violate RMT properties.
Additionally, we contribute to the ongoing discussion about removing small singular values in LLMs. Previous studies suggest that small singular values can be relevant for generalization [23], while other work indicates potential benefits from removing them [41] However, the most common perspective that they are negligible, as their removal is the optimal low rank W' solution for weights W under the L2 norm |W 22 [15]. We reconcile these perspectives by showing in which matrix types small singular values are important, and that the potential damage that is done by removing the smallest singular values from pretrained transformer can be recovered by fine-tuning step. Our results are of crucial relevance to any researcher doing SVD-based pruning with LLMs. All code to generate the figures is open source and available under (2).
2 Related Work
RMT has been widely used as a calculational tool for performing statistical averages in the analysis of machine learning models. Early applications of RMT to neural networks, such as (35), analyzed the spectral properties of loss surfaces in deep learning, providing insights into learning dynamics. Building on this foundation, Baskerville et al. [6] derived universal aspects of outliers in loss surfaces. Beyond its role in statistical analysis, RMT has been proposed as a tool for analyzing trained network weight matrices 34. In 31,, RMT was applied
weight matrices by examining the learning dynamics of image recognition models through their spectra. Following up on this work, Martin et al. 32 suggested that large outliers in the singular value spectrum are indicative of well-trained matrices. Further studies [28] 44) reinforced RMT's s utility in understanding how networks evolve during training. They demonstrated that deviations from RMT predictions indicate where feature learning occurs, as opposed to lazy learning (12), where weights remain close to their initial random state. These findings underscore RMT's potential for identifying regions of learned features without the need for training data.
Transformers present unique challenges in understanding information storage. Prior work 25 has shown that different matrix types specialize in storing distinct types of knowledge, while Aken et al. [1] examined how semantic information is encoded in neuron activations. In [39], LLMs compressed by a low-rank approximation of the weights that optimizes for a minimal change in the activations. 44 compress LLMs based on similar metric but transform the data which the activations are computed first, as they find that the rank of the singular values does not represent their relevance otherwise. To overcome such issues with SVD-based approximations, several works 50 focused on quantizing weights instead of creating low-rank approximations.
38
11 16
47
The low-rank structure of features in neural networks has been explored in 20). Yu and Wu 152 highlight that even though the feature matrices of transformers are often low rank, their weight matrices are not, revealing complex relationship between representations and parameters. Positional encodings, crucial to transformer performance, have also been studied for their role in shaping the leamed feature space 466
3 Notation and Spectral Properties of LLMs
Large language models (LLMs) are typically composed of three main parts: an initial embedding layer, repeated stack of transformer blocks, and final output layer. Each transformer block
2 Flleuhher//iyyhhi---on--dddddddd mnta--lamaaMMta. Mww.rMMMMMa---aaa.... Llama-3.1

---

Random Control
Pythia Key 5
Random Control
Pythia Up-Proj 5
(a)
(b)
(c)
(d)
0.5
MP-Thoo Theory
Nx 4N
MP-Thoory Outlier
p(2)
Outlier
0.5
0
0
2
0
2
0
2.5
2.5
V
Figure 1: Spectra of random matrices (a,c) and trained matrices (b,d) in comparison the theoretically predicted Marchenko curve for square matrices (a-b) and non-square matrices (c-d). Panel (a) shows that the theoretical prediction for a square matrix with dimension N=768 agrees perfectly with the empirical spectrum of a untrained weight matrix. Panel (b) shows that training led to the emergence of outliers in the Key matrix of the fifth block from a Pythia model. Panel (c) shows the perfect agreement of theory and initialized weight matrices in the case of non-square matrices. Square and non-square matrices differ fundamentally, as non-square matrices can also exhibit outliers to the left, as displayed in panel (d) for the Up- Pp-Projection matrix of block for Pythia model.
B(0, where I [1, N] denotes the block index, has the same internal structure and weight matrix shapes. In this work, we use a consistent naming convention for the weight matrices within each transformer block. The attention sub-layer uses the Query matrix W the Key matrix W the
Value matrix W and the MientionnOutphe matrix W:O These matrices form the multi-head attention mechanism, which can be written as
(WWX))(><)) (WVx)) Wo.
Att(X)= softmax
(1)
vdz
Following the attention sub-layer, the feedforward sub-layer typically employs Up-Projection
matrix l and a Projerrooection matrix ( Depending on the architecture, there may also be
Gate-Projection matrix w:(.) In the case of a Gated Linear Unit (GLU), these matrices are related by GLU(X) = [o ( + by) (Wa. + bG)l WD.
(2)
where denotes the Hadamard product. If no GLU is used, Wy and Wo are combined with nonlinearity to form a standard feedforward layer.
To analyze these weight matrices, use singular value decomposition (SVD) to factor each weight matrix into its singular values and singular vectors. For a matrix Rmxx,, the SVD
W USV
where E mm and E Raxn are orthogonal matrices containing the left and right singular vectors, respectively, and S E Raxx is diagonal matrix holding the non-negative singular values.
When q n/m stays constant, m, n + oo and the entries of are i.i.d. with finite variance a and zero mean, the distribution of its singular values follows the Marchenko-Pase (MP) law [30]
P(v) = u (v - v²) -v2) V VE [4]
(4)
0
else
f1 = 6 1//) a=ovm,
(5)
where 2 without loss of generality. In large, randomly initialized neural networks, the singular values weight matrices typically approximate this distribution. After training, deviations from the
MP law indicate learned structure, often manifested as outliers beyond the MP bounds [v_, Past work suggests that such outliers can signal effective feature learning
32
When plotting the theoretical Marchenko- Pastur curve of an empirical matrix, we estimate the standard deviation of the underlying matrix as described in 4 This is used in Figure [ illustrate the Marchenko-Pastee matrices. We see that for random weight matrices of BERT size, where the smaller dimension has N 768, the theoretical prediction agrees perfectly with the empirical spectrum shown in
where we distribution together with the spectra of trained and random weight

---

Bert, Key 5
Pythia, Key 15
Llama, Query 25
1.0
1.0
1.0
p(v)
0.0
0.0
0.0
0.2
30 Interval
0.2
0.10
0.1
0.1
0.05
2
V
Figure 2: Spectra and maximum overlap O₂ of the corresponding right singular vectors with eigen- vectors of the activation covariance matrix (see Eq. 77) for square matrices of different LLMs. We observe that outliers of the Marchenko-Pastue bulk have significantly increased overlap, indicating that important information might be carried. In case of square matrices, singular values can only exit the bulk in the region of large singular values.
panels (a) and (c). When training weight matrices, outliers emerge as shown in panels (b) and (d) for the Key and Up-Projection matrix of Pythia. An important observation is that in the case of square matrices, outliers can only emerge towards larger values. In the case of non-square matrices, the theoretical spectrum has a minimal value that is strictly larger than zero, allowing for outliers towards small singular values. Neglecting these values to construct a low rank matrix W from W is often considered optimal as the norm WW W22 is altered the least. However, we will show in the following that these singular values contribute significantly more to the performance of the network than larger singular values that remain in the bulk.
4 Overlap of Features and Weights
The following chapter shows how the singular vectors that correspond to outliers of the spectrum- small and large- can be related to important directions of the data matrix. This is done by computing the covariance matrix of the activations that enter each matrix in each block and comparing the eigenvectors of this covariance matrix the singular vectors of the corresponding weight matrix.
Formally we define the activation covariance matrix C(W(O) of matrix (()) using the input activa- tions h (9) to matrix W(()) and the average input activation entering that matrix h(W()). Using the index as a token index that runs over the dataset, we compute
C (h: h:)) (h; h.)) h.)))
(6)
where we dropped the explicit dependence (1) for cleaner notation. The resulting matrix C is symmetric, and therefore has an orthonormal set of eigenvectors f. with eigenvalues Aç.
Since the transformation of matrix W is given as Wh+ + b, where W has singular value decomposition W VT the activations are mapped onto the space spanned by the right singular vectors V. To see whether particular eigenvector fj of the activation covariance matrix aligns with one of the right singular vectors Ug of W. we compute
O₂ max, (t% fz), j (1,
This measure quantifies the extent to which singular vector captures specific direction of the data represented by the activation covariance matrix. In the following, we estimate from the WikiText 33 dataset. The appendix shows additional results from the BookCorpus [53] datasets.
Figure 2 shows the overlap computed based on Eq. 7 in comparison the spectrum of our three transformers. As the displayed matrices of Bert, Pythia, and Llama are square, we find outliers from Marchenko-PPi Pasttr distribution only on the right side of the spectrum (upper panels). The bottom panels show the overlap O₃ of each singular vector at the position where the corresponding singular value is in the spectrum. We observe that singular vectors corresponding to outliers of the spectrum have a strong correspondence to one of the eigenvectors of the activation covariance matrix. We can

---

Bert, Up-Proj. 5
Pythia, Up-Proj. 15
Llama, Up-Proj. 25
0.5
p(v)
0.5
0.5
0.0
0.0
0.0
0.3
30 Interval
0.3
0.3
0.0
0.0
0.0
4
Figure 3: Spectra and maximum overlap O₁ of the corresponding right singular vectors with eigenvectors of the activation covariance matrix for rectangular matrices of different LLMs. The singular vectors corresponding to large singular values exhibit a significant overlap with the activation covariance matrix. However, the case of non-square matrices, we also observe an increased overlap with the singular vectors corresponding to the smallest singular values. This is surprising as the smallest singular values contribute very little the variance of the activations in the next layer and are typically regarded as something that can be discarded.
interpret these singular vectors as picking up direction of the data with high variance and weighting highly. For reference, the grey lines display the 30 interval when assuming singular vectors from random matrix. We find several statistically highly significant points above these lines.
In Figure Rl we show rectangular matrices of all three transformers, where in this case, outliers of the spectrum can emerge towards large values and small values. Interestingly, we observe that singular vectors of both groups have strong correspondence to an eigenvector, indicating that the singular vectors corresponding to the smallest singular values encode properties of the activation covariance matrix and hence of the data! This is exceptional result small singular values contribute the least to the overall matrix with respect to the 1.2 norm, and they are the first ones to be discarded when pruning naively. The results are again statistically highly significant even with respect 3o interval.
The plots illustrated the relation between small singular values, and the data for specific matrices. To further look into this phenomenon, we present all matrix types of block 10 for both Pythia and Llama in Figure 4 Other blocks are displayed in the appendix. We see that for Pythia (upper 2 panels) Query. Key, and Value matrix exhibit increased overlap only for their largest singular values. The Attention-Output matrix has significant overlap with the activation covariance matrix, a phenomenon that holds for BERT, Pythia, and Llama, as elaborated in the appendix. The Up and Down-Projes Down-Projection matrices exhibit significantly increased overlaps for both their largest and smallest singular values. However, we also find singular values with small overlaps in this region. This is different for Llama's 's Up and Projection matrix, where the small and large singular values significantly overlap with the covariance matrix. Interestingly, the singular vectors of the Down-Projection matrix have no increased overlap with any of the eigenvectors from the activation covariance matrix. We suspect that the placement of this matrix at the end of the Gated Linear Unit-rather than after a separate nonlinearity may render certain weights redundant, leading to no significant overlap. Surprisingly, we find that the Key and Value matrix do not exhibit large overlaps for their smallest singular values while being non-square matrices. We speculate that this arises from their indirect relation to the activations of the previous layer, which are processed in the Attention mechanism. The development of the overlap as training progresses can be found in the appendix.
5 Contribution of Singular Values to Model Performance
In the preceding section, we observed a strong overlap between the singular vectors of weight matrices particularly in regions outside the Marchenko-Pastur (MP) boundary and the eigenvectors activation covariance matrix. To further evaluate the importance of these singular values and their corresponding vectors, we conduct experiments in which we selectively remove certain groups of singular values. Specifically, removing singular value v, from weight matrix W involves zeroing
5

---

Query
Key
Value
Att.-Out
Up-Proj.
Down-Proj
Pythia Block 10
3a Interval
0.25
0.00
Gate-Proj. 0.0
22.5
22.
Llama Block 10
3a Interval
0.11
0.0
0.0
2.5
0.0
0.0
0.0
2.5
2.5 V
V
Figure 4: Spectra and the overlap O₃ for block 10 of Pythia and Llama. We find that in the case of non-square matrices, the singular value outliers to the left of the spectrum can have a significantly increased overlap Og with the eigenvectors of the activation covariance matrix. This is the case for the Up-Projection matrix of Pythia and Ilama, as well as Llama's Gate-Projest Projection matrix. In case of the Attention-Outpu matrices and the Down-Projection matrix of Llama, we find very little overlap. This is systematic finding and may reflect the training dynamics (see Appendix for details).
out v. in the diagonal matrix S of its singular value decomposition,
W USV
(8)
3 S
for else
W=UŠVT
(9)
We then reconstruct the weight matrix W using the original singular vectors. To compare the relative impact of different parts of the spectrum, we group the rank-ordered singular values of each matrix into ten equal-sizee squel-sized deciles, with the smallest 10% in the first decile and the largest 10% in the tenth, We apply this procedure by choosing one matrix type (e.g., Query) and zeroing out one of the ten deciles
weight matrices of that type. We then measure the resulting increase in A Perplexity PPL(Jw)) PPL(Sw). on the WikiText dataset. The results are shown in Figure where removing the largest singular values consistently causes large perplexity increase in all matrix types for both Pythia and Llama. This is expected the largest singular values and corresponding vectors are the directions that describe the largest percentage of the variance the next layers' activations.
When going through the deciles from large to small, we observe that the contribution of the deciles decreases monotonically for all square matrices. For the non-square matrices, the largest nine deciles also follow this trend. However, for non-square matrices, the smallest decile always contributes more than some of the larger deciles. This observation underscores that valuable information can be encoded at the lower end of the spectrum, potentially explaining why standard pruning strategies often struggle with transformers 3
These findings are further backed up by Llama results on the GSM8K benchmark, which tests basic math problems. These results are depicted in Table I. where decile one corresponds to the smallest and decile ten to the largest singular values. For the Down-Projection matrix, we find that the smallest singular values (decile 1) are the second most important decile, which is excellent agreement with our previous results for this rectangular matrix (compare Figure 5). In contrast, considering the quadratic Attention-Ouie Output matrix, the smallest singular values are one of the least important deciles, in agreement with theoretical expectations and the previous observations. The smallest singular values of the non-quadratic Gate-Pros Projecrront matrix are fourth important decile, also excellent agreement with the results of Figure For the quadratic Query matrix, only the two largest deciles appear to important. Further results for the HumanFyal benchmark show similar pattern and are displayed in the Appendix.

---

Query
Key
Value
Att.-Out
Up-Proj.
Down-Prop
APerplexity Perpleexity
10a
109
105
10a
Pythia
10-
10a
100
109
10%
10a
10
10
Gate-Proj.
10
10
10a
100
10a
Jlama
10"
10
10
10
10'
10
10'
1 3 579 decile removed decile removed decile removed decile removed decile removed decile removed decile removed
3 5 9
3 579
3
5
1 3.579 9
Figure 5: Increase in perplexity on WikiText for Pythia and Llama when removing deciles of rank- ordered singular values. Singular value deciles are removed from all blocks, but only from specific matrix type, e.g., all Key matrices. The inset shows the respective spectra averaged over all blocks. As expected, removing the largest singular values substantially affects perplexity for both models and all matrix types, as the matrix changes significantly upon the removal. Interestingly, we find for non-square matrices, i... matrices with spectra that have outliers towards small values, that the decile with the smallest singular values more important than some of the larger deciles. In case of the Down- Projecctoonnn matrix, these singular values even the second most important group. These findings confirm that meaningful information may reside in the lower end of the spectrum.
We provide further experiments for removing deciles from all weight matrix types for LLaMA 3, 8B Chat on the RULER [22] benchmark for context length of 8192. The tasks needle in haystack (niah) were evaluated with several values to be extracted (niah multiV) and for several queries (niah multiQ). We also conducted experiments on value tracking (vt), common word extraction (cwe), and frequent words extraction (fwe). The results when removing singular values from all layers simultaneously are shown in [ We find that the smallest singular values are even the second most important decile in this task.
and SST2 [42]. We compare two scenarios: (i) removing a particular decile of singular values before fine-tuning and then fine-tuning the altered model, versus (ii) fine-tuning the model first and then removing the singular value decile. Now, we are removing deciles from all weight matrices simultaneously to compare to previous literature results and estimate the 30 interval from the standard deviation of six runs without pruning. Figure 6 shows the results when removing singular value deciles from Bert, where in case (i) -left pannel, prune first-, removing small singular values reduces the accuracy slightly but statistically significantly in the case of RTE and SST22. For BoolQ. the lost knowledge was either recovered not necessary. By contrast, case (ii) -right panel, fine-tune first-, removing the same small singular values degrades accuracy significantly in all cases, showcasing their surprising relevance. Large singular values (decile ten) are consistently important, as removing them drops performance to near random guessing (off-scale points in the right panel). This finding provides insights into a recent debate about the importance small singular values transformers: some argue that they are essential for good performance [23], while others report performance gains from their removal Our results clarify this conflict by showing that small singular values do matter,
For Bert additionally conduct fine-tuning experiments on BoolQ 13] rte 8 19 21
Table 1: Effect of removing deciles of singular values from various weight matrices of Llama3 3-BB on the GSM-8K accuracy under 3-shot prompting. Decile 1 corresponds to the smallest singular values, and Decile 10 to the largest. Removal of the smallest singular values from the non-square Down-Projection matrix significantly reduces accuracy (from the 43.2% baseline 2%), confirming their importance. The smallest singular values of the square Attention Preettoo--uuput and Query matrices are not particularly relevant.
Matrix
Dec. 1
Dec. 2
Dec.
Dec.
Dec. 5
Dec. 6
Dec. 7
Dec.
Dec. 9
Dec. 10
Down-Proj.
2.0%
30.2%
28.0%
27.1%
26.2%
22.0%
11.0%
15.0%
5.0%
0.0%
Att.-Out.
40.0%
41.8%
39.0%
38.8%
39.8%
37.1%
37.6%
31.4%
25.1%
0.0%
Gate-Proj.
34.1%
37.0%
39.1%
39.6%
38.1%
38.0%
35.9%
33.3%
23.6%
0.0%
Query
40.3%
44.3%
41.4%
40.5%
39.5%
40.1%
40.7%
40.7%
35.8%
0.0%

---

Table 2: LLaMA 3.1 8B Chat results on RULER benchmark tasks (context length 8192) when
removing singular value deciles from all layers simultaneously.
Task
Dec.1
Dec.2
Dec.3
Dec.4
Dec.5
Dec.6
Dec.7
Dec.8
Dec.9
Dec.10
niah multi
0.0
61.0
88.5
7.0
65.5
99.25
13.25
0.0
0.0
0.0
niah multiQ
0.0
69.5
80.5
0.0
57.75
99.0
11.75
0.0
0.0
0.0
vi
60.6
56.2
33.6
6.4
47.6
0.0
0.0
cwe
0.0
57.8
37.8
21.0
13.7
8.3
9.0
3,2
0.0
0,0
fwe
0.0
86.33
83.0
69.33
17.33
84.0
33.67
3.0
2.0
0.0
reduce, finctune
finctune, reduce
validation accuracy
0.8
0.6
X
BoolQ
RTE X SST2
3r Interval
5
7
9
3
7
9
decile removed
decile removed
Figure Effect on validation accuracy on BoolQ, RTE, and SST2 when removing deciles of singular values from all matrices (except embedding weights) in Bert transformer. Decile ten corresponds to the largest 10% of singular values, and removal (both before and after fine-tuning) significantly lowers accuracy (left and right panels). For the smallest singular values (decile one), the two scenarios differ. When the smallest values are removed before fine-tuning, final accuracy is either fully recovered, as in the case of BoolQ, slightly reduced (left panel). If the model is fine-tuned first and then has its smallest singular values removed, accuracy declines statistically signifieantly for all three datasets (right). This indicates that the information that is stored in small singular values, along with their corresponding vectors, can sometimes be recovered by fine-tuning or is unused in a specific dataset, as seen in the of BoolQ.
but mainly once the model has been fine-tuned. Hsu et al. [23], fine-tune before pruning, and observed that small singular values are critical, while Sharma et al. [4] found gains from pruning them, and evaluating the model without additional fine-tuning. We interpret this behavior evidence that fine-tuning and potentially alignment- relies on small singular values and their associated vectors. Since alignment may degrade performance reasoning tasks (36). removing these directions improve task metrics. However, doing so after alignment could unintentionally erase that alignment.
Minimal Random Matrix Theory Model
The aim of this section is to provide minimal RMT model for the occurrence of small singular value outliers. For this, we consider a simple two-layer linear student model, trained on Gaussian inputs, and labels produced by linear rank-1 teacher. The effects of non-trivial correlations in the noise depend on whether the noise has predominantly small or large covariance the direction the outlier.
common model describe weight matrices W consists of low-rank matrix W,, encoding the rule and a noise matrix X as W Wo + X. Under the common assumption [24 43 of white noise, (X) O, Cov (X) = 1, singular values of W, can only outliers Marchenko-Pasee
if they lie above bulk of X, i.e. above the BBP critical value 5 7 10] In the following, we present a model with a non-trivial covariance of the noise, which can have a singular value of below the
MP bulk.

---

We start from a teacher-stude sttdent setup with teacher function dr({) u Xy with normalized u e RNxI and À E IR. The student is a linear two-layer network o5 v W with normalized, fixed second-layer weight
R and trained weight matrix W R of aspect ratio q = K/N 1. Here, may be interpreted as a learned, stabilized weight singular vector in a subsequent layer of real network. We show in the following that this model explains the occurrence of small outliers below the MP bulk, and that it relates the amount of information contained in the outlier singular vector the distance of the outlier to the bulk.
We consider a mean caan---uared-eree error loss function C(E) = -2 Ag - T.S-oTWES W/))222 for inputs & E EV(0,1), 1),
giving rise to generalization error
Eg (w =- (C(E)) = Tr [(u À w])]
(10)
2
Here we adopted the notation A² = A A. We study the weight matrix ensemble
N (T(WTw) w] x exp 2 2a
P(W) x exp - (W)
Tr [(W - EZ (W Wo)
(11)
where Sa + DO corresponds to perfect learning, and the a term has the role of a prior for weight
matrices at initialization
L2 regularization strength. In annealed approximation, / corre-
sponds to the number of examples shown to the network divided by the input dimension [7] the mean is Wo Itop vu and the correlation matrix is given by
Here,
aBA
a²B 1 aB vo
== a1
(12)
Weight matrices of this matrix ensemble have the form W = Wo + X, with a random matrix X of zero mean and row covariance matrix Σ (while columns are uncorrelated). For 0 (1 + a,3) < 1, the noise matrix has an outlier singular value below its Marchenko-Pane Pastur bulk. This stabilizes a small singular value outlier of W if < Va(1- (1)1-))-
g)
at [8
1/2
(Umin) =
+ +x + 0,3(1++888 BX²)
go
(13)
a:))
1 1+a0(1+3XX
a1+
For the case of perfect learning, i.e. - 0o, the outlier singular value approaches X A/1 -=x² only differing from À of Wo by the level repulsion with the bulk.
In Fig. we depict (Mmin) (left panel) and the singular vector overlaps (right panel) (lã . ul) (blue) and (1ō - wI) (red) of the corresponding singular vectors ú, of W = VUŪT as a function of 1/(1 a.8). We find that ideal learning. 1/(1 + aB) + O, corresponds separated from the MP bulk with best overlap of the corresponding singular vectors of W with those of Wo. For poorer learning, the singular value Vmin moves towards the bulk and the vector overlap (lū ui) approaches the expected overlap between random vectors (blue, dashed line), when outlier and MP bulk fuse at Boul = a² (1 v a² ) /2X?.
outlier maximally
Limitations
We show empirical evidence that the singular vectors corresponding the smallest singular values encode important directions in the data. In case of the Down-Projection matrix of Llama and the Attention-Outpe Output matrices we do not observe an enhanced overlap with the activation covariance matrix, while the smallest singular values of the performance, as the perplexity increases upon their removal. We argue that this is due to the lack of a non-linearity in these layers, however further research is needed to clarify this. Similarly, we argue that the laziness of the Attention-Output matrix is responsible for the lack of overlap with the activations, which needs to be verified in more detail. To further strengthen the validity of our theoretical model, the covariance of the noise present multiple LLM weight matrices should be computed, which is only possible when doing multiple training runs.
matrix are still important for model

---

MP bulk
1.0
0.4
overlap
0.5
0.2
ga
0
X
17
0.0
0.0
0.0
0.2 1/(1 + a.3)
0.4
0.0
0.2
0.4
1/(1 0:3)
Figure 7: Left panel: Smallest singular value (Mmin) as a function of learning parameters a and B where 1/(1 ax3) 0 corresponds to perfect learning. We observe that (Umin) lies outside the Marchenko-Pastur bulk in the case of good learning (small values of 1/(1 + a,3)). The red line corresponds our analytical result Eq (13), while the blue line corresponds
empirical values determined by drawing 100 matrices from the ensemble. Right panel: Overlap of the learned vectors and with the corresponding teacher vectors in Wo function of the learning parameters. The results are averaged over 100 data points. The blue, dashed line depicts the expected overlaps of random vectors of length N. For perfect learning with 1/(1 + Ba)
the overlap of the singular vectors with those of Wo is maximized, while there is a strong decrease in the overlap as (Vmiu) approaches the bulk. The results in both panels are computed for 1, À 0.2, N 2048, and K 512.
7 Conclusion
In this paper, analyzed the singular value spectra and activation covariance matrices of Bert, Pythia, and Llama-8B models. We demonstrated that the spectra deviate from the Marchenko-Pastte distribution, not only for large singular values, but also for small singular values in the case of non- square matrices. The singular vectors corresponding to the singular values outside of the theoretically predicted spectrum
shown be related specific directions in the activations entering that matrix. When removing these singular values in deciles for Llama and Pythia, we found that while the largest singular values the most important decile in all cases, the smallest singular values even be the second most important decile. This importance is established for non non-square matrices, where the smallest singular values fall outside of the Marchenko-Pase
region.
Furthermore, our minittning results shed light on the debate over the relevance of small singular values in LLMs. While removing small singular values after fine-tuning drastically lowers model per- formance, removing them beforehand sometimes has no statistical effect on the final task, explaining some discrepancies in the literature. Using random-matrix model we demonstrate that for singular values well below the MP-bulk, the associated singular vectors are more similar to the teacher vector than for singular values close inside the bulk. Thus, small singular values and associated vectors carry more information than larger ones.
In summary, this work demonstrates the importance of small singular values, showcasing when and where they are important, while additionally providing a theoretical basis for how this phenomenon oceur. We expect these insights to be crucial for future SVD-based pruning algorithms.
Acknowledgments and Disclosure of Funding
Computations for this work were done using resources of the Leipzig University Computing Center.
References
[1] Betty van Aken, Benjamin Winter, Alexander Löser, and Felix A Gers. Visbert: Hidden-state visualizations for transformers. Companion Proceedings of the Web Conference 2020, pages 207-211, 2020.
10

---

[2] Author Anonymous, All code, scripts, and data used in this work are included in Zenodo archive: https://zenodo. org//rooordd/11559855 Zenodo, 2025. doi: 10.528 I/zenodo. 15409885.
[3] Saleh Ashkboos, Maximilian L Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. Slicegpt: Compress large language models by deleting rows and columns. arXiv preprint arXiv:2401. 15024, 2024.
[4] Mona Azadkia. Adaptive estimation of noise variance and matrix estimation via usvt algorithm. arXiv preprint arXh::1801.10111 2018.
[5] Jinho Baik, Gérard Ben Arous, and Sandrine Péché. Phase transition of the largest eigenvalue for nomnull complex sample covariance matrices. 2005.
[6] Nicholas P Baskerville, Jonathan P Keating, Francesco Mezzadri, Joseph Najnudel, and Diego Granziol. Universal characteristics of deep neural network loss surfaces from random matrix theory. Journal of Physies A: Mathematical and Theoretical, 55(49)4 December 2022. ISSN 1751-8121. doi: 10.
[7] Florent Benaych-Georges and Raj Rao Nadakuditi. The eigenvalues and eigenvectors of finite, low rank perturbations of large random matrices. Advances Mathematics, 227(1):444-521,
2011.
[8] Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. The fifth pascal recognizing textual entailment challenge. TAC, 7(8):1, 2009.
[9] Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O' O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pages 2397 2430. PMLR, 2023.
[10] Mireille Capitaine, Catherine Donati-Martnn and Delphine Féral. The largest eigenvalues of finite rank deformation of large wigner matrices: convergence and nonuniversality of the fluctuations. 2009.
[11] Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, and Christopher M De Sa. Quip: 2-bit quantiza- tion of large language models with guarantees. Advances in Neural Information Processing Systems, 36:4396 4429, 2023.
[12] Lenaic Chizat, Edouard Oyallon, and Francis Bach. On lazy training in diffcrentiable program- ming. Advances in neural information processing systems, 32, 2019.
[13] Christopher Clark, Kenton Lee, Ming-Wos Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905 10044, 2019.
[14] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
[15] Carl Eckart and Gale Young. The approximation of one matrix by another of lower rank. Psychometrika, 1(3):211-218, 1936.
[16] Vage Egiazarian, Andrei Panferov, Denis Kuznedelev, Elias Frantar, Artem Babenko, and Dan Alistarh. Extreme compression of large language models via additive quantization. arXiv preprint arXiv:2401.061 18, 2024.
[17] A. Engel and C. Van den Broeck. Statistical Mechanics of Learning. Cambridge University Press, 2001.
[18] Niklas Forner. The singular value spectrum of large random matrices with gaussian noise of power law distributed variances and low-rank perturbation. Master's thesis, University of Leipzig, Leipzig, April 2024.

---

[19] Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and William B Dolan. The third pascal recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL workshop on textual entailment and paraphrasing, pages 1-9, 2007.
[20] Florentin Guth, Brice Ménard, Gaspar Rochette, and Stéphane Mallat. A rainbow in deep network black boxes. arXiv preprini arXiv:2305 185/2, 2023.
[21] R Bar Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. The second pascal recognising textual entailment challenge. In Proceedings of the Second PASCAL Challenges Workshop on Recognising Textual Entailment, volume 7, pages 785-794, 2006.
[22] Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. Ruler: What's s the real context size of your long-context language models? arXiv preprini arXi::2404.06654 2024.
[23] Yen-Chang Hsu, Ting Hua, Sungen Chang, Qian Lou, Yilin Shen, and Hongxia Jin. Language model compression with weighted low-rank factorization. June 2022. doi: 10.48550/ARXIV. 2207.001 12.
[24] Stanislaw Jastrzębski, Zachary Kenton, Devansh Arpit, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amos Storkey. Three factors influencing minima in sgd. arXiv preprint 1rX1v=1711.04623, 2017.
[25] Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. What does bert learn about the structure of language? In ACL 2019-57th Annual Meeting of the Association for Computational Linguistics, 2019.
[26] Jacob Devlin Ming-Wod Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of naacL-HLT, volume page 2, 2019. URL http::::///aacahholoooooooo org/ org/N19- 1423. 142...df22........ sedffutm_meet medimmeimmm emailmmlll 1& utm www.ce
[27] Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Iliè, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. Bloom: A hibb-parameter apennaccess, multilingual language model. 2023.
[28] Noam Levi and Yaron Oz. The underlying scaling laws and universal statistical structure of complex datasets. June 2023. doi: 10.48550/ARXII.2366.11 14975.
[29] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Dangi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A Robustly Optimized BERT Pretraining Approach. Technical report, July 2019. URL http::://axxv..rggg.. abggg// 11692
arXiv:1907.1169: [es] type: article.
[30] V. A. Marčenko and L. A. Pastur. Distribution of eigenvalues for some sets of random matrices. Mathematics of the USSR-SDA VSSR-Sbornik, 1(4):457-483, 1967. ISSN 0025-5734. doi: \url 10. 1070/
0M119677001044AB80000999
[31] Charles H Martin and Michael W Mahoney. Implicit self-regularization in deep neural networks: Evidence from random matrix theory and implications for learning. The Journal of Machine Learning Research, 22(1):779- 2021.
[32] Charles H Martin, Tongsu Peng, and Michael W Mahoney. Predicting trends in the quality of state-of-the-art neural networks without access to training or testing data. Nature Communica- tions, 12(1):4122, 2021.
[33] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv: (xXi::669.00844 2016.
[34] Vardan Papyan. Traces of class/cross-class structure pervade deep learning spectra. Journal of Machine Learning Research, 21( 2020. URL httt:://jmlr.. wwwpilr.rrg//ppaeee prg/papers/v21/
20-933 html
12

---

[35] Jeffrey Penningtonn and Yasaman Bahri. Geometry of neural network loss surfaces via random matrix theory. In International conference on machine learning. pages 2798-28000 PMLR, 2017.
[36]
Ethan Perez, Sam Ringer, Kamilè Lukošiūté, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, Andy Jones, Anna Chen, Ben Mann, Brian Isracl, Bryan Seethor, Cameron McKinnon, Christopher Olah, Da Yan, Daniela Amodei, Dario Amodei, Dawn Drain, Dustin Li, Eli Tran-Johnson, Guro Khundadze, Jackson Kernion, James Landis, Jamie Kerr, Jared Mueller, Jeeyoon Hyun, Joshua Landau, Kamal Ndousse, Landon Goldberg. Liane Lovitt, Martin Lucas, Michael Sellitto, Miranda Zhang, Neerav Kingsland, Nelson Elhage, Nicholas Joseph, Noemí Mercado, Nova DasSarma, Oliver Rausch, Robin Larson, Sam McCandlish, Scott Johnston, Shauna Kravec, Sheer El Showk, Tamera Lanham, Timothy Telleen-Lawton, Tom Brown, Tom Henighan, Tristan Hume, Yuntao Bai, Zac Hatfield-Doddss Jack Clark, Samuel R. Bowman, Amanda Askell, Roger Grosse, Danny Hernande7, Deep Ganguli, Evan Hubinger, Nicholas Schiefer, and Jared Kaplan. Discovering language model behaviors with model--ritter evaluations, 2022. URL https
/ //arxiv. org/ abs/ 2212 09251
[37] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, al. Language models are unsupervised multitask learners. OpenAl blog, 18))9, 2019.
[38] Emily Reif, Ann Yuan, Martin Wattenberg, Fernanda Viegas, Andy Coenen, Adam Pearce, and Been Kim. Visualizing and measuring the geometry of bert. Advances in Neural Information Processing Systems, 32, 2019.
[39] Rajarshi Saha, Naomi Sagan, Varun Srivastava, Andrea J. Goldsmith, and Mert Pilanci. Com- pressing large language models using low rank and low precision decomposition. In A. Glober- son, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak. and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 18981-89011 Curran Associates, Inc., 2024. URL https: htwRaaatppssppprrccddnnnnneeee neurips.....................r... cc tttpcc/cppppco wwcpppppe_ffteeeeeeeeee files/ paper/2024/file/ a 220 24
pdf
[40] Charbel Sakr and Brucek Khailany. Espace: Dimensionality reduction of activations for model compression. Advances in Neural Information Processing Systems, 17:11489-17517, 2024.
[41] Pratyusha Sharma, Jordan T. Ash, and Dipendra Misra. The truth in there: Improving reasoning in language models with layer-selective rank reduction. December 2023. doi: (0.48500/AXXI.2322..5555
[42] Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher 1) Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference empirical methods in natural language processing, pages 1631-1642, 2013.
[43] Max Staats, Matthias Thamm, and Bernd Rosenow. Boundary between noise and information applied to filtering neural network weight matrices. Physical Review E, 108(2):L02230,, August 2023. ISSN 2470-0045, 2470-0053. doi: Btw.://hhyseeE.18..0222222 https:/ :ttts://linkk aps wwtp:///////i/ 1103//1ys/eee.....mm 108. L022302
URL
[44] Matthias Thamm, Max Staats, and Bernd Rosenow. Random matrix analysis of deep neural network weight matrices. Physical Review E, 106(5):05412, November 2022. ISSN 2470-0045, 2470-0053, doi: 10.1 103// 106.054414 URL https://linkk aps org/doi/10 1103/PhyaRevE. 106. 054124]
[45] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint 207.0222077.92888.. 2023.
[46] Yao-Hung Hubert Tsai, Shaojie Bai, Makoto Yamada, Louis-Philippe Morency, and Ruslan Salakhutdinov. Transformer dissection: a unified understanding of transformer's attention via the lens of kernel. arXiv preprint arXiv:1908 11775, 2019.
13

---

[47] Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. Quipt: Even better Ilm quantization with hadamard incoherence and lattice codebooks. arXiv preprint 244.0,990 2024.
[48] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural infor- mattoo processing systems, 30, 2017. URL htRphttt///prroceiinng wwwninuurpp... ccpparrr///ppppppppppppppp 7181 at aenttiinn all
[49] Xin Wang, Yu Zheng, Zhongwei Wan, and Mi Zhang. Svd-IIm: Truncation-aware singula value decomposition for large language model compression. arXiv preprini arXivv2403.07378, 2024.
[50] Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pages 38087-38099, PMLR, 2023.
[51] Zhilin Yang. Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoe Le. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, 32, 2019.
[52] Hao Yu and Jianxin Wu. Compressing transformers: Features are low-rank, but weights are not!, 2023.
[53] Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. The IEEE International Conference on Computer Vision (ICCV), December 2015.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We provide substantial numerical evidence and theoretical explanations regard- ing the claims.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Before we summarize the results in the conclusion, we have a dedicated limitation section.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiitiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
15

---

Answer: [Yes]
Justification: All assumptions for our theory model are stated in the main text. We explain how each theoretical result is obtained in the main text and provide further details in the supplement. We perform specific calculations for a model, but do not provide general theorems which would require proof.
Guidelines:
The answer NA means that the paper does not include theoretical results. the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We define all quantities rigorous and clearly explain from which models these quantities are computed.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
16

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We provide the code that is needed to reproduce the presented empirical results in linked Zenodo archiv.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
wwt.aaammmiioooooooooooooooeeiie for more details. mublic//guiees/CdeSebbiiiss
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
//nips //ips.cc/ cttp:/scccpbblccgcuiceeeeeesc guides/ http://wwllllgliieeeeooeeuaaaaaa
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: The experimental setting is mathematically well defined for both perplexity computations. In case of fine-tuning provide relevant insights and the code.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification:
We provide 30 intervals and standard deviations for our results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
17

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The appendix provides estimates of the used GPU time for our experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https://neers
Answer: [Yes]
Justification: We did not collect data, the only involved persons were well payed for there research. The expected social impact does not conflict with the code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: The paper discusses fundamental aspects of machine learning that could have various implications, none of which is foreseeable now.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
18

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do release datasets models.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite all papers that introduce the models and datasets that we throughout the paper.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
19

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
No new assets are released.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We did not use crowdsourcing, nor did we do research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification:
We did not use crowdsoureing, nor did we do research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
20

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer:[NA]
Justification:
None of the core method development in this research involves LLMs.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: //appr////eurpp..... http://arripsccccooffeerreeeee/e cc//oonferences/2225//LL. for what should or should be described.
21

---

Query
Key
Value
Att.-Out
Up-Proj.
Down-Proj
Pythia Block
p(1)
Interval
0.25
0.00
2
Gate-Proj. 0.0
2.5
2.5
Jama Block 5
30
Interval
0.1
0.0
0.0
0.0
0.0
2.5
2.5 V
Figure 8: Block 5 for Pythia and Llama showcasing the overlap between right singular vectors and eigenvectors of the activation covariance matrix computed on the WikiTeet dataset. We observe that the singular vectors corresponding singular values outside the Marchenko-Pastu region have a significantly increased overlap with the eigenvectors of the covariance matrix as computed in Equation
Query
Key
Value
Att.-Out
Up-Proj.
Down-Proji
Pythia Block
0
Interval
0.25
0.00
2
0
2
2
0
Gate-Proj. 0.0
2.5
0.0
v
2.5
V
Llama Block 15
(22)
0
Interval
0.11
0.0
0.0
2.5
0.0
2.5
0
0.0
2.5
0.0
2.5
0.0
2.5
1
Figure 9: Block 15 for Pythia and Llama showcasing the overlap between right singular vectors and cigenvectors of the activation covariance matrix computed on the WikiText dataset. We observe that the singular vectors corresponding to singular values outside the Marchenko- Pastur region have a significantly increased overlap with the eigenvectors of the covariance matrix as computed in Equation
A Activation Covariance Overlap
We show additional results that demonstrate the generality of the observed overlaps between the right singular vectors of the weight matrix and the eigenvectors of the corresponding activation covariance matrix. The results for block of Llama Pythia are provided Fig. while the results for block 15 are shown in Fig. 9 When singular values deviate from the Marchenko- Pastur distribution, again find significant overlap of the corresponding singular vectors and the eigenvectors of the activation covariance matrix, particularly in the case of small singular values.
22

---

Query
Key
Value
Aut.
Up-Proj.
Down-Proj
0.4
T=
T=0 T=1
T= T=1
T=0 T=1
T= T=1
T
T=1
T=1
0
0.4
2=
T=20
T=
T=
T-
T=
... 1=143
T-143
T=143
T= 143
T=
T=143
400 800
400
800
0
400
800
0
400
800
0
400
800
0
400 800
singular vector index singular vector index singular vector index singolar vestor index singular vector index singular vector index
Figure 10: Time evolution of the overlap between the activation covariance matrix and the weight matrices during pre-training of Pythia (block 10), computed on WikiText. Here, plot the overlap by singular vector index with the one corresponding to the smallest singular value on the left. At T 0, matrices and activations random, giving overlap. By T (1000 gradient updates), large singular-value directions already begin to align with the activation covariance for every weight matrix. However, Attention-Outpu loses this overlap later stages. Notably, the smallest singular values (e.g., in the Down-Projection matrix) only gain significant overlap in the later phases of pretraining.
To further study the outliers, in particular the ones corresponding to smaller singular values, we additionally analyze the temporal evolution. Fig. [0]shows the overlap of the activation covariance matrix and the weight matrix in the training of Pythia for block 10, computed on the WikiText dataset. As expected, there is no overlap at initialization as both the weight and the activation covariance matrix are random. Interestingly, already at T (corresponding to 1000 gradient updates), the model forms a significant overlap with the activation covariance matrix for all matrix types. In case of the Aitettion-uutuu matrix, this overlap is later reduced, while the overlap of Query, Key, and Value increases significantly. which we interpret as repositioning of the information in the weight matrices. The overlap of the singular vectors corresponding small singular values emerges in the later stages of training for the Down-Projection matrix and increases drastically for the Up-Projection matrix. Considering that fine-tuning can significantly impact the smallest singular values, we speculate that the smallest singular values can be related to potentially more sophisticated concepts, which may be learned in the later stages of pre-training.
B Additional Results on Book Corpus
To show that the results presented in the manuscript are of a general form and do not depend on the Wiki'Text dataset, we conduct additional experiments the Book Corpus dataset. Figure confirms that for all non-square matrices, the smallest singular values are more important than some of the larger deciles. In case of the Llama Down-Projection matrix, the smallest decile is even the second most important group. These results can be attributed to the same phenomenon as shown in Figure [2] The overlap between singular vectors and eigenvectors of the activation covariance is very similar to the one observed in the main manuscript, showcasing large overlaps for some of the smaller singular values.
C Special Feature of the Activation Covariance Basis
When computing the overlaps between a particular singular vector and the eigenvectors of the activation covariance basis, defined the quantity
O₂ max, (Ak fz). j (1,2,..,,
(14)
as the maximum coefficient when developing the singular vector the basis of the eigenvectors fj- If one of the 768-4096 coefficients is large, we argue that these two vectors have strong correspondence. natural questions the behavior of the quantity in a different basis. In principal there exist trivial basis for which the overlap is exactly one, i.c. the singular vectors vy themselves.
23

---

Query
Key
Value
Att.-Out
Up-Proj.
Down-
APerplexiyy APerplexity
10
10
10a
109
10a
10a
Pythia
10
104
10
109
10
10
Gate-Proj.
10a
10
10
10a
10a
10a
Jaaaa
10-1
10"
10
10
10
10
1 3 5.9 decile removed decile removed decile removed decile removed decile removed decile removed decile removed
3 5 9
3 579
579
5
3 5
3 337779
Figure 11: Increase perplexity on the Book Corpus dataset for Pythia and Llama when removing deciles of rank-ordered singular values. Singular value deciles are removed from all blocks, but only from a specific matrix type, e.g., all Key matrices. The inset shows the respective spectra averaged over all blocks. Removing the largest singular values substantially affects perplexity for both models and all matrix types, as the matrices change significantly. As in the case WikiText, we find that for non-square matrices, the decile with the smallest singular values is more important than some of the larger deciles.
Query
Key
Value
Att.-Out
Up-Proj.
Down-Proj.
Pythia Block 10
0
0.25
3x Interval
0.00
2
Gate- Paoe-Proj.Pc 0.0
2.5
2.5
Jilamo Block
- 30 Interval
00.11
0.0
0.0
2.5
0.0
2,5
0
0.0
2,5
0.0
2.5
2.5 2
1
11
V
1
Figure 12: Block 10 for Pythia and Llama, showcasing the overlap between right singular vectors and eigenvectors of the activation covariance matrix when the activation covariance matrix is computed based the Book Corpus dataset. We observe that the singular vectors corresponding singular values outside the Marchenko-Passee region have significantly increased overlap with the eigenvec- tors of the covariance matrix.
In Figure 13 we display the quantity
M₁ = max ez), jE (1,
(15)
which just the maximal component of each singular vector. For Pythia, M, is small for the singular vectors corresponding to largest singular values in all cases but the Down-Projection matrix, showcasing that the activation covariance basis special basis in this case. For Llama we observe different picture where the new basis has almost complementary behavior to the activation covariance eigenbasis. While the Attention-Output and Down-Projection matrix Llama not show increased overlaps Os, they appear to be strongly localized, indicated by large M₁. This is an interesting additional observation and may explain the large perplexity increases when removing the smallest singular values of the Down- Down-Projectin Projection matrix.

---

Query
Key
Value
Att.-Out
Up-Proj.
Down-Proj
Pythia Block 10
0.25
Jo Interval
M
0.00
0
2
Gate-Proj. 0.0
v2.5
22.5
Llama Block 10
3a Interval
M2.
0.1
0.0
0.0
2.5
0.0
2.5
1
1/ 0.0
2.5
0.0
2.5
0.0
2,5 V
/
1/
Figure 13: Block 10 for Pythia and Llama, showcasing the overlap between right singular vectors in the basis of standard unit vectors We observe that this basis does not have large maximum overlap with the singular vectors for largest singular values in most cases. However, it appears be complementary to the activation covariance matrix in the case of the Down-Projection matrix and Attention- Outpuut matrix of Llama where we observe strong overlaps.
D Lazy Learning Regime
is possible train neural networks in the lazy regime where the final weights of the trained model are very close to the initial ones [12] By rescaling the input of the softmax function in the final layer by a constant 1
0r = softmax (a(Wr.rr.-1 by.))
we achieve that very small changes in the output logits prior to the softmax function have a large effect on the output after the softmax function. To allow for learning with usual learning rate, the loss is changed to
IW b) = No Σ (k) (n
(16)
incorporate the large differences in the output activations az induced by small weight changes. When training such networks, previous studies (28 44] demonstrated that RMT properties stay intact in the case of lazy learning, where weights remain close their initial random state. However, large outliers (i.e. singular values outside of the MP-boundary [V_ 1+4) (/-,++ often reflect critical learned features leading to the results that models trained in the lazy regime generally perform worse than models trained in the rich or feature learning regime. In Figure 14 we compare the average singular value spectrum all blocks to initial state for the Query and Attention-Outpa matrix. We find that for all three models, the Attention-Outpee matrix has significantly fewer outliers. Considering that there no overlap with the activation covariance matrix either, we speculate that the Attention-Outpu matrix may be trained in the lazy regime.
E
Computational Resources
The computational resources needed to compute the shown results are mostly defined from the perplexity computations. Computing spectra and the activation covariance matrices for models can be estimated as less than 100 GPU hours on V100. To compute the perplexity subparts of the WikiText and BookCorpus dataset several times with different parts of the model removed, we estimate less than 1000 GPU hours.
25

---

Bert
Pythia
Llama
1.0
1.5
Query
Query
Query
Att.-Out
1.0
Att.-Out
1.0
Att.-Out
0.5
0.5
0.5
0.0
0.0
0.0
0
2
2
Figure 14: Averaged Spectra of Bert, Pythia, and Llama compared to the initial distribution of singular values for Query and Attention-Output matrices. We observe that the Onention-Outpon matrix develops substantially fewer outliers than the Query matrices, which we interpret as sign that the Attention-Output matrix may be trained in the lazy regime.
F Details on Random Matrix Model
As described in the main text, we consider the matrix ensemble
P(W) (,A) exp
-N
N
T((WTW)
(17)
2a
Using the expression for the generalization error Eq. (10). we first complete the square
28),(W) - Ir(WTW) Tr [(W T Wo)
(18)
a
with
Wo = oBA
vu
(19)
+ aB
Σ 1 + 300
(20)
Computing the inverse of Σ yields Eq.
We compute the partition function, i.e. the normalization, by performing the Gaussian integral
Z(a, B,A) / dw exp
N
(W Wo) Σ
Wo)
(21)
X (1+ (+,
NK
X2 NB
N
= -2x)
exp
(22)
2 + ag
We can then compute the generalization error
(W))
1 (Z B)
2(1 Ba) + 2(1 + Ba)²
a
2
(23)
N
03
The expectation value of the generalization error is minimal for largest separation (1/(1 + a; 0) of the observed singular value Vmin from the MP bulk of the spectrum of W, suggesting that moving singular values of relevant directions out of the bulk can occur naturally when minimizing the loss during training.
In addition, the expectation value of the small outlier singular value (Vuii)), Eq. 13). can be an- alytically obtained the limit N - 2o, 9 K/N const. For this, the outlier eigenvalue Mmin a a² 3/23/(1 + a.3) (Aa33 (1 + 22 of Σ + W/ Wo is related to (Umin)² via the blue
function
18
xgx(x)
B(y) = +
dr
(24)
-o0 1 - xy
(Vmin)² ) B(1/
(25)
26

---

where (x) = ó(z - a) is the density of Σ eigenvalues in the K + do limit, for which the single outlier contribution negligible. Thus (yy) 1/y qa/((1 -ay) and (Vmin) follows from Eq. (25) as given in Eq. 13)
G Layer Specific Statistics
As first step towards providing more comprehensive statistics for each layer, we present outlier counts for several layers of Llama3-8B in the format [number of left outliers I number of right outliers] in Table 3 We observe that for some matrices related to the attention mechanism (Query, Key, Value, Att.-Output) the number of outliers to the right of the spectral bulk is significantly reduced in layers closer to the output. However, the number outliers to the left the spectrum in rectangular matrices is slightly increased in later layers, with peak in layer 19 in example.
Layer
Query
Key
Value
Att.-Out.
Up-Proj. [75] 189]
Groj.Proj.
Down-Proy Proj.
Sum
Layer
[0 1485]
[356 411]
[113 152]
[23)
[89] 409]
[90 153)
[723 | 3523]
Layer
[01670]
[211 [61]
[111180]
[0 [244
[69 1891
196 294]
[161 [221]
[648 2128)
Layer
[01 769]
[2191
[1191199]
[0 [63)
[1201253]
[470]
[241 347]
[896] 2989]
Layer 14
[01 796]
[2171258
[105 [22]
[01671]
|210]
[2161450]
[196 2245
[245]
[851 2751]
Layer 19
[01
[166 221]
[131191]
[014455]
[184
[205 [311]
[183 172]
[869 1904]
Layer 24
[0] 484]
[220 230]
[139 [114]
[0|2599]
[172 1855
[183 266]
[206 |226]
[920] 1764]
Layer 29
[01375]
[148] 180]
[39132]
[01384]
[93] 205]
[106 1844
[541 | 1672]
Table 3: Singular value outlier count for Llama3-8B in the format [number of left outliers number of
right outliers] with layer resolution.
To also provide metric on the layer-specific importance the smallest singular values, we remove deciles from all matrices in that specific layer and measure the perplexity on the BookCorpus dataset to see whether a trend emerges. The base perplexity without removal is 6.0045, and the results are presented in Table We see that removing singular values from a single layer has only a small effect on perplexity in general, a notable exception being the removal of the largest singular values from Layer O. The second most important decile of Layer 0 is that of the smallest singular values. For the other layers studied, removal of the largest singular values has largest effect on perplexity, in good agreement with the number of outliers in the first layer. Furthermore, layers 19 and 24 appear to be the least important ones with respect to perplexity, which also in good agreement with the number of large outliers.
Layer
Dec. I
Dec. 2
Dec. 3
Dec. 9
Dec. 10
Layer 0
6.3161
6.0259
6.0303
6.2138
62355.6
Layer
6.0405
6.0350
6.0308
6.1828
7.3107
Layer
6.0667
6.0284
6.0329
6.1943
6,8791
Layer
6.0507
6.0371
6.0349
6.1929
6,5998
Layer 19
6.0214
6.0242
6.0318
6.1359
6.4036
Layer 24
6.0373
6.0317
6.0382
6.1070
6,3084
Layer 29
6.1321
6.0629
6.0508
6.1390
6.7912
Table 4: Effect of the removal of singular value deciles from Llama3-8B on the perplexity score,
computed for the BookCorpus dataset.
H Scaling Relations
To analyze the scale which results for small singular values hold, compare the scaling of the number outliers for various model sizes. In general, we know that larger weight matrices display less finite-size effects when comparing them to random matrix theory results derived for infinitely large matrices. To analyze whether there might be a finite size scaling, we compare our three models: Bert with 1 100 parameters (embedding dimension d 768), Pythia with 506M parameters and 1024, and Llama with 8.03B parameters and = 4096.
To quantify the number of singular value outliers, we considered the percentage of singular values smaller than the lower Marchenko-PP Pastur bound (averaged over all layers) in the matrix types where showed that small singular values important: For the Up-Projection matrices find for Bert,
27

---

Llama, Query 25
Llama, Up-Prop 25
eigenvector index
eigenvector index
0.04
0.04
0.02
0.02
0.00
0.00
singular vector index
singular vector index
Figure 15: Cosine similarity, v; ej, between weight singular vectors Vi and activation covariance eigenvectors e, for the Query matrix of layer 25 of the Llama model (left panel) and for the Up- Projection matrix the same layer (right panel). The vectors are rank ordered with respect to the eigenvalues (singular values) such that vectors for the largest values correspond small indices, i.c. to the bottom (left) of the (x) axis.
Pythia, and Llama (38%, 5.9%, 3.2%], and for the Down-Projection matrix [2.1%, 6.5%, 4.7%]. We do not observe clear trend here, which might indicate that the number relevant small singular values scales linearly with model size, i.e., is constant in terms of percentages. However, it is challenging to derive scaling relations from only three models, especially since they are far from identical, and other details of training may be relevant too.
I Reduction in the Space of Activations
In the main text, we address the relevance of small singular values and demonstrate a correspondence between their corresponding singular vectors and the eigenvectors of the activation covariance matrix. As modern reduction algorithms often aim to reduce the number of network parameters by reducing the dimension of the activation space 3 40 it is also crucial to identify exactly which eigenvectors these singular vectors correspond. This would provide additional information about the relevance of specific directions and, therefore, may enable the construction of more precise reduction algorithms.
While a detailed analysis would exceed the scope of this paper. Figure 13 provides an idea of what insights may be gained from such analysis. Here, small indices (bottom left corner) correspond to eigenvectors (y axis) and singular vectors (z: axis) for the largest eigenvalues and singular values, respectively. We observe that in the case of the Query matrix of Llama (left panel), the largest overlaps can be found between the eigenvectors corresponding to the smallest eigenvalues and indicates the
singular vectors corresponding the largest singular values, which is surprising relevance directions in activation space which are small in terms of the variance that they carry. A be found for the Up-Projection matrix of Llama (right panel), where both
similar observation the eigenvectors for smallest and largest eigenvalues have an increased overlap with the singular vectors for the largest singular values. Surprisingly, there is correspondence between vectors for the smallest singular values and smallest eigenvalues, which may indicate that they are very far from random noise. However, further investigations with wider scope necessary to provide clear picture of this phenomenon.
J Additional Results on HumanEval
When removing deciles of singular values (starting with the smallest to the largest ones) from the Down-Projection matrix (Down-Proj.). the Attention-Oulput matrix (Att.-utt),, the Gate-Projection matrix (Gate-Proj.), or the Query matrix, we observe the performances displayed in Table HumanEval benchmark. In excellent agreement with previous results, find that the smallest singular values of the coon-Projection and Gate-PPooeccton matrices are important, which is not the case for the smallest singular values of the quadratic matrices.
28

---

Table 5: Effect of removing deciles of singular values from various weight matrices of Llama3-8B on HumanEval benchmark accuracy. Decile corresponds to the smallest, and Decile 10 largest singular values. The full model reaches 32.32% accuracy. Removing the smallest singular values from the Down-Projection and Gate-Projection matrices causes a large drop performance, confirming their importance. For the quadratic matrices Attention-Oun Output and Query, the smallest singular values appear less critical.
Matrix
Dec. 1
Dec. 2
Dec.
Dec. 4
Dec. 5
Dec. 6
Dec. 7
Dec.
Dec. 9
Dec. 10
Proj-Proj.
0.0%
29.9%
27.4%
20.7%
20.1%
18.3%
17.7%
1.2%
0.0%
0.0%
Att-OOut.
34.8%
32.3%
34.1%
30.5%
36.0%
29.9%
30.5%
28.0%
29.9%
0.0%
Gate-Proj.
28.7%
29.3%
36.0%
27.4%
29.3%
32.3%
29.3%
23.8%
20.7%
0.0%
Query
32.3%
32.3%
31.7%
33.5%
31.1%
33.5%
32.9%
34,8%
30.5%
0.0%
29


Reviews:
[{"review_id": "t5KYUABIgj", "reviewer": "Reviewer_7D7J", "summary": "The paper applies Random Matrix Theory to pretrained Transformer weights, showing that both large and small singular-value outliers—identified against the Marchenko–Pastur baseline—align with salient activation directions and materially affect performance. Experiments on BERT, Pythia and Llama plus a teacher-student RMT model support these claims and highlight the oft-overlooked importance of small singular values for pruning and fine-tuning.", "questions": "No major concerns at this stage.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n\n1. The paper proposes a novel and interpretable use of the Marchenko–Pastur law as a random baseline to dissect learned signal (outliers) from noise (bulk) in the weight spectra of LLMs, and supports it with overlap ( $O_k$ ) measures and pruning experiments.\n2. It rigorously distinguishes between square and rectangular matrices—showing how only the latter can exhibit small singular value outliers—and explains this asymmetry with a rank-1 RMT teacher–student model in Section 6.\n\n### Weaknesses\n\n1. The core experiments are conducted on older models (e.g., BERT, Pythia), which limits the relevance of the findings. Replacing these with more recent state-of-the-art models such as Qwen 2.5/3 would make the results significantly more meaningful.\n2. The impact of pruning order and fine-tuning is not yet validated on modern models, and results on real-world tasks such as GSM-8K or MT-Bench are missing.\n3. The analysis of outlier counts, overlap scores, and pruning sensitivity across layers is limited to selected blocks; more comprehensive layer-wise statistics are needed.\n4. The RMT model in Section 6 makes strong simplifying assumptions (e.g., linear two-layer, Gaussian inputs, rank-1 teacher), which may limit generalizability. Its assumptions and scope should be better discussed and tested numerically.", "was_revised": true, "final_justification": "Most of my concerns have been adequately addressed, and I appreciate the clarifications and new results. I will maintain my original score.", "rebuttal": "We thank the reviewer for their thoughtful and positive report. We appreciate the reviewer's valuable feedback and suggestions.\n\n**1-2)** We agree that enhancing the analysis beyond our currently largest model (Llama3-8B) would be valuable, but such an extension is challenging in the short time period provided for author answers. However, we conducted additional experiments on the GSM-8K benchmark and HumanEval for Llama3-8B to enhance the empirical verification of our findings.\nIn the spirit of our previous experiments, we remove deciles of singular values from all matrices of a specific type while testing the ability of Llama to solve basic math problems. Under 3-shot prompting and without any removal, Llama 3 reaches an accuracy of 43.2% on the GSM8K benchmark in our settings.\n\nWhen removing deciles of singular values (starting with the smallest singular values to the larger ones) from the Down-Projection matrix (Down-Proj.), the Attention-Output matrix (Att.-Out.), the Gate-Projection matrix (Gate-Proj.), or the Query matrix, we observe the following performance on GSM-8K:\n\n\n| Llama3.1-8B   | Dec. 1 | Dec. 2 | Dec. 3 | Dec. 4 | Dec. 5 | Dec. 6 | Dec. 7 | Dec. 8 | Dec. 9 | Dec. 10 |\n|-----------|--------|--------|--------|--------|--------|--------|--------|--------|--------|---------|\n| Down-Proj. | 2%    | 30.2% | 28%   | 27.1% | 26.2% | 22%   | 11%   | 15%   | 5%    | 0%    |\n| Att.-Out.  | 40%   | 41.8  | 39%   | 38.8% | 39.8% | 37.1% | 37.6% | 31.4% | 25.1% | 0%    |\n| Gate-Proj. | 34.1% | 37%   | 39.1% | 39.6% | 38.1% | 38%   | 35.9% | 33.3% | 23.6% | 0%    |\n| Query      | 40.3% | 44.3% | 41.4% | 40.5% | 39.5% | 40.1% | 40.7% | 40.7% | 35.8% | 0%    |\n\nHere, decile one corresponds to the smallest and decile ten to the largest singular values. For the Down-Projection matrix, the results show that the smallest singular values (Dec. 1) are the second most important decile, which is in excellent agreement with our previous results for this rectangular matrix (compare Fig. 5 in the manuscript). Looking at the quadratic Attention-Output matrix, the smallest singular values are one of the least important deciles, in agreement with theoretical expectations and previous results.\n\nThe smallest singular values of the non-quadratic Gate-Projection matrix are the fourth important decile, in excellent agreement with the results of Fig. 5. For the quadratic Query matrix, only the two largest deciles appear to be important.\n\nA very similar pattern can be observed if we evaluate the capabilities of Llama on the code generation benchmark HumanEval, where the full model scores 32.32% in our settings:\n\n| Llama3.1-8B   | Dec. 1 | Dec. 2 | Dec. 3 | Dec. 4 | Dec. 5 | Dec. 6 | Dec. 7 | Dec. 8 | Dec. 9 | Dec. 10 |\n|--------------|--------|--------|--------|--------|--------|--------|--------|--------|--------|---------|\n| Down-Proj.   | 0.0%   | 29.9%  | 27.4%  | 20.7%  | 20.1%  | 18.3%  | 17.7%  | 1.2%   | 0.0%   | 0.0%    |\n| Att.-Out.    | 34.8%  | 32.3%  | 34.1%  | 30.5%  | 36.0%  | 29.9%  | 30.5%  | 28.0%  | 29.9%  | 0.0%    |\n| Gate-Proj.   | 28.7%  | 29.3%  | 36.0%  | 27.4%  | 29.3%  | 32.3%  | 29.3%  | 23.8%  | 20.7%  | 0.0%    |\n| Query        | 32.3%  | 32.3%  | 31.7%  | 33.5%  | 31.1%  | 33.5%  | 32.9%  | 34.8%  | 30.5%  | 0.0%    |\n\nWe again find that the smallest singular values of the Down-Projection and  Gate-Projection matrix are surprisingly important, which is not the case for the quadratic matrices. \n\nWe suggest to add these results to the appendix of the manuscript, together with results for the remaining matrix types. We thank the reviewer for their suggestion to improve our manuscript with the inclusion of this additional benchmark.\n\n\n**3)** We agree with the reviewer that more comprehensive statistics for each layer would be an interesting addition to the paper, as our current analysis does not consider these differences quantitatively, but only qualitatively. As a first step towards this goal, we provide outlier counts for several layers of Llama3-8B in the format [#LeftOutlier | #RightOutliers] :\n\n\n| Layer   | Query        | Key           | Value         | Att.-Out.     | Up-Proj.      | Gate-Proj.     | Down-Proj.     | Sum             |\n|---------|--------------|---------------|---------------|---------------|---------------|----------------|----------------|-----------------|\n| Layer 0 | [0 \\| 1485]  | [356 \\| 412]  | [113 \\| 152]  | [0 \\| 723]    | [75 \\| 189]   | [89 \\| 409]    | [90 \\| 153]    | [723 \\| 3523]   |\n| Layer 4 | [0 \\| 670]   | [211 \\| 261]  | [111 \\| 80]   | [0 \\| 524]    | [69 \\| 78]    | [96 \\| 294]    | [161 \\| 221]   | [648 \\| 2128]   |\n| Layer 9 | [0 \\| 769]   | [219 \\| 288]  | [119 \\| 199]  | [0 \\| 663]    | [120 \\| 253]  | [197 \\| 470]   | [241 \\| 347]   | [896 \\| 2989]   |\n| Layer 14| [0 \\| 796]   | [217 \\| 258]  | [105 \\| 121]  | [0 \\| 671]    | [117 \\| 210]  | [216 \\| 450]   | [196 \\| 245]   | [851 \\| 2751]   |\n| Layer 19| [0 \\| 526]   | [166 \\| 221]  | [131 \\| 91]   | [0 \\| 415]    | [184 \\| 168]  | [205 \\| 311]   | [183 \\| 172]   | [869 \\| 1904]   |\n| Layer 24| [0 \\| 484]   | [220 \\| 230]  | [139 \\| 114]  | [0 \\| 259]    | [172 \\| 185]  | [183 \\| 266]   | [206 \\| 226]   | [920 \\| 1764]   |\n| Layer 29| [0 \\| 375]   | [148 \\| 180]  | [39 \\| 32]    | [0 \\| 384]    | [93 \\| 205]   | [106 \\| 284]   | [155 \\| 212]   | [541 \\| 1672]   |\n\nIt seems that for some matrices related to the attention mechanism (Query, Key, Value, Att.-Output), the number of outliers to the right of the spectral bulk is significantly reduced in layers closer to the output. However, the number of outliers to the left of the spectrum in rectangular matrices slightly increases in later layers, with a peak in layer 19 in our example.\n\nStudies on the overall relation between depth and relevance can be found in the literature, see e.g. [Chen et al., Streamlining redundant layers to compress large language models, ICLR (2025)]. However, the importance of small singular values has not been studied so far. To provide a metric on the importance of the smallest singular values in these layers, we remove deciles from all matrices in that specific layer and measure the perplexity on the BookCorpus dataset to see whether a trend emerges. The base perplexity without removal is 6.0045.\n\n| Layer   | Dec. 1  | Dec. 2  | Dec. 3  | Dec. 9  | Dec. 10 |\n|---------|---------|---------|---------|---------|---------|\n| Layer 0 | 6.3161  | 6.0259  | 6.0303  | 6.2138  | 62355.6 |\n| Layer 4 | 6.0405  | 6.0350  | 6.0308  | 6.1828  | 7.3107  |\n| Layer 9 | 6.0667  | 6.0284  | 6.0329  | 6.1943  | 6.8791  |\n| Layer 14| 6.0507  | 6.0371  | 6.0349  | 6.1929  | 6.5998  |\n| Layer 19| 6.0214  | 6.0242  | 6.0318  | 6.1359  | 6.4036  |\n| Layer 24| 6.0373  | 6.0317  | 6.0382  | 6.1070  | 6.3084  |\n| Layer 29| 6.1321  | 6.0629  | 6.0508  | 6.1390  | 6.7912  |\n\nWe see that removing singular values from only one layer has only a small effect on perplexity in general, a notable exception being the removal of the largest singular values from Layer 0. The second most important decile of Layer 0 are the smallest singular values. For the other layers studied, removal of the largest singular values has the largest effect on perplexity,  in good agreement with the number of outliers in the first layer. Furthermore, layers 19 and 24 appear to be the least important ones with respect to perplexity, which is also in good agreement with the number of large outliers.\n\nWe suggest adding the above layer-wise results to the appendix. \n\n\n**4)**  We thank the reviewer for pointing out that the limitations of the theoretical model should be discussed more thoroughly in our manuscript. The reviewer is right that our model uses simplifying assumptions.\nOur goal is to provide a minimal model that captures a mechanism for how small outliers can be produced during training. To us, it was initially very counterintuitive that gradient descent would produce these small outliers, as the general wisdom is that they are negligible. Unfortunately, we are unable to provide numerical results for non-linear activations within the time window for author-reviewer discussions.\n\nHowever, we will add the following paragraph to section 6 about the minimal RMT model:\nThe aim of this section is to provide a minimal RMT model for the occurrence of small singular value outliers. For this, we consider a simple two-layer linear student model, trained on Gaussian inputs, and labels produced by a linear rank-1 teacher. The effects of non-trivial correlations in the noise would depend on whether the noise has predominantly small or large covariance in the direction of the outlier."}, {"review_id": "EHzVFPkQVv", "reviewer": "Reviewer_XvMa", "summary": "By analyzing weight matrices through Random Matrix Theory, the study finds that both the largest and smallest singular values deviate from randomness, indicating they encode meaningful information. Removing non-random singular values greatly impacts model performance, highlighting the overlooked importance of small singular values and offering practical insights for compressing large language models.", "questions": "1) I noticed that since small singular values have a significant impact, I speculate that amplifying them will be beneficial for the model's learning, and this seems to be related to the muon optimizer [1]. What is your opinion on this?\n\n2) About evaluation, multiple choice and PPLs are often misleading, and some studies have revealed that many components in the network contribute very little to multiple choice questions[2].  The impact on generating questions is quite different [3].  Therefore, in addition to needle in a haystack, I also suggest that the author measure more benchmarks for generative tasks.\n\n\n\n\n[1] https://kellerjordan.github.io/posts/muon/\n\n[2] Understanding the skill gap in recurrent language models: The role of the gather-and-aggregate mechanism.\n\n[3] Shortgpt: Layers in large language models are more redundant than you expect.", "limitations": "See weakness.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n1. Theoretical analysis is solid. The paper explain the phenomenon that small singular values that occur in practice also have a significant impact.\n\n2.  It has certain practical significance and provides insight for designing a more suitable pruning or LLM optimization scheme in the future.\n\n3. The article is well presented and easy to read.\n\n\n\n\n**Weakness**\n1.  Evaluation is limited. We know that PPL is often inaccurate, especially in long context tasks [1][2], so I strongly recommend the authors to evaluate tasks like needle in a haystack. \n\n[1] Can perplexity reflect large language model’s ability in long text understanding?  ICLR 2024.\n\n[2] Base of RoPE Bounds Context Length. NeurlPS 2024.", "was_revised": true, "final_justification": "I am inclined to accept, but I hope to include evaluations related to long-term contextual abilities.", "rebuttal": "We thank the reviewer for their thoughtful and positive report. We appreciate the reviewer's valuable feedback and suggestions. In the following, we answer the reviewer's questions in detail:\n\n**1)** The reviewer pointed out another potential application where our theoretical approach to small singular values could be fruitful. To our understanding, the success of Adam compared to SDG with momentum lies mainly in rescaling the update size based on variances, allowing for faster learning along almost flat directions. However, Adam is initialized without amplification and needs a number of update steps to find flat but relevant directions. Under the assumption that small singular values of the gradient matrix are more relevant than their size suggests (something we did not test but appears plausible, considering our results), Muon would then effectively increase the learning speed in these important directions from the start, where Adam takes a long time to figure them out. This could explain the success of Muon in speed learning scenarios.\n\n\n**2)** We agree with the reviewer that evaluation tasks like 'needle in a haystack' could further improve the analysis. Considering that current implementations of this task are testing modern models via their respective APIs, we did not implement this task for our models. Analyzing such commercially available models would be interesting; however, for our analysis to work, we need access to the weights of the model and need to modify them. If the reviewer is aware of a hugginface model-compatible implementation of this task, we would gladly include it.\n\nTo strengthen our analysis, we conducted additional experiments on the GSM-8K benchmark and HumanEval for Llama3-8B to enhance the empirical verification of our findings.\nIn the spirit of our previous experiments, we remove deciles of singular values from all matrices of a specific type while testing the ability of Llama to solve basic math problems. Under 3-shot prompting and without any removal, Llama 3 reaches an accuracy of 43.2% on the GSM8K benchmark in our settings.\n\nWhen removing deciles of singular values (starting with the smallest singular values to the larger ones) from the Down-Projection matrix (Down-Proj.), the Attention-Output matrix (Att.-Out.), the Gate-Projection matrix (Gate-Proj.), or the Query matrix, we observe the following performance on GSM-8K:\n\n\n| Llama3.1-8B   | Dec. 1 | Dec. 2 | Dec. 3 | Dec. 4 | Dec. 5 | Dec. 6 | Dec. 7 | Dec. 8 | Dec. 9 | Dec. 10 |\n|-----------|--------|--------|--------|--------|--------|--------|--------|--------|--------|---------|\n| Down-Proj. | 2%    | 30.2% | 28%   | 27.1% | 26.2% | 22%   | 11%   | 15%   | 5%    | 0%    |\n| Att.-Out.  | 40%   | 41.8  | 39%   | 38.8% | 39.8% | 37.1% | 37.6% | 31.4% | 25.1% | 0%    |\n| Gate-Proj. | 34.1% | 37%   | 39.1% | 39.6% | 38.1% | 38%   | 35.9% | 33.3% | 23.6% | 0%    |\n| Query      | 40.3% | 44.3% | 41.4% | 40.5% | 39.5% | 40.1% | 40.7% | 40.7% | 35.8% | 0%    |\n\nHere, decile one corresponds to the smallest and decile ten to the largest singular values. For the Down-Projection matrix, the results show that the smallest singular values (Dec. 1) are the second most important decile, which is in excellent agreement with our previous results for this rectangular matrix (compare Fig. 5 in the manuscript). Looking at the quadratic Attention-Output matrix, the smallest singular values are one of the least important deciles, in agreement with theoretical expectations and previous results.\n\nThe smallest singular values of the non-quadratic Gate-Projection matrix are the fourth important decile, in excellent agreement with the results of Fig. 5. For the quadratic Query matrix, only the two largest deciles appear to be important.\n\nA very similar pattern can be observed if we evaluate the capabilities of Llama on the code generation benchmark HumanEval, where the full model scores 32.32% in our settings.\n\n| Llama3.1-8B   | Dec. 1 | Dec. 2 | Dec. 3 | Dec. 4 | Dec. 5 | Dec. 6 | Dec. 7 | Dec. 8 | Dec. 9 | Dec. 10 |\n|--------------|--------|--------|--------|--------|--------|--------|--------|--------|--------|---------|\n| Down-Proj.   | 0.0%   | 29.9%  | 27.4%  | 20.7%  | 20.1%  | 18.3%  | 17.7%  | 1.2%   | 0.0%   | 0.0%    |\n| Att.-Out.    | 34.8%  | 32.3%  | 34.1%  | 30.5%  | 36.0%  | 29.9%  | 30.5%  | 28.0%  | 29.9%  | 0.0%    |\n| Gate-Proj.   | 28.7%  | 29.3%  | 36.0%  | 27.4%  | 29.3%  | 32.3%  | 29.3%  | 23.8%  | 20.7%  | 0.0%    |\n| Query        | 32.3%  | 32.3%  | 31.7%  | 33.5%  | 31.1%  | 33.5%  | 32.9%  | 34.8%  | 30.5%  | 0.0%    |\n\nWe again find that the smallest singular values of the Down-Projection and  Gate-Projection matrix are surprisingly important, which is not the case for the quadratic matrices. \n\nWe suggest to add these results to the appendix of the manuscript, together with results for the remaining matrix types (once they are ready). We thank the reviewer for their suggestion to improve our manuscript with the inclusion of this additional benchmark.\n\n\nWe believe we have thoroughly addressed all of your concerns, and would be grateful if you could let us know in case anything remains unclear or if further clarification is needed."}, {"review_id": "5G3Os7Wr1N", "reviewer": "Reviewer_2hdo", "summary": "This paper revisits transformer models via random matrix theory. Empirically, they found out that while small singular values typically represents \"outliers\", removing those information significantly impacted model performance, even more so than removing bulk values.\nAfter fine-tuning, the bottom decile can become the third–most influential spectral region. \nTo explain this, the paper proposes a linear teacher–student RMT model in which noise covariance induces small‐value outliers whose vectors carry more task‐relevant information the further they lie from the bulk.", "questions": "1. Can the authors provide further explanation on the connection to lazy training? (Appendix D) As to my knowledge, lazy training only seems to happen when one has infinitely wide networks.\n2. How does the phenomenon scale with model size? Are small-value outliers equally important in different sizes of models?", "limitations": "Yes, it is right before the conclusion.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strength\n1. I think the insights from this paper is novel. This paper highlights an overlooked spectral regime (small singular values) as carriers of learned features, overturning the conventional emphasis on large‐value outliers alone.\n2. I think the paper carries out several interesting potential directions in transformer theories, like a new perspective of RMT, analyzing small singular values etc.\n3. While the paper mainly present empirical findings, I do think it provides practical guidance, theoretical implications to the ML community.\n4. The paper is very well written, easy to follow for readers.\n\n## Weakness\n1. To my knowledge, the RMT model used in the paper assumes the inputs to follow Gaussian distribution, and with linear attention. The latter assumption does not seem to be general to all transformer variants.\n2. It would be nice to see some further explanation on the connection to \"lazy training\". So far, it is not clear to me why \"lazy training\" would lead to their empirical results.", "was_revised": true, "final_justification": "The authors have addressed my concerns, I remain my score.", "rebuttal": "We thank the reviewer for their thoughtful and positive report. We appreciate the reviewer's valuable feedback and suggestions. In the following, we answer the reviewer's questions in detail:\n\n**1.** In the lazy training regime, the weights change only very little during training, such that the model depends linearly on the weights around the initialization. This phenomenon is therefore often studied in the Neural Tangent Kernel (NTK) regime: The reviewer is right that this regime emerges naturally in the infinite width limit. However, the lazy regime can also be realized in finite networks.\n\n\nOne can manually tune to the lazy regime by rescaling the input of the softmax function in the final layer by a constant $\\alpha>1$\n$$\na_L = \\text{softmax} \\left( \\alpha ( W_L a_{L-1} + b_L ) \\right) \\ ,\n$$\nwhile also scaling down the loss by a factor $\\( 1/\\alpha^2 \\)$:\n$$\nl(W, b) = -\\frac{1}{N \\alpha^2} \\sum_{k=1}^N y^{(k)} \\cdot \\ln\\left( a_{\\rm out}^{(k)} \\right)\n$$\n(see [Chizat et al., NeurIPS 32 (2019)]).\nThese changes have the effect that small updates to the weights can have a large impact on the output activations. Additionally, it shows that the learning regime can be tuned continuously from feature to lazy learning.\n\nDepending on model architecture, training data, and many more parameters, different layers of a model may effectively be trained in different learning regimes, even when not manually applying a rescaling. For example, in simple MLP networks on Cifar-10, layers closer to the output tend to be more lazy than layers closer to the input. The learning regime of a weight matrix can be revealed by an RMT analysis, as weights close to initialization follow RMT predictions, such as the Marchenko-Pastur distribution, more closely. This is what we discuss in Appendix D and Fig. 14.\n\n**2.** Scaling relations: When going to larger model sizes, we typically see larger weight matrices that display less finite-size effects. To analyze whether there might be a size scaling, we compare our three models, Bert with 110M parameters (embedding dimension $d=768$), Pythia with 506M parameters and $d=1024$, and Llama with 8.03B parameters and $d=4096$. A first observation is that for all three models, the small singular values in the Up-Projection and Down-Projection layers are important. \n\nTo quantify the number of outliers, we considered the percentage of singular values smaller than the lower Marchenko-Pastur bound (averaged over all layers):\n- Bert: Up-Proj:  $3.8\\\\%$, Down-Proj: $2.1\\\\%$\n- Pythia: Up-Proj: $5.9\\\\%$, Down-Proj: $6.5\\\\%$\n- Llama: Up-Proj: $3.2\\\\%$, Down-Proj: $4.7\\\\%$\n\nWe do not observe a clear trend here, which might indicate that the number of relevant small singular values scales linearly with model size, i.e., stays identical in terms of percentages.\n\nFor Pythia and Llama, we consider the model's perplexity on WikiText. Quantitatively, removing the decile with the smallest singular values (decile 1) results in a comparable perplexity increase relative to the perplexity increase when removing the second largest decile (decile 9):  \n- Pythia: Up-Proj: $\\Delta PPL(1) / \\Delta PPL(9) = 0.72$, Down-Proj: $0.88$\n- Llama: Up-Proj: $\\Delta PPL(1) / \\Delta PPL(9) = 0.89$, Down-Proj: $1.02$\n\nThis suggests that the relevance stays approximately constant, similar to the percentage outside of the Marchenko-Pastur distribution. However, it is challenging to derive scaling relations from only three models, especially since they are far from identical, and other parameters of training might come into play. We will add these numbers to the appendix with a remark that the results show no clear trend, which indicates that this phenomenon persists over several model scales."}, {"review_id": "nhumCNgDV4", "reviewer": "Reviewer_iFU6", "summary": "This paper uses the MP law to estimate what a typical distribution of singular values should look like. For trained matrices, outliers arise from the typical distribution, on both left and right side (small and large values) for non-square matrices. This reveals that there exist some small singular values that are of relevance to the training of the matrix - thus truncating singular values by magnitude (as is usually done for frobenius norm minimization, per Ben-Eckart) may not be a good idea.\n\nUsing activation covariance eigenvectors, correlations are measured with weight singular vectors. It turns out that indeed, singular vectors associated with those singular values falling outside the MP law do indeed excite activation eigenvectors.", "questions": "- It is interesting to study RMT applied to weight matrices. But what about activations, there are much more redundancies in activations which can be better exploited to compress a model using matrix factorization, as was shown in [1]. Can we perform a similar study for activations?\n[1]: \nSakr, Charbel, and Brucek Khailany. \"ESPACE: Dimensionality Reduction of Activations for Model Compression.\" Neurips 2024\n\n- For the spectral overlaps in Fig 2 & 3, it seems rather qualitative, and more like the authors are trying to to convince us that something is there by telling us it's there (namely by switching between blue and red colors). However, it does seem like some vectors in the blue region do exhibit strong overlap with the activation eigenvectors. I wonder if there is a more proper way to visualize these (maybe log plots). And please comment on why some blue singular vectors have such strong overlap.\n\n- In the experimental setup of section 5, why are deciles used? Why can't we just do the truncation as per the description of the earlier section, i.e., take out the singular values that supposedly arise from the randomness, and leave the outliers that supposedly came about during training? Such a 1-to-1 experiment correlating with the earlier theory would be interesting.\n\n\n- Not a question but the last sentence of the abstract is repeated twice.", "limitations": "Yes, it's there on pages 8-9 just before the conclusion.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Nice paper giving a fresh perspective into singular values decomposition of weights. I like the departure from the usual frobenius norm minimization and the fact that the study compares trained matrices to random ones and essentially asks what different because such differences are likely what makes the model strong. Good introduction of interesting concepts.\n\nIn terms of weaknesses, there are two main issues:\n1) the paper does not showcase empirically how strong the proposed method derived from the study can be at compressing models. It would be good to see comparisons of the method to SOTA techniques in tensor decomposition such as ESPACE, BLAST, etc..\n2) Some minor issues that need answering. Please refer to the \"Questions\" part of the review.", "was_revised": true, "final_justification": "The authors have provided qualitative replies to my questions. I maintain my rating of borderline accept.", "rebuttal": "We thank the reviewer for their thoughtful and positive report. We appreciate the reviewer's valuable feedback and suggestions. In the following, we answer the reviewer's questions in detail:\n\n**1)** Relevance of eigenvectors corresponding to small eigenvalues in the activation covariance matrix: We thank the reviewer for pointing out this important topic. We have analyzed eigenvalue spectra of activation covariance matrices, but not included them in the manuscript. As in the case of weight matrices, we found outliers to the left of the spectrum, i.e., eigenvalues that are smaller than the bulk. Furthermore, we observed that the singular vectors of small singular values of weight matrices often correspond to eigenvectors of the activation covariance matrix with small eigenvalues. \nThis matching could indicate that a similar analysis for the activations could indeed be relevant to compressing algorithms that rely on eigenvalues of the activation covariance matrix. \n\nIn [Ashkboos et al., SliceGPT: Compress Large Language Models by Deleting Rows and Columns, ICLR (2024)], the authors project into a lower-dimensional activation space spanned only by the directions of large eigenvectors of the activation covariance matrix, which could be non-optimal in our understanding. The mentioned paper [1] improves upon this by letting the network itself choose the important directions during training by having a static projection matrix. However, the mentioned method [1] cannot reduce the size of already trained networks. In cases like [Ashkboos et al., ICLR (2024)], our results may hence add an important piece of information and may lead to increased performance of pruned networks. \n\nWe are happy to add a plot in the final version of the manuscript that illustrates this relation of small singular vectors to small eigenvectors of the activation covariance matrix.\n\n**2)** We thank the reviewer for their question and would like to argue that the results shown in Figures 2 and 3 do go beyond a pure visualization: Figure 2 and Figure 3 display the maximum overlap $\\max_k (\\textbf{s}_i \\cdot \\textbf{e}_k )$ for each singular vector $\\textbf{s}_i$ with the eigenvectors $\\textbf{e}_k$ of the activation covariance matrix, which we interpret as a singular vector to feature correspondence. To give meaning to these plots, we indicate the 3-sigma interval, e.g., the interval in which we expect 99.7\\% of the data to fall, if the singular vectors were random. In Figure 3, we observe that the number of blue dots outside this region is indeed in a similar range as statistically expected for a completely random process (2-12 are statistically expected depending on the number of singular vectors, i.e., the matrix size).\nFor the red dots, around 2/3 of the datapoints fall outside of this region, which is statistically highly significant, i.e., cannot be explained by a random chance.\n\nIn the case of Figure 2, we find seven blue dots outside the region for Bert, which is more than the statistically expected 2.3 dots. We interpret these deviations as a sign that even in the bulk part, we may find some learned information. However, the ratio of statistically significant points to the number of total points is extremely low when compared to the singular values outside of the spectrum.  For Pythia and Llama, the number of outliers is again in the statistically expected range of ~3 for Pythia and ~12 for Llama.\n\nWe agree with the reviewer that a log scale could have been chosen, but we tested both and found that a linear scale provides a cleaner view, considering that the range of displayed values only spans roughly one order of magnitude (0.03-0.3).\n\n\n**3)** We agree with the reviewer that other test designs could have been chosen, but we chose our design due to the following reasons: The current setup of removing deciles is easy to understand, and it clearly shows the point that small singular values are more important than larger ones in the case of non-square matrices, as we compare the removal of sets of equal size.\n\nMore importantly, in an exploratory study, we have found that removing all singular values that we classify as random (keeping only the outliers to the left and right of the Marchenko-Pastur bulk) would lead to a significant reduction of network performance. One can ask the question: why does the performance decrease if one only removes randomness? This can happen if the network is not fully in the feature learning regime. One can demonstrate this effect in a toy model: when training a network with two hidden layers, with the weights between the input and the first hidden layer completely random and frozen during training. Since the trained weights depend on the random transformation of inputs due to the frozen and random layer, removing singular values from this random layer reduces the network performance significantly. For this reason, a pruning scheme based on the findings presented in our manuscript would need to be more involved: one would need to distinguish between weight matrices close to lazy training (e.g., with only a few outliers) and weight matrices in the feature learning regime. Due to these complications, we have decided not to include pruning experiments in the present manuscript. Nevertheless, we agree with the reviewer that further experiments can be conducted that investigate the phenomenon further.\n\n\n**4)** We thank the reviewer for pointing out the duplicated sentence and will remove it in a revised version.\n\n\nWe believe we have thoroughly addressed all of your concerns, and would be grateful if you could let us know in case anything remains unclear or if further clarification is needed."}]

Author rebuttal:

Meta-review: This paper analyzes the singular value spectrum of pretrained language models using random matrix theory and finds that both large and small singular values are outliers. The analysis is further validated by removing singular values from different weight matrices from pretrained language models. All reviewers agree that the paper offers novel insights and perspectives, and the authors have successfully addressed most of their concerns during the rebuttal. Therefore, I recommend acceptance in consensus with the reviewers.

The AC suggests that the authors incorporate the reviewers’ suggestions in their camera-ready version, such as adding evaluations on long-term contextual abilities.
Decision: Accept (poster)