Eigenspectrum Analysis of Neural Networks without Aspect Ratio Bias
Authors: Yuanzhe Hu
Kinshuk Goel
Vlad Killiakov
Yaoqing Yang
Venue: ICML 2025 poster (2025)
Keywords: LLM Pruning
Weight Matrix Analysis
Spectral Analysis
Marchenko-Pastur Law
Hyperparameter Tuning
Primary area: deep_learning->algorithms
TL;DR: We reduce the bias of eigenspectrum diagnosis using a method based on matrix subsampling

Abstract: Diagnosing deep neural networks (DNNs) by analyzing the eigenspectrum of their weights has been an active area of research in recent years. One of the main approaches involves measuring the heavytailness of the empirical spectral densities (ESDs) of weight matrices. This analysis has been shown to provide insights to help diagnose whether a model is well-trained or undertrained, and has been used to guide training methods involving layer-wise hyperparameter assignment. In this paper, we address an often-overlooked challenge in estimating the heavytailness of these ESDs: the impact of the aspect ratio of weight matrices. We demonstrate that matrices of varying sizes (and aspect ratios) introduce a non-negligible bias in estimating the heavytailness of ESDs, leading to inaccurate model diagnosis and layer-wise hyperparameter assignment. To overcome this challenge, we propose FARMS (Fixed-Aspect-Ratio Matrix Subsampling), a method that normalizes the weight matrices by subsampling submatrices with a fixed aspect ratio. Instead of measuring the heavytailness of the original ESD, we measure the average ESD of these subsampled submatrices. We show that this method effectively mitigates the aspect ratio bias. We validate our approach across various optimization techniques and application domains that involve eigenspectrum analysis of weights, including image classification in computer vision (CV) models, scientific machine learning (SciML) model training, and large language model (LLM) pruning. Our results show that despite its simplicity, FARMS uniformly improves the accuracy of eigenspectrum analysis while enabling more effective layer-wise hyperparameter assignment. In one of the LLM pruning experiments, FARMS reduces the perplexity of the LLaMA-7B model by 17.3% when compared with state-of-the-art methods.

Eigenspectrum Analysis of Neural Networks without Aspect Ratio Bias
Yuanzhe Hu' Kinshuk Goel 23
Vlad Killiakov Yaoqing Yang"
Abstract
while enabling more effective layer-wise hyperpa- rameter assignment. In one of the LLM pruning experiments, FARMS reduces the perplexity of the LLaMA-7B model by 17.3% when compared with state-of-the-art methods.
Diagnosing deep neural networks (DNNs) by an- alyzing the eigenspeetrum their weights has been an active area of research in recent years. One of the main approaches involves measuring the heavytailness of the empirical spectral densi- ties (ESDs) of weight matrices. This analysis has been shown to provide insights to help diagnose whether a model well-trained or undertrained, and has been used to guide training methods in- volving layer-wise hyperparameter assignment. In this paper, address often-overlooked challenge in estimating the heavytailness of these ESDs: the impact of the aspect ratio of weight matrices. We demonstrate that matrices of vary- ing sizes (and aspect ratios) introduce a non- negligible bias in estimating the heavytailness of ESDs, leading to inaccurate model diagnosis and layer-wise hyperparameter assignment. To over- come this challenge, we propose FARMS (Fixed- Aspect-Ratio Matrix Subsampling), method that normalizes the weight matrices by subsampling submatrices with a fixed aspect ratio. Instead of measuring the heavytailness of the original ESD, we measure the average ESD of these subsampled submatrices. We show that this method effectively mitigates the aspect ratio bias. We validate our approach across various optimization techniques and application domains that involve eigenspec- trum analysis of weights, including image classifi- cation in computer vision (CV) models, scientific machine learning (SciML) model training, and large language model (LLM) pruning. Our results show that despite its simplicity, FARMS uniformly improves the accuracy of eigenspectrum analysis
1. Introduction
Random Matrix Theory (RMT) has long been used a foundational tool in multiple research domains (Guhr et al., 1998; Tulino & Verdúú 2004; Bai & Silverstein, 2010; Tao, 2012; Nadakuditi & Newman, 2012) and has now been applied to providing precise characterizations of neural net- works (NNs) (Pennington & Worah, 2017; Dobriban & Wa- ger, 2018; Adlam & Pennington, 2020; Mei Montanari, 2022; Hastie al., 2022; Hu & Lu, 2022: Ba al., 2022; Couillet & Liao, 2022; Derezinski et al., 2020). Among several emerging topics in this field, Heavy-Tailed Self- Regularization (HT-SR) (Martin & Mahoney, 2021; Ma- honey & Martin, 2019; Martin et al., 2021) Theory has been gaining significant attention. Unlike conventional RMT studies, HT-SR Theory focuses on studying weight matrices with strongly correlated elements, typical characteristic of well-trained, practical DNNs. These strong correlations lead to heavy-tailed (HTed) ESDs (Martin & Mahoney, 2021). Analyzing these HT ESDs provides valuable insights into the quality of trained models, enabling both model diag- nosties and improvements in model training. For exam- plc, Martin & Mahoney (2021) show that one can locate undertrained layers by finding weight matrices with less HTedd ESDs, which provides useful metrics for model se- lection and ranking (Martin al., 2021; Yang et al., 2023). Furthermore, one can use a larger learning rate on the un- dertrained layers (Zhou et al., 2024). which provides an em- pirically successful training method to balance these layers with other, more well-trained ones. This technique has been developed into a general approach improve test accuracy, training efficiency, and model interpretability in various ap- plications, such image classification (Zhou et al., 2024), LLM model pruning (Lu et al., 2024), and SciML model training and fine-tuning (Liu et al., 2024; 2025b).
'eparrmment of Computer Science and Engineering, Univer- sity of California, San Diego, CA, USA 'Department Com- puter Science, Dartmouth College, NH, USA 'Department of Computer Science & Engineering, SRM Institute of Science & Technology, India 'Independent Researcher, University of Cal- iforiia, Berkeley, CA. USA. Correspondence to: Yuanzhe Hu <yuanzhe. hu @darmmoth..ddu.....
The core of the HT-SR theory lies in the measurement of the heavytailness of ESDs. In this context, we identified an often-overlooked issue in prior studies: weight matrices
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Eigenspectrum Analysis of Neural Networks without Aspect Ratio Bias
Our Method
Sampling for Woll-Trainod Layer
More Heavy Tailed
Accuratc Alpla Values
Exsmple Application: Leanring Assignment
Well Oprimired
Wi
PL Piting
Weight Matrices with Different Aspect Natio
Wi:
Prediction
m
Traned.
Conrect
W;
Good Assignment
Layers assigned with inagpropricte LR
Well Trained Layer Without Sampling
Less Heavy Tailed
PL Alpha Valoes
Inaccurate
Rudly Optimined Models
PL
W
Prediction: Bodly Trained.
21
Wroon!!
Imbalanced Assignment Leamingg
Previous Method
Figure I. Comparing F FARMS with common HT-SR methods in analyzing the weight matrices of a DNN. We use the weight matrix W; a well-traines layer with large aspect ratio an example (i.e., y m/n is much larger than 1). Duc to the influence of the aspect ratio y = m/n inn >> 1, the ESD is more concentrated and less HTed than other layers. As a result, previous methods for fitting a power-law distribution (shown "PL Fitting") lend to overestimate the heavytailness metric PLAAppha (the HT index) and incorrectly suggest that the layer is poorly trained. Due to this aspect ratio bias, some optimization methods (such as TempBalance, canmot accurately assign per-layer hyperparameters, leading to suboptimal training. In contrast, method, FARMS, which conducts ESD analysis with fixed aspect ratio, accurately measures the training quality of the layer. See concrete numerical example in Figure 7 in Appendix A.
of different sizes are analyzed using the same HT measur- ing procedure, even though their ESDs can differ in theory. Specifically, algorithms inspired by the HT-SS theory often begin by measuring the ESD a weight matrix W, that is, the empirical distribution of eigenvalues of the correlation matrix W W. The shape of the ESD is then analyzed by fitting it to certain distribution families, such the power- law (PL) distribution. For weight matrices initialized using the i.i.d random Gaussian elements, it is well-known that the ESD converges to the Marshenko-Pastur (MP) distribu- tion as the dimensions of the matrix grow infinitely large in the proportional limit, i.e., when the number of rows m and the number of columns increase but the ratio m/ stays fixed. The MP distribution depends explicitly on the aspect ratio 7 m/n, since this ratio determines the limiting bulk range. [11 (1-) (1+ ?I and it influences the overall shape of the ESD. As a result, weight matrices of different aspect ratios exhibit distinct shapes in their ESDs, especially when they transition from the i.i.d. initialization to being correlated. Comparing ESDs with different aspect ratios y directly, as is commonly done in prior work, overlooks this dependency on aspect ratio and can lead inaccurate shape estimates. Therefore, careful consideration of the aspect ratio is critical when analyzing or comparing weight matrices, especially in eigenspectrum analysis.
suring HT characteristics. FARMS partitions each weight matrix into (overlapping) submatrices of fixed aspect ratios and then does an eigenspectrum analysis on the average ESD of these submatrices. This approach enables accurate computation of heavytailness metrics regardless of varia- tions in matrix aspect ratios, ensuring a robust evaluation of layer quality across diverse matrix sizes that one have in DNN.
To validate the effectiveness of FARMS, we conduct exper- iments across various application domains, including CV, SciML, and LLM pruning. In addition, these experiments are conducted on different parameter settings and model architectures. We compare FARMS with several prior HT-- SR approaches that use weight eigenspectrum analysis for layer-wis hyperparameter assignment (Zhou et al., 2024; Lu et al., 2024; Liu et al., 2024). We also apply FARMS to measure various post-training and pruned models, making FARMS useful for model compression. Our findings demon- strate that models optimized using FARMS exhibit lower mean and variation in HT-SR metries across layers, sign of good-qqulity training reported in prior work (Martin et al., 2021; Liu et al., 2024). Our code is available here!. Our key contributions are summarized below.
Considering these challenges, we introduce FARMS (Fixed- Aspect-Rate Ratio Matrix Subsampling), a new method for mea-
2

---

Eigenspectrum Analysis of Neural Networks without Aspect Ratio Bias
Mitigating Aspect Ratio Bias in Eigenspectrum Analysis. FARMS addresses the aspect ratio bias of existing HT-SR eigenspectrum analysis, and it enables better computation of HT metrics on DNN models that have heterogeneous layer types. This improvement is achieved through a subsampling-bas baase HT estimation method independent of the aspect ratio of weight matri- ces. In particular, we use a numerical example in Fig- ure 6 and multiple real-data experiments in Section 4.3 to convincingly demonstrate that FARMS mitigates the aspect ratio bias in training.
foundation of HT-SR, model analysis tool called Weight- Watcher (Martin & Mahoney, 2021) was developed. With- out accessing any training or test data, methods based on HT-SR Theory can be used to assess the training quality of models across various domains, such as CV and NLP (Mar- tin & Mahoney, 2021; Martin et al., 2021; Yang et al., 2023; Liu et al., 2025b).
2.2. Optimization Methods Based on Eigenspectrum Analysis
Our paper is mainly motivated by HT-SR theory, focusing on the eigenspectrum analysis of weight matrices. Our pa- per connects to several DNN optimization methods that use eigenspectrum analysis to improve model performance. For example, spectral norm regularization has been applied to improve generalizability of trained models (Yoshida & Miy- ato, 2017; Miyato et al., 2018; Farnia et al., 2019). Stable rank normalization (SRN) (Sanyal et al., 2020), a normal- ization technique, scales matrices using their stable rank to enhance training stability in GANs and generalization in DNNs. TE-NAS (Chen et al., 2021) is training free neural architecture search framework that identifies high- performing networks by analyzing the neural tangent kernel spectrum and input space linear regions.
Improved Layer-wise Hyperparameter Tuning. Since HT-SR Theory has been recently applied to various layer-wise hyperparameter tuning methods, FARMS thus improves these methods by offering more accurate evaluation of HT metrics. In partic- ular, we conduct experiments on DNN training and pruning across various model architectures. In CV models like ResNet and VGG, integrating FARMS into learning rate assignments yields improved accuracy compared to TempBalance, a recently proposed layer-wise learning rate assignment method (Zhou et al. 2024). In LLMs pruning, FARMS improves Al hllpphaPuminn (Lu et al., 2024), the SOTA method in assigning layer-wise pruning ratios. Specifically, FARMS can reduce the perplexity of the LLaMA- 13B model from 2029.20 to 413.76 using the mag- nitude pruning method a 0.7 sparsity ratio. As another example LLaMA-7B, it reduces the per- plexity from 96.02 to 79.42 using the SparseGPT pruning method (Frantar & Alistarh, 2023) at a 0.8 sparsity ratio. In SciML, FARMS helps scientific models achieve up to a 5.66% error reduction dur- ing one-tuning compared to the HT-SR based method TB si smood (Zhou et al., 2024), even at relatively low L2 relative error (L2RE) levels.
A lthough the aforementioned optimization methods based on eigenspectrum analysis have achieved certain degree of model improvement, they do not provide fine-grained layer-wiss wise optimization within DNNs. However, eigen- spectrum analysis based on HT-SR theory offers novel perspective by analyzing the shape of the HT ESDs, en- abling a more precise estimate of the training quality for each layer. TempBalance (Zhou et al., 2024) performs eigenspectrum analysis on ESDs of the weight matrices of DNNs assess the training progress of each layer. It enables balanced adjustment of the learning rate across layers during training- an important parameter that func- tions as a "temperature- parameter within the language of statistical mechanics of learning. A phap PhunPrun i ng (Lu et al., 2024), on the other hand, uses a similar weight analy- sis method to evaluate the training quality of each layer in LLMs. It then uses this information strategically allocate layer-wies wise sparsity ratios to maintain model performance after pruning.
2. Related Work
2.1. Prior Work on HT-SR Theory
In this section, we provide overview of prior work HT-SR theory, framework derived from RMT that is rele- vant to understanding modern, practical DNNs. HT-SR the- ory (Martin& & Mahoney, 2021; Mahoney & Martin, 2019) originates from RMT but extends well beyond it. The theory was proposed based on the observation that well- trained, state-of-the-art DNNs often exhibit HTed structures in the ESD of each layer. In the meantime, several rigorous theories in SGD relating HT phenomena to generalization performance were established, providing further theoreti- cal support for HT-SR theory (Hodgkinson & Mahoney, 2021; Hodgkinson et al., 2022; Gurbuzbalaban et al., 2021; Simsekli et al., 2019; 2020). Building on the theoretical
3. Method
In this section, we first discuss the motivation behind our method FARMS, which is address the aspect ratio bias of typical HT-SR methods. Then, we describe FARMS in detail and show how F FARMS can reduce the aspect ratio bias.
3

---

Eigenspectrum Analysis of Neural Networks without Aspect Ratio Bias
3.1. Typical HT-SR Analysis
amount of data generally results in lower standard devia-
tion (STD) of PI. A Alpha Hill across the model, indicat- ing more balanced training progression across different layers (Liu et al., 2024).
Before introducing FARMS, we first review the commonly adopted procedures in HT-SR weight analysis. HT-SR anal- ysis relies on estimating the layer quality based on the HT characteristic of the layer ESDs, which is quantified by HT metric PL_A] Ah_Alph,, introduced below. The ESD of a weight matrix is the histogram of cigenvalues. The ESD of weight matrices evolves during training, transitioning from a bulk- dominated regime to an HTed regime (Martin et al., 2021). The HTeed portion can be modeled by power-law (PL.)
3.2. Rationale of FARMS: Why Typical HT-SR Methods Are Insufficient
Here, we explain the rationale behind our method FARMS. In Section 3.1, we mentioned that layers exhibiting more HTed ESDs tend to be more well-trained. This observation is the key to using HT-SR methods to measure model quality, However, beyond the training quality of weights, the aspect ratio of weight matrix also influences the heavytailness of its ESD. Specifically, in RMT, the Marchenko-Passee (MP) distribution describes the limiting behavior of singular value distributions in large rectangular random matrices, Accord- ing to the MP distribution, the ESD of the correlation matrix W T W of an i.i.d. Gaussian random matrix W Wmxn be- comes more concentrated as the aspect ratio m/n deviates from 1. An example is presented in Figure 2. Consequently, weight matrices with different aspect ratios naturally exhibit varying ESD shapes, leading to different degrees of heavy- tailness, which, if not analyzed carefully, can interfere with the quality assessment of model layers. We refer to this issue as the aspect ratio bias in HT-SR methods.
distribution within an interval (Amin: Amax):
P(A) x a= Amin x < Amars
(1)
where a, the PL exponent, is a critical metric for analyzing training quality. To fit PL distribution to the ESD, methods in HT-SR often use the Hill Estimator (Hill, 1975; Zhou et al., 2024; Lu et al., 2024; Liu et al., 2024) 2. For the i-th layer, suppose the weight matrix is W; and the correlation matrix W W, has ascending eigenvalues {A, The Hill estimator calculates PL.Alph Hill (the PL expo- nent PLLAA pha estimated using the Hill estimator) as:
k
PL Alpha Hill = 1 1
1 In
where k is an adjustable parameter. Changing k essentially changes the lower eigenvalue threshold Amin for (truncated) PL estimation. Various metrics have been proposed to an- alyze the properties of ESDs, among which shape metrics, which characterize the distributional shape of ESDs, have been shown to effectively predict the training quality of in-
m m-4000, n-4000
m 4000, n- 66000
Imrpiiccl
Eipetica
Micdeelo-Parne 7DF
Marceko-Patte FDF
Density
Density
2-
0.5
0
0.0
dividual layers ( (Yang et al., 2023). The PL, A1 phe Hi Hi metric one such shape metrics.
0
2
4
2
Eigenvalue /n
Eigenvalue/ /n
There are several metrics used to quantify the structure of ESDs in HT-SR theory. In this work, we mainly consider the PL.Alpha.Hill which is empirically shown to be ef- fective for training tasks (Zhou et al., 2024; Lu et al., 2024; Liu et al., 2024; Qing et al., 2024). In general, undertrained layers in DNNs tend to exhibit larger P AL Al.Alpha.H.. pha_Hill val- ues, whereas well-trained or over-trained layers typically have smaller PL Alpha Hill values. This metric can also be used for a comprehensive analysis across a series of DNN models with similar architectures to find the best model. Well-trained models tend to exhibit a lower aver- age PI.Alpha.Hill across all layers (Martin et al., 2021; Yang et al., 2023). Moreover, in fine-tuning tasks, a larger
Figure 2. ESD shapes can be biased by the aspect ratio. Here we visualize the eigenvalues of the symmetric matrix A X.. The left ESD more HTed, while the right ESD is more concentrated. However, this change in the shape of the ESD is entirely due to the different aspect ratios of these two matrices and is not caused by differences in training quality. See detailed settings and supplementary results in the Figure 8 in Appendix B.
As another example, consider a well-trained layer with a large aspect ratio, such as the final layer of a ResNet 18 model trained on the CIFAR 100 dataset. This weight ma- trix has size 512 x 100 and large aspect ratio of 512/100 Thus, the ESD may not exhibit an obvious enough HT struc- ture (See a related example in Figure in Appendix A). If previous HT-SR methods are used, such layers may be quan- tified as poorly trained, but that conclusion results from the aspect ratio bias and is an inaccurate assessments of their training quality. Such misjudgment not only interferes with the evaluation of model performance but also leads to inac- curate tuning of hyperparameters in layer- wise optimization
"One important point to note is that estimating PLs is inherently a challenging task (Clauset et al., 2009). Previous research suggests using the maximum likelihood estimate (A (Alstott et al., 2014; Martin & Mahoney, 2021). However, empirical evidence indicates that the Hill estimator performs more reliably in DNN optimization applications (Zhou et al., 2024; Lu et al., 2024: Liu et al., 2024).
4

---

Eigenspectrum Analysis of Neural Networks without Aspect Ratio Bias
characteristics.
Fixed Matrix Sampling
Performing SVD on Each Submatrix
For CNN layers, since they have 4D tensor weight matri- ces with four dimensions C2 k kw] (which repre- sent input channels, output channels, height, and width, respectively), we employ slightly different method for subsampling and measuring. We first flatten the two dimen- sions representing the convolution kernel size, [KH, kw], in the 4D tensor into a single dimension. This results in y' = kn x kw two-dimensional matrices of shape [C1, C2]. We assume here that Ci 2 C2 for convenience. Then, for the 3D tensor of shape l' x C1 x C2. we perform subsam- pling of size m' x n' on each C x C2 matrix corresponding to each I'. This results in [ x [ x l' submatrices, each of size m' n'. Next, we average the ESDs by concate- nating the rooted singular values for each submatrix along the l' dimension, and then measure the HT metrics of the ESDs. In this way, we obtain G [ Finally, we take the average of these metrics values to ob- tain the HT metrics corresponding to the original 2D CNN. We also considered concatenating all the rooted singular values into a single list and calculating the corresponding ESD to measure the degree of heavy-tailedness. However, our experimental results in Table in Appendix C.2 show that averaging the results separately leads to better model performance. The detailed procedures of FARMS are shown in Figure 3.
W
W a
n!'
W
Wa
772
W:
W= UEV
PL Pittng
U dip.... )in'
Calculate HT-SR Metrics
Concatenating Figenvalues
Figure 3. Main Steps in FARMS.
HT metrics values.
methods (Zhou et al., 2024; Lu et al., 2024), as these meth- ods all assign different layer-wise hyperparameters based on the HT estimates. Therefore, eliminating such aspect ratio bias is crucial for improving HT-SR eigenspectrum analysis.
3.3. Analyzing ESDs Using FARMS
To mitigate the aspect ratio bias in analyzing ESDs, we use block-wise sampling method when processing weight matrices. We partition each weight matrix into overlapping sub-matrices with a fixed aspect ratio (across all layers) following a predefined scheme described below.
4. Empirical Results
In this section, we apply FARMS to measure HT metrics. To demonstrate the effectiveness of the new approach, these metrics across various optimization methods and tasks from different machine learning subfields.
Consider the weight matrix W, of the i-th layer, which has a shape of m x n. Without loss of generality, consider the case when m 2 n.. Note that the shape m changes across layers. To process this weight matrix, apply sliding window approach to partition it into several equally sized submatrices (potentially with overlap), denoted as W.,, W.,,..., Wa, where represents the number of sub- matrices. Each submatrix has shape m' 'n' and satisfies fixed aspect ratio Q = m' /n'. The parameters m', n', and Q are tunable hyperparameters (on which we will pro- vide ablation studies in Section 4.6), allowing flexibility in the partitioning strategy. The key requirement is that even if the aspect ratio of the whole weight matrix m/n changes for different layer index i, that of the submatrices Q = m' /nn is fixed across all layers.
In Section 4.1, we give full details of the experimental setup. In Section 4.2, we applied FARMS to LLM layer- wise pruning. In Section 4.3, we employ FARMS on training VGGs and ResNets on image classification tasks. In Section 4.4, we validate that FARMS achieves better performance in SciMI. fine-tuning experiments. In Section 4.5, we ap- ply FARMS to analyze model quality, such as the balance of different layers. Finally, we perform ablation studies in Section 4.6.
4.1. Experimental Setup
Datasets. For image classification, we consider the CI- FAR100 dataset (Krizhevsky, 2012). CIFARI00 consists of 50K pictures for the training set and 10K pictures for the testing set with 100 categories. For evaluating LLM pruning methods, we calculate model perplexity held-out WikiText (Merity et al., 2017) validation set and use seven tasks, including BoolQ (Clark et al., 2019). RTE (Wang et al., 2019), HellaSwag (Zellers et al., 2019).
Subsequently, for 2D Linear layers we compute the eigen- values of the correlation matrices X.j W Wi for each submatrix in the i-th layer. We then average the ESDs of W.,, W W2,..., Wa (by merging their corresponding eigen- value scries) and measure the HT metrics of the averaged ESD instead. We merge the eigenvalue series because that's equivalent to averaging the empirical densities. The av- erage of the ESDs leads to a less variant estimate of HT
the
5

---

Eigenspectrum Analysis of Neural Networks without Aspect Ratio Bias
Table WikiText validation perplexity for pruned I. a- and I.a.A-133B models at different sparsity settings. Our method compared to h.lpharruning each paired with magnitude based pruning, Wanda, and SparseGPT. Lower perplexity indicates improved model performance. calculating the standard deviation(STD) and demonstrating the stability of our method, we sample different calibration sets with 128 samples using six different seeds [0. 1, 2, 3, 4, 5] in Wanda and SparseGPT.
Sparsity Ratio
Layer-wise Method
LLaMA-7B
LLaMA-13B Wanda
Magnitude
Wanda
SparseGPT
Magnitude
SparseGPT
0.7
AlphaPruning
231.76
0.25
18.66+ 0.49
2029.20
14,47+ one
13.29= 0.17
Ours
173.49
22.61 0.18
18.531
413.76
14.20
13.06 0.14
0.75
AlphaPruning
2046.22
104,5324.49
36.52+1.13
2710.49
32.18
22,26=0.59
Ours
1704.56
71.67
35.47
2634.82
29.56
20.80
0.8
AlphaPruning
28865.67
772.20=27.700
96.02.11.99
5399.87
160.59
47.57
Ours
12799.58
504,58_23.15
79.42:t386
5026.86
127.49:2.12
41.44=1.58
0.85
AlphaPruning
71710.96
4609.70
272.84433.88
38140.95
3144.01
122.38
Ours
66808.51
3595.54=x10.54
234.46::16.2
37453.06
2847.85
101.06::448
WinoGrande (Sakaguchi et al., 2021), ARC Easy and Chal- lenge (Clark et al., 2018) and OpenbookQA (Mihaylov et al., 2018) for downstream zero-shot evaluation (Gao et al., 2021). For SciML, we fine-tune the models on simulated solutions of time-dependent PDE dataset 2D Compressible Navier-Stokes (CFD) from PDEBench (Takamoto al., 2022). All datasets considered in this paper are standard and widely studied.
els. All three optimization methods adopt ESD analysis tech- niques, which are susceptible to aspect ratio bias because the HT metrics are measured on the whole weight matrix. Therefore, one can replace the HT measurement procedures in these methods with FARMS. The goal is to evaluate the improvement of FARMS compared to the existing HT-SR analysis used in these optimization methods.
4.2. Improving LLM Pruning with FARMS
Models. We consider different types of NNs in various research fields: VGG (Simonyan & Zisserman, 2015),
To explore the effectiveness of FARMS, we conduct ex- periments in the task of LLM pruning. As have men- tioned,
ResNet (He et al., 2016), OPT (Zhang et al., 2022).
LLaMA (Touvron et al., 2023a;b; Grattafiori et al., 2024)
replace the HT measurement procedures in A runiPr with F FARMS. Therefore, we can compare AlphaPrunt runing with FARMS and without. We make the comparison with different sparsity ratios in the range (0.7, 0.75, 0.8, 0.85] and different LLM pruning methods, includ- ing Magnitude-based (Han et al., 2015), SparseGPT (Frantar & Alistarh, 2023) and Wanda (Sun al., 2024). We use different L.MM pruning methods because A phap Pran ng assigns only layer-wise ratios, not specific pruning locations, and thus can be applied to various LLM pruning methods. We followed the experimental setup from Lu et al. (2024) and our hyperparameter ranges are reported in Table 14 of Appendix
and DPOT (Hao et al., 2024). For VGG series, we con- sider VGG16 and VGG19. For ResNet series, consider ResNet18 and ResNet34. For OPT series, we consider For
four different model size: TTT-155//300M/133B/66 LLaMA series, we consider six different moodels: LLaMA- 7B/13B, LLaMA-V2-7B/133 B/B and LLLMA-V3-3B/SS V3-3B/8B. For DPOT series, we consider DPOT-Tiny and Smal-Small.
Baseline. We considered several model diagnostic and layer- wise hyperparameter scheduling methods as baselines for comparison. TempBalance (Zhou et al., 2024) is layer- wise learning rate allocation algorithm based on HT-SR the- ory, designed to analyze the training progress of each layer and allocate learning rates accordingly. Temp ance measures the HT metrics of layers, and it assigns a larger learning rate to weight matrices with a more righted-tailed ESD.
Language Modeling, The results in Table illustrate that using F FARMS helps A AlphaPrunimg achieve better per- formance under different sparsity ratios and LLM pruning methods. Specifically, when using the Magnitude method, our approach reduces the perplexity of LLaMA-7B from 231.76 to 173.49 sparsity ratio of 0.7. Similarly, when employing more advanced pruning methods such as Wanda and SParseGPT, our approach further reduces the perplexity of LLaMA-77 from 96.02 to 79.42 and LLaMA-133 from 47.57 to 41.44 at a sparsity ratio of 0.8. One notable advan- tage of A AlphaPrunt ng that allows pruning L.Mss
(Lu al., 2024), on the other hand, is a layer-wise pruming method for LLMs based HT-SR. It assigns larger pruning ratio to Transformer weight matri- ces with a more lighted-tailed ESD. Additionally, we com- pared our method with the TB. TBBSSiin Sigmo i d (Liu et al., 2024) approach, which was applied to experiments on SciML mod-
*CFD means compressible fluid dynamics or, equivalently, the compressible Stoker equations.
6

---

Eigenspectrum Analysis of Neural Networks without Aspect Ratio Bias
Table Comparison of mean zero-shot accuracies (%) for pruned I.I LM--BB and LLaMA-138 models different sparsity settings. We evaluate method against AJ AlphaP unnnng each integrated with magnitude-basee pruning, Wanda, and SparseGPT. Higher accuracy values indicate better sero-shot ability.
Sparsity Ratio
Layer-wise Method
LLaMA-7B
LLaMA-13B
Magnitude
Wanda
SparseGPT
Magnitude
Wanda
SparseGPT
0.7
AlphaPruning
35.67
43.67
44.79
38.23
47.46
49.07
Ours
35.96
44.26
44.84
38.87
47.75
49.61
0.75
AlphaPruning
34.59
37.99
40.89
38.16
42.73
44.17
Ours
35.88
39.66
40.93
37.68
42.66
44.47
0.8
AlphaPruning
33.79
34.06
36.63
35.59
38.42
39.07
Ours
34.33
35.76
37.50
36.94
39.23
40.18
0.85
AlphaPruning
33.29
31.69
34.86
33.11
32.05
36.73
Ours
34.27
33.09
35.25
33.29
32.41
37.04
to a sparsity ratio of 0.8 without significantly impacting perplexity. With FARMS, this performance can be further improved. We can also find that FARMS achieves lower perplexity STD in most settings. In Table 8 and Table 9 in Appendix C, we provide additional experiment results on OPT series models and more recent LLaMA models.
the cosine annealing schedule. These excluded layers have Hall-and-skimi matrices," which are matrices with large aspect ratio and a relatively small number of eigenvalues. For example, the final layer of the ResNet 18 model has dimensions of 512 100, resulting in large aspect ratio of 512/100 5.12. This kind of extreme size makes difficult for previous methods accurately estimate the heavytailness of the ESDs. This issue further exacerbates the inability of the previous TempBalance method to ac- curately allocate per-layer learning rates, resulting in certain layers remaining at either excessively high or extremely low learning rates. We illustrate this phenomenon in more detail in Figure 12 of Appendix C. Consequently, this imbalance leads suboptimal model performance after training.
Zero-Shot Tasks, We test the zero-shot ability of pruned LLMs on seven zero-shot downstream tasks mentioned in Section 4.1 with prompting. Results are shown in Ta- ble 2, where we report the mean zero-shot accuracy on seven zero-shot tasks of pruned L.LMMA-BB and L-aMA- 13B models. FARMS can outperform AlphaPruning in improving accuracy across most settings. For example, although AlphaPrunimg achieved significant improve- ments over the uniform method, FARMS further improved accuracy by values of 1.11% % in SparseGPT pruning over sparsity ratio of 0.8. We report the detailed performance for each individual task Table 10 and Table 11 in Appendix C.
Therefore, according to the experimental setup described in TempBalance, the first and last layers of ResNet and VGG models, along with certain layers whose correlation matrix weight matrix contains a relatively small number of eigenvalues, will be excluded from the layer-wise learn- ing rate scheduling. To ensure a comprehensive study, we compare FARMS and TempBalance with and without this "layer selection" (LS) heuristic. See Figure 4, when the LS heuristic is not applied (and all layers will be included in the adjustable learning rate scheduling), the performance of the original TempBalance method (shown as "TB (no LS)")) deteriorates significantly. For example, compared to the LS enablede setup (shown as "LS+TB"). ). the test accuracy of the VGG 19 model trained on CIFAR 100 without LS using the TempBalance method drops from 74.19% to 73 73,22%, while the standard deviation increases from 0.159 to 0.277. In contrast, when training models using the TempBa ance method with FARMS, performance remains stable or even improves when LS is disabled. This observation demon- strates that our method is more robust to weight matrices of extreme sizes, and the reason is that our method provides more accurate assessment of the HTed ESDs independent of the matrix size.
AalphaPruning
4.3. Improving Image Classification with FARMS
In this subsection, compare FARMS TempBal ance on image classification tasks. As mentioned, TempBalance is layer-wise learning rate assignment method using HT metrics. In Figure 4, we report the evalua- tion results of training the ResNet and VGG series models under different optimizing settings. Again, we compare TempBal ance with or without FARMS. "Ours" means re- placing the method to measure heavytailness previously employed in the TempBalancs with FARMS.
An important configuration used in the TempBal lance method is the exclusion of certain layers from the learning rate assignment process. Instead of receiving laser-wise learning rate adjustments as in TempBalance, these lay- ers are assigned a global learning rate that decays following

---

Eigenspectrum Analysis of Neural Networks without Aspect Ratio Bias
LS+TB
LS+Ours
Tn LS)
Ours(no LS)
80.0
81.0
76.0
75.0
Test Accuracy
Test Accuracy
Test T Accuracy
80.5
75.5
74.5
79.5
80.0
74.0
75.0
79.5
73.5
79.0
79.0
74.5
Test
73.0
78.5
78.5
74.0
72.5
a ResNet 18
b ResNet 34
C VGG 16
VGG 19
Figure Comparing our method to TempBalance at different layer usage settings training ResNet and VGG series models on CIFAR100. Higher test accuracy values indicate better model performance. See Table 13 in Appendix F for the details in all hyperparameters.
In TempBalance algorithm, the layer-wise adjustable learning rate is scaled the range of (817)t 821)), where I is the global learning rate time and 81 and 82 are two tunable lower and upper limits of learning rates. To demonstrate that FARMS more accurately evaluate the training quality of each model layer, we present the test per- formance of models trained using the different learning rate scaling ratios (81, 82). in which we consider five different settings for ($1, #2): [(0.1, 1.9), (0.2, 1.8), (0.3, 1.7). (0.4, 1.6), (0.5, 1.5)]. Additionally, we consider that these five scaling ratios are generally close the optimal scaling ratio. We run tasks CIFAR 100 with four VGG and ResNet architectures. Our results in Table 3 show that FARMS improve the test accuracy of models among the five scaling ratios and help models achieve a lower overall standard de- viation compared to TempBalance. Detailed results can be found in Figure 9, Appendix C.
Table 4, FARMS achieves lower L2RE(() on the test datasct than gmoid d and baseline fine-tuning method SciMI. tasks.
Subsampling Ratio
Method
DPOT-Tiny
DPOT-Small
5%
FT Sig
1.863e-2
1.546e-2
1.856e-2
1.539e-2
Ours
1.842e-2
1.536e-2
10%
FT
1.747e-2
1.426e-2
TB Sig
1.730c-2
1.415e-2
Ours
1.706e-2
1.407e-2
25%
1.543e-2
1.226e-2
TB_Sig
1.517e-2
1.203e-2
Ours
1.499e-2
1.189e-2
50%
FT 1.309e-2
1.025e-2
TBSSig
1.283e-2
1.005e-2
Ours
1.242e-2
9.822e-3
100%
FT 1.09e-22
8,400e-3
Table 3. Comparing method to TempBa ance among the five scaling ratios. All results are reported as the mean test accuracy and standard deviation obtained across five different scaling ratios. this experiment, we allowed TempBal ance baseline to use the LS heuristic achieve slightly better test accuracy.
TB Sig
1.078c-2
8.13e.
Ours
1.017e-2
7.949e-3
with layer selection and the baseline fine-tuning (shown as "FT") with a uniform learning rate for each layer in different data subsampling ratios in the range (5%, 10%, 25%, 50%, 100%). Our method consistently outperformed TB sigmoid across all subsampling ratios. For instance. with full dataset (100% of the data), our approach re- duced the model's L2RE 1.017e-2. Comparing to the TB_Si TB_Sigmoid d, the L2RE decreased by 5.66%.
Method
ResNet 18
ResNet 34
VGG 16
VGG 19
TB
79.81 19. 80.07:00097
74.8720214 75.16:0211
13.89
Ours
19.35....26
74.03
4.4. Improving SciML. Fine-tuning with FARMS
To explore the potential applications FARMS in multiple domains of machine learning research, we also performed SciML fine-tuning experiments using the 2DCFD dataset with DPOT-Tiny and DPol-Small models and compared FARMS with the TB S gmoic method (Liu et al., 2024). We followed the experimental setup from Liu et al. (2024), and our hyperparameter ranges are detailed in Table 14 of Appendix Table 4, we show the results of compar- ing FARMS with the i gmoid (shown as "TB Sig")
4.5. HT Metries Analysis
In this subsection, demonstrate that FARMS effec- tively control the shape of ESDs, resulting in more bal- anced distribution of PL Alpha Hill values among the layers of NNs compared to the previous methods. We as- sess the models presented in Section 4.3, which include 2D Convolution and 2D Linear layers of ResNets and VGG
8

---

Eigenspectrum Analysis of Neural Networks without Aspect Ratio Bias
CAL
CAL
CAL
CAL
Temp Bolance
Temp Balance
Tomp Ralance
Tamp Balance
Ours
1.5
1.3 PL Alphu Hill
2.5
1.5
25
2.0 PL A Apha lpha_Hiill Hill
PL Alphu Hill
PL PL_Alphe Alpha Hill
a ResNet 18, C0f
ResNet 34, C0f
e VGG 16, C0f
d VGG 19, Cifar
Figure 5. Comparing the distribution of PL.Alpha.Hi Hill ResNet and VGG series models. The top blue-shaded section in each subplot of the experiments (shown as "CAL") represents models trained without employing layer-wise optimization methods. The middle brown-shaded section each subplot indicates models trained using TempBalance Lastly, the bottom orange-shaded section in cach subplot corresponds to the results obtained using PARMS. Figures 5a, 5b, 5c and 5d present the averaged results obtained from multiple experiments within the optimal parameter range.
networks. Results in Figure 5 show the distribution of ILLAlpha_Hill of models trained using baseline cosine annealing (CAL), using our method FARMS. can be seen that FARMS gener- ally leads to a more concentrated distribution and, in most cases, reduces the average Pl.AlphaaHiil value. These experimental results suggest that our method FARMS en- ables a more balanced training progression across differ- ent layers of the model. These observations align well with the findings reported by Martin et al. (2021); Liu et al. (2024), which suggest that decrease in both aver- age PL.Alpha. _iill values and the variance across layers indicate better training quality.
est dimension is 4096. Therefore, when using window size of 4096, sliding window downsampling is applied only along the larger dimension. This process generates mul- tiple 4096 4096 submatrices. We report our results in Table 6. We find that using an appropriately sized sampling window, such as 2000 x 2000, the model achieves lower perplexity. As a comparison, baseline perplexity achieved by AlphaPrun without using FARMS is 96.02.
CompBalance, and TempBalance
Table 6. Different Window Size and Sampling Steps for prun- ing the ILaMA-7B 0.8 sparsity ratio. The pruning method is SparseGPT. The model is evaluated on WikiText dataset with perplexity (L).
4.6. Ablation Study
Sampling Steps
Window size
5
10
15
20
Different Aspect Ratios of Weight Matrix. Here, we study the effect of the aspect ratio of the subsampled matrices in FARMS. We consider image classification tasks and use FARMS to replace the HT metrics in TempBale CampoBlannce lance for as- signing layer-wise learning rates. We consider five different aspect ratios in the range (0.5, 0.75, 1.0, 1.5, 2.0} and keep other hyperparameters optimal. We again use ResNet18 and VGG16 as our architectures and show results on CIFARIOM Results in Table 5 show that FARMS achieves higher test accuracy when the aspect ratio is 1.0.
500
96.34=215
95.93
99,23
88.00 48.0884423
1000
84.711497
81.96.4.22
89.201191
84.041248
2000
82.38
79.42:1386
80. 14
86.541338
4096
82.63
89.61
84.19
84.07.1544
5. Conclusion
We propose a subsampling strategy to address the measure- ment bias introduced by varying aspect ratios of weight matrices in HT-SR eigenspectrum analysis. The main idea of our method is to extract submatrices of the original matrix with fixed aspect ratio, followed by averaging the ESDs of these submatrices to accurately assess the training quality of each weight matrix. Our extensive experiments demonstrate that our method can precisely estimate layer-wise training quality, improve the performance of layer-wise optimization algorithms, and contribute to more balanced training and efficient pruning. Furthermore, results on visual analytics confirm the effectiveness of our approach, showing that it successfully eliminates the measurement bias caused by aspect ratio variations.
Table 5. Comparing the different Aspect ratios settings for subma- trices on ResNet and VGG 16 models trained on CIFARI00 Dataset.
Ratio
0.5
1.0
1.5
ResNet 18
79.13s
79.33ss122
79,53=6177
79.13s020
79.31
VGG 16
=5.05 75.19=0310
75.36=4.13
75.21
75.10
Subsampling Window Sizes and Sampling Steps. Here, we study different subsampling window sizes and sampling steps (i.e., the number of subsampled submatrices) in the task of LLaMA-7B pruming. The aspect ratio Q is fixed to be 1.0. For all weight matrices in LLaMA-7B, the small-
9

---

Eigenspectrum Analysis of Neural Networks without Aspect Ratio Bias
Impact Statement
Derezinski, M., Liang, F. T., and Mahoney, M. W. Exact expressions for double descent and implicit regulariza- tion via surrogate random design. Advances in neural information processing systems, 33:5152-55555 5164, 2020.
This paper presents research aimed at advancing the fields of Machine Learning and Deep Learning, particularly in train- ing using information from eigenspectrum analysis. While work has various potential societal implications, we do not find it necessary to highlight any specific ones here.
Dobriban, E. and Wager, S. High-dimensionall asymptoties of prediction: Ridge regression and classification. The Annals of Statistics, 46(1 (6(1))247-279, 2018.
Acknowledgments
Farnia, F., Zhang, J., and Tse, D. Generalizable adversar- ial training via spectral normalization. In International Conference on Learning Representations, 2019.
This work supported by DOE under Award Number DE- SC0025584 and Dartmouth College.
Frantar, E. and Alistarh, D. Sparsegpt: Massive language models can be accurately pruned in one-shot. In Inter- national Conference on Machine Learning, pp. 10323 10337. PMLR, 2023.
References
Adlam, B. and Pennington, J. The neural tangent kernel in high dimensions: Triple descent and multi-scale theory of generalization. In International Conference Machine Learning, pp. 74-84 84. PMLR, 2020.
Gao, L., Tow, J., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu, J., McDonell, K., Muennighoff, N., et al. A framework for few-shot language model evaluation. Version vO. 0.I. Sept, 10:8-9 9, 2021.
Alstott, J., Bullmore, E., and Plenz, D. powerlaw: a python package for analysis of heavy-tailed distributions. PloS one, (1):e85577 2014.
Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. The Ilama 3 herd of models. arXiv preprint 2rXi:2247. 2024.
Ba, J., Erdogdu, M. A., Suzuki, T., Wang, Z., Wu, D., and Yang, G. High-dimensional asymptotics of feature leam- ing: How one gradient step improves the representation. Advances in Neural Information Processing Systems, 35: 37932-37946, 2022.
Guhr, T., Müller- Groeling, A., and Weidenmüller, H. A. Random-matio theories in quantum physics: common concepts. Physics Reports, 299(4): 425, June 1998. ISSN 13370-1573 doi: 022101///0370-1553377700
Bai, Z. and Silverstein, J. W. Spectral analysis of large di- mensional random matrices, volume 20. Springer, 2010.
Gurbuzbalaban, M., Simsekli, U., and Zhu, L. The heavy- tail phenomenon in sgd. In International Conference on Machine Learning, pp. 3994-39777 PMLR, 2021.
Chen, W., Gong, X., and Wang, Z. Neural architecture search on imagenet in four gpu hours: A theoretically inspired perspective. In International Conference on Learning Representations (ICLR), 2021.
Han, S., Pool, J., Tran, J., and Dally, W. Learning both weights and connections for efficient neural network. In Cortes, C., Lawrence, N., Lee, D., Sugiyama, M., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015.
Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M.. and Toutanova, K. Boolq: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, Volume (Long and Short Papers), pp. 2924 2936, 2019.
Hao, Z., Su, C., Liu, S., Berner, J., Ying, C., Su, H., Anand- kumar, A., Song, J., and Zhu, Dpot: Auto-regressive denoising operator transformer for large-scale pde pre- training. In International Conference on Machine Learn- ing. pp. 17616-17635. PMLR, 2024.
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint 1003::880.00544
Hastie, T., Montanari, A., Rosset, S., and Tibshirani, R. J. Surprises in high-dimensional ridgeless least squares in- terpolation. Annals of statistics, 50(2):949, 2022.
2018.
Clauset, A., Shalizi, C. R., and Newman, M. E. Power- law distributions in empirical data. SIAM review, 51(4): 661-703, 2009.
He, K., Zhang, X., Ren, S., and Sun, J. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE inter- national conference on computer vision, pp. 1026-1033 2015.
Couillet, R. and Liao, Z. Random Matrix Methods for Machine Learning. Cambridge University Press, 2022.
10

---

Eigenspectrum Analysis of Neural Networks without Aspect Ratio Bias
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learn- ing for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Mahoney, M. and Martin, C. Traditional and heavy tailed self regularization in neural network models. In Chaud- huri, K. and Salakhutdinov, R. (eds.), Proceedings of the 36th International Conference on Machine Learning, vol- ume 97 of Proceedings of Machine Learning Research, pp. 2784-4293, PMLR, 09-15 Jun 2019.
Hill, B. M. A simple general approach to inference about the tail of a distribution. The annals of statistics, pp. 1163-1174, 1975.
Martin, C. H. and Mahoney, M. W. Implicit self- regularization in deep neural networks: Evidence from random matrix theory and implications for learning Jour- nal of Machine Learning Research, 22(165):1-73, 2021.
Hodgkinson, L. and Mahoney, M. Multiplicative noise and heavy tails in stochastic optimization. In Interna- tional Conference on Machine Learning, pp. 4262-4274.
PMLR. 2021.
Martin, C. H., Peng, T., and Mahoney, M. W. Predicting trends in the quality of state-of -ftt neural networks without access training or testing data. Nature Com- munications, 12(1), July 2021. ISSN 2041-1723. doi: 10.10388/44447--011-244000
Hodgkinson, I.,, Simsekli, U., Khanna, R., and Mahoney, M. Generalization bounds using lower tail exponents in stochastic optimizers. In International Conference on Machine Learning, pp. 8774-8795, PMLR, 2022.
Mei, S. and Montanari, A. The generalization error of random features regression: Precise asymptotics and the double descent curve. Communications on Pure and Applied Mathematics, 75(4):667-766, 2022.
Hu, H. and Lu, Y. M. Universality laws for high-dimensional learning with random features. IEEE Transactions Information Theory, (9(3):1932-199) 2022.
Kothapalli, V., Pang, T., Deng, S., Liu, 7., and Yang, Y. From spikes to heavy tails: Unveiling the spectral evolu- tion of neural networks. Transactions on Machine Learn- ing Research, 2025. ISSN 2835-8856.
Merity, S., Xiong, C., Bradbury. J., and Socher, R. Pointer sentinel mixture models. In International Conference on Learning Representations, 2017.
Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. Can a suit of armor conduct electricity? new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2018
Krizhevsky, A. Learning multiple layers of features from tiny images. University of Toronto, 05 2012.
Liu, L., Zhao, X., Li, G., Li, D., Wang, Han, Y., Li, X., and Wang. Y. BaWA: Automatic optimizing pruning metric for large language models with balanced weight and activation. In Proceedings of the 42nd International Conference Machine Learning, ICML '25. PMLR, 2025a.
Miyato, T., Kataoka, T., Koyama, M., and Yoshida, Y. Spec- tral normalization for generative adversarial networks, 2018.
Nadakuditi, R. R. and Newman, M. E. Graph spectra and the detectability of community structure in networks. Physi- cal review letters, 108(18): 2012.
Liu, Z. HU, Y., Pang, T., Zhou, Y., Ren, P., and Yang, Y. Model balancing helps low-data training and fine- tuning. In Proceedings of the 2024 Conference on Em- pirical Methods in Natural Language Processing, pp. 1311-1331, 2024.
Pennington, J. and Worah, P. Nonlinear random matrix the- ory for deep learning. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.. 2017.
Liu, Z., Pang. T., Balabanov, O., Yang. C., Huang, T., Yin, L., Yang, Y., and Liu, S. Lift the veil for the truth: Prin- cipal weights emerge after rank reduction for reasoning- focused supervised fine-tuning. In Proceedings of the 42nd International Conference Machine Learning (ICML), Vancouver, Canada, July 2025b.
Qing. P., Gao, C., Zhou, Y., Diao, X., Yang, Y., and Vosoughi, S. Alphalora: Assigning lora experts based on layer training quality. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 2051 1-20523, 2024.
Lu, H., Zhou, Y., Liu, S., Wang, Z., Mahoney, M. W., and Yang, Y. Alphapruning: Using heavy-tailed self regular- ization theory for improved layer-wist wise pruning of large language models. Advances in neural information pro- cessing systems, 37:9117-9155 2024.
Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106 2021.

---

Eigenspectrum Analysis of Neural Networks without Aspect Ratio Bias
Sanyal, A., Torr, P. H., and Dokania, P. K. Stable rank normalization for improved generalization in neural net- works and gans. In International Conference on Learning Representations. 2020.
Wang, Z., Engel, A., Sarwate, A. D., Dumitriu, I,, and Chiang, 'T. Spectral evolution and invariance in linear- width neural networks. Advances in neural information processing systems, 06:20695520077777 2023.
Simonyan, K. and Zisserman, A. Very deep convolutional networks for large-scale image recognition. In 3rd Inter- national Conference on Learning Representations (ICLR 2015). Computational and Biological Learning Society, 2015.
Yang, G. and Hu, E. J. Tensor programs iv: Feature learning in infinite-width neural networks. In International Con- ference on Machine Learning, pp. 11727-11737. PMLR, 2021.
Yang, G., Hu, E., Babuschkin, I., Sidor, S., Liu, X., Farhi, D., Ryder, N., Pachocki, J., Chen, W. and Gao, Tuning large neural networks via zero-shot hyperparameter trans- fer. Advances in Neural Information Processing Systems, 34:1 14:17084-17097, 2021.
Simsekli, U., Sagun, L.,, and Gurbuzbalaban, M. A tail-index analysis stochastic gradient noise in deep neural net- works. In International Conference on Machine Learning, pp. 5827-5837. PMLR, 2019.
Simsekli, U., Sener, O., Deligiannidis, G., and Erdogdu, M. A. Hausdorff dimension, heavy tails, and generaliza- tion in neural networks. Advances in Neural Information Processing Systems, 33:5138-511 -151, 2020.
Yang. Y., Theisen, R., Hodgkinson, L., Gonzalez, J. E., Ramchandran, K., Martin, C. H., and Mahoney, M. W. Test accuracy vs. generalization gap: Model selection in nip without accessing training or testing data. KDD '23, pp. 3011 3011-3021, New York, NY, USA, 2023, Association for Computing Machinery. ISBN 9798400701030. doi: 10. N0.1445558030533599555
Sun, M., Liu, Z., Bair, A., and Kolter, J. Z. simple and effective pruning approach for large language models. In The Twelfth International Conference on Learning Representations, 2024.
Yin, L., Wu, Y., Zhang, Z., Hsich, C.-Y., Wang, Y., Jia, Y., Li, G., Jaiswal, A. K., Pechenizkiy, M., Liang, Y., et al. Outlier weighed layerwise sparsity (owl): A missing se- cret sauce for pruning Ilms to high sparsity. In Inter- national Conference on Machine Learning, pp. 57101- 57115. PMLR, 2024.
Takamoto, M.. Praditia, T., Leiteritz, R., MacKinlay, D., Alesiani, F,, Pflüger, D., and Niepert, M. Pdebench: An extensive benchmark for scientific machine learning. Advances in Neural Information Processing Systems, 35: 1596-1611, 2022.
Yoshida, Y. and Miyato, T. Spectral norm regularization for improving the generalizability of deep learning, 2017.
Tao, T. Topics in random matrix theory, volume 132. Amer- ican Mathematical Soc., 2012.
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can a machine really finish your sentence? Proceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguisties, pp. 4791-4804 2019.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation lan- guage models. arXiv preprint arXiv:2302. 13971, 2023a.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint arXiv:2307.09288 2023b.
Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01065 2022.
Zhou, Y., Pang, T., Liu, K., Mahoney, M. W., Yang, Y., et al. Temperature balancing, layer-wise weight analysis, and neural network training. Advances in Neural Information Processing Systems, 36, 2024.
Tulino, A. M. and Verdú, S. Random Matrix Theory and Wireless Communications. Foundations and Trends@ in Communications and Information Theory, 1(1):1-122, June 2004. ISSN 1567-2190, 1567-2328, doi: 10.1561. 0100000001. Publisher: Now Publishers, Inc.
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations, 2019.
12

---

Eigenspectrum Analysis of Neural Networks without Aspect Ratio Bias
Appendix
A. Detailed Matrix Analysis
A.I. Random Initialization
In this section, we do weight analysis for ResNet 34 and VGG 19 models that are randomly initialized using methods proposed in He et al. (2015). The weights w are drawn from normal distribution with mean and variance 2/nan. where Bin represents the number of input neurons. This is represented as:
~~w .V (0,
2 Thin
We present our results of comparing FARMS with the previous HT estimation methods in measuring randomly initialized models in Figure 6. In Figure 6a and Figure 6c, all layers in these models should be similarly under-trained (because they are all just initialized). However, previous methods tend to report significantly higher values for PL.Alpha.Hil in certain layers, suggesting that these layers are much more mnderetrained. In contrast, FARMS produces results where the training quality of all initialized layers is nearly identical, aligning more elosely with the expected result. Similarly, in Figure 6b and Figure 6d, for randomly initialized ResNet-34 and VGG-19 models with varying widths, the previous method measures PL.Apph..Hi 11 values that increase as model width increases. In contrast, F FARMS shows PL Ah.lppha... Hill iill values that remain unaffected by the aspect ratio bias introduced by model width variations.
10
PL. Alpha Hill
Previoaa Extiratee
PL_ Alpha Iill
Pecsicus Bainnkr
Our
Alpha Ilill
Provinse Hill Beimaser
Gurs
PL Alpha Illll
Pecricua
Ous
2
20
30
0.5
1.5
2.0
3.0
15
1.5
2.0
3.0
Layer Index
Widen Factor
Layer Index
Widen Factor
a ResNet 34, All Layers
ResNet 34, Different Width
C VGG 19, All Layers
VGG 19, Different Width
Figure Comparing FARMS and previous HT-SR methods for measuring the randomly initialized ResNet and VGG 19 weights. Figure 6a and Figure 6C show the FL.Alpha_Hill values for cach layer in models (widen factor is 1.0) by using different methods. Figure 6b and Figure show the measured PL.Alpha_Hil values of the final linear layer across different model width factors. As the width increases, the aspect ratio of the weight matrix also becomes larger, leading to bias in previous HT-SR methods.
A.2. Mitigating Aspect Ratio Bias Specific Layers
In this section, we do weight analysis for the last layer from the ResNet 34 and VGG 16 models trained on CIFARI00. These models are trained with hyperparameters according to the Appendix F. The weight matrices of the final layers in both models have dimensions of 512 100 (aspect ratio = 5.1).. This means that the previous weight matrix analysis method will be significantly affected by aspect ratio bias, leading to inaccurate assessments of the layer's training quality.
In Figure 7, the presented experimental results validate this observation. Figure 7a and Figure 7c show the ESD fitting results obtained using the previous weight analysis method, where the measured P PL.Alhha.Hill values are larger than those typically observed in well-trained layers. This leads to the erroneous classification of these layers as poorly trained, ultimately affecting both model performance evaluation and optimization. In contrast, Figure 7b and Figure 7d present the ESD fitting results obtained using FARMS. Our method produces measurements that align more closely with the expected ESD of well-trained layers, providing more accurate assessment of training quality.
B. Marchenko-Pastur Distribution
In RMT, the Marchenko- -Psttrr distribution, also known as the Marchenko-Pastur law, characterizes the asymptotic behavior of singular values of large rectangular random matrices.
Let X S i m. 1 S j S n, independent random variables with EX,,, = and EX? 1, and X,,
13

---

Eigenspectrum Analysis of Neural Networks without Aspect Ratio Bias
PL Alpha Hill: 3,091
PL Alpha Hill: 1.899
PL Alpha Hill: 2.951
PL Alpha Hill: 1.905
10
10
10
10"
ESD
ESD
10y
ESD
ESD
10"
10
1011
10
10-1
10 Eigenvalues
10
10
0
10
10 Eigenvalues
10"
Eigenvalues
Eigenvalues
a Previous Method, ResNet 34
b FARMS, ResNet 34
C Previous Method, VGG 16
d FARMS, VGG 16
Figure 7. Comparing 1 PARMS and the previous method for measuring the ESD of final layers in ResNet 34 and VGG 16 trained on CIFAR
100.
m 4000, 4000
m 4000, 5000
m m==000, 6000
Imppincal
Pmpirial
Empircoal
Martesko-Panu
1.0
Maroecko-Foouu PDF
Density
Density
Density
2-
0.5
0.0
2
Eigenvalue /n
Eigenvalue /
Eigenvalue /n
m 000-4000, n n-8000
m 4000, n=
1-2000
111 4000, 16000
Impiincal Marterko-oote PDF
Empirial MarceskoPaatuu FDF
Bmpirica
MarcekooPotur PDF
Density
Density
Density
0.5-
0.5
0.5
0.0
0.0
0.0
2
2
Eigenvalue /n
Eigenvalue/ /
Eigenvalue / n
Figure 8. The Marchenko-Pastiee Pastur (MP) Law for different values of n. The ESD (blue histogram) of the eigenvalues of the sample covariance matrix is compared with the theoretical Manchenko-Pastur probability density function (red curve) for various aspect ratios, where m- = 4000 is fixed, and n varies from 4000 to 16000.
(Xi))ssis.1sssn Denote by X1 S ... S Am the eigenvalues of the symmetric m
W Wm 1
Xm X
n
and defined its empirical distribution by
1 Σ unisoh)
Fm (I)
where T(B) denotes the indicator of an event B. One often investigates the rate of convergence of the expected spectral distribution EFF((x) (x) as well as Fm (I) to the Marchenko-Pastur distribution function (x) with density
[x)x) = 2nyz V(b - x)(x - (14,),))())) (x) + +nn - -y -))(x,,
where y = m//n y E (0, oc) and a = (1 - 2 b = (1 + v Here we denote by 8(x) the Dirac delta function and by [(a.00) (1) the indicator function of the interval (a, bj.
'Note. The transpose operation applies to the second (right-hand) matrix in this expression, which differs from the transpose placement used in HT-SR Theory, shown X,y W. Wa, in Section 3.3. Thus, the "large aspect ratio" regime discussed in HT-SR Theory corresponds to a large value of the ratio n/m in Appendix B.
14

---

Eigenspectrum Analysis of Neural Networks without Aspect Ratio Bias
We visualize the MP distribution and the ESD of the weight matrices in Figure 8. In this figure, we can observe that the empirical distribution converges to the theoretical MP distribution. Additionally, increases, the ESD distribution exhibits an increasingly concentrated shape with a reduced degree of HT. Therefore, ignoring the impact of aspect ratio and directly measuring the HT degree of the ESD to estimate layer's training quality may lead to inaccurate results.
C. Additional Experiment Results
C.1. Detailed Performance on Different Scaling Ratios
We provide detailed results of a hyperparameter study on learning rate scaling ratio (s1, 82). We set other hyperparameters (including layer selection, initial learning rate, and on) as the optimal value for each training set. The results in Figure 9 show that across all tested architectures and most scaling ratios, FARMS consistently outperforms TempBalance. This demonstrates that F FARMS provides more accurate measurement of each layer's training quality, As result, it helps the model achieve better training performance when using different learning rate scaling ratios.
TB
Ours
73.5
Test Accuracy (%)
74.5
Accurayy
Tost Accuracy
Accurazy
75.0
72.0
79.0
70.75
14.5
73.5
70:50
78.5
(0.1 5t (0.2- R0 (0.3-11. 1) 10 mil As (0.00... (.551.5)
181-115 (022-1 8) (0 3-1. 73 10 (8) (0.5-.))
10.141.0 51 (0.2-1 59 (0.3-1 7) 10 (D.5-11))
10.111 9y 1022-1 10 10.3 1.7) 10 10..1 AY (0.5-1.5)
Leannng Scaling Ronge
Leaniing Scating (2))
Leannng Scaling Honge
Learnng Scaling
a ResNet 18, CIFAR 100
b ResNet 34, CIFAR 100
VGG 16, CIFAR 100
d VGG 19, CIFAR 100
Figure 9. Comparison test accuracy across different architectures and leaming rate scaling ranges. The figure presents the test accuracy (%) of TempBal lance ance (blue, dotted) and FARMS (orange, hatched). These two methods are evaluated on the CIFAR 100 dataset using ResNet and VGG series architectures. The x-axis represents different learning rate scaling ranges (81, #2), while the y-axis indicates test accuracy. Error bars denote standard deviations across multiple runs.
C.2. Comparison of Different CNN Processing Methods
We compared two different methods for processing the ESD of CNNs. The first method concatenates the rooted singular values of all submatrices and then measures the HT metrics of the resulting ESD. The second method measures the HT metrics of the ESD formed by concatenating the rooted singular values of each of the C1 x C2 submatrices separately, and then averages these HTT metrics from each group. In Table 7, we provide a performance comparison of training ResNet- 18 and VGG-16 models using these two methods. For each experiment, all other hyperparameters are set to their optimal values. We find that although the performance difference between the two methods is not very large, the second method brings more performance improvement.
Table 7. Comparing Two Stratgy in CNN Processing with FARMS.
Method
ResNet 18
VGG 16
Calculating the Overall ESD
79.3610.101
75.27 75.36=2..18
Averaging ESDs Computed from Subsets
79.53
C.3. Experiment Results for Additional Models
We prune the OPT models (Zhang et al., 2022) and I.LMMA-2//33 models (Touvron et al., 2023b; Grattafiori et al., 2024) with AlphaPruning and compare the model performance with F FARMS. We report the additional experiment results on LLM pruning in Table and Table 9. For the OPT series models, we use four sparsity ratios: {0.6, 0.7, 0.75, 0.8). For the LLaMA series models, we select (0.7, 0.75, 0.8, 0.85] as the sparsity ratios. Experimental results show that our method can help A.l phaP runing achieve better model performance on models such as OPT.
15

---

Eigenspectrum Analysis of Neural Networks without Aspect Ratio Bias
Table Wiki'ext validation perplexity for pruned OPT models different sparsity settings. Our method is compared wwwhapapppan ng. each paired with Wanda and SParseGPT. Lower perplexity indicates improved model performance.
Sparsity Ratio
Layer-wise Method
OPT-125M
OPT-350M
OPT-1.3B
OPT-6.7B
Wanda
SparseGPT
Wanda
SparseGPT
Wanda
SparseGPT
Wanda
SparseGPT
0.6
AlphaPruning
67.12
59.18
87.95
50.94
27.09
22.76
15.6
13.71
Ours
58.86
82.55
50.52
27.15
22.69
15.59
13.67
0.7
AlphaPruning
263.71
196.64
595.26
147.20
101.13
50.30
44.92
20.76
Ours
261.84
188.31
489.75
136,80
100.74
50.01
42.03
20.76
0.75
AlphaPruning
718.69
515.55
1453.21
330.11
613.78
142.68
245.16
35.27
Ours
721.52
491.98
1298.22
303.07
601.24
122.01
181.92
34,88
0.8
AlphaPruning
1713.45
1525.25
2869.8
921.17
2763.33
511.66
5781.0
98.3
Ours
1609.36
1412.59
2551.46
798.25
2502.60
499.82
5212.51
90.88
Table WikiText validation perplexity for pruned LLaMA-V2 and LLaMA-V3 models at diffcrent sparsity settings. Our method compared to Alphapruning, each paired with Wanda and SparseGPT. Lower perplexity indicates improved model performance.
Sparsity Ratio
Layer-wise Method
LLaMA-V2-7B
LLaMA-V--13B
LLaMA-V3.2-3B
LLaMA-V3.I -3.1-8-
Wanda
SparseGPT
Wanda
SparseGPT
Wanda
SparseGPT
Wanda
SparseGPT
0.7
AlphaPruning
34.71
20.92
15.37
13.69
123.19
64.33
105.64
38.43
Ours
31.09
20.49
15.36
13.64
120.03
63,78
107.00
37.81
0.75
AlphaPruning
170.74
41.39
35.26
23.21
356.69
121.53
240.17
81.30
Ours
161.58
39.98
33.67
22.98
311.73
117.19
229.63
81,93
0.8
AlphaPruning
868.66
89.73
150.01
46.32
1412.53
262.44
688.23
180.40
Ours
831.37
88.25
127.69
44.74
1219.10
212.65
607.58
175.93
0.85
AlphaPruning
6425.25
217.79
874.97
107.21
5857.39
505,8
3498.94
400.69
Ours
4437.46
204.44
748.91
98.99
4407.94
463.34
3766.13
411.35
C.4. Zero-shot Tasks Performance
demonstrate the task-wise performance in detail in Table 10 and Table 11.
C.5. Detailed PL Alpha Hill Distribution
We provide additional experimental results on LLM pruning and SciML fine-tuning to support the PL Alpha Hill distribution analysis presented Section 4.5. The results in Figure 10 and show that FARMS consistently helps models achieve better layer quality across different experimental settings.
C.6. Layer-wise Visualization over Training
We provide visualization to compare how the PL.Alpha_Hin and learning rates are distributed over layers during the training with different layer-wise optimization settings. In Figure 12, report the learning rate and PL Alpha Hill every five epochs throughout the 200-epoch training duration. We can find that in Figure 12a, 12e and Figure 12b, 12f, if the previous TempBalancs does not apply LS to exclude layers with aspect ratio bias (e.g.. the first and last layers of the model), the layer-wise learning rate allocation will be inaccurate. This misallocation leads to few layers having excessively large learning rates while most layers have very small ones. This ultimately results in poor model performance or an imbalanced training process. In contrast, FARMS (see results in Figures 12c, 12d, 12g and 12h) effectively eliminates the measurement inaccuracies caused by aspect ratio bias. As a result, it ensures that learning rates are properly allocated regardless of whether L.S applied.
C.7. Computation Cost Analysis
In Table 12, we report the computational cost of eigenspectrum analysis across different models and methods. The most computation-intensiie aspect of these layer analysis methods involves performing SVD on weight matrices, which can be
16

---

Eigenspectrum Analysis of Neural Networks without Aspect Ratio Bias
Table 10. Accuracies (%) of L.aaM-7BB for zero-shot tasks with unstructured sparsity from 70% 85%. We compare FARMS with
uniform pruning ratios and AlphaPrupnan runing using Sienitude-basee pruning, Wanda and SparseGPT.
Sparsity Ratio
Method
BoolQ
RTE
HellaSwag
WinoGrande
ARC-e
ARC-c
OBQA
Mcan
Magnitude
38.29
52.71
25.59
51.22
26.73
19.62
11.60
32.25
AlphaPruning w. Magnitude
41.31
52.71
30.37
51.46
35,44
22.01
16.40
35.67
Ours un. Magnitude
40.09
52.71
30.49
53,83
34.22
23.21
17.20
35.96
Wanda
56.06
55.60
28.90
50.99
32.11
18.26
13.60
36.50
0.7
AlphaPruning Wanda
64.34
57.40
35.57
61.09
45,58
24.49
17.20
43.67
Ours w. Wanda
64,40
59.93
36,54
60.62
45,24
24.66
18,40
44.26
SparseGPT
65.14
53.79
33,95
58.72
44.44
23.98
17.20
42.46
AlphaPruning SparseGPT
65.74
53.07
37.72
64.01
47.35
26.88
18,80
44.79
w. SparseGPT
65.54
53.07
36.85
64.33
48.11
26.37
19.60
44.84
Magnitude
42.26
52.35
25.88
48.54
26.68
21.42
14.00
33.02
AlphaPruning w. Magnitude
50.67
52.71
26.80
49.96
27.19
21.42
13,40
34.59
Ours ur. Magnitude
55.54
53.07
27.80
49.25
29.21
21.08
15.20
35.88
Wanda AlphaPruning w. Wanda
37.83
53.79
27.01
49.96
27.74
19.37
12,60
32.61
0.75
62.17
53.43
29.52
53.75
33.04
20.82
13.20
37.99
Ours w. Wanda
62.17
58.12
30,99
55,49
35.69
21.76
13,40
39.66
SparseGPT
62.14
53.43
29.76
51.22
33,80
19.11
13,40
37.55
AlphaPruning SparseGPT
63.71
52.71
33.11
59,91
38.68
23.89
14,20
40.89
Ours SparseGPT
64.07
53.07
32.74
59.83
38.43
23.55
14.80
40.93
Magnitude
48.81
49.10
25.59
48.78
24.92
22.01
14,20
33.34
AlphaPruning w. Magnitude
44.40
53.07
26.17
50.75
25.67
22.27
14.20
33.79
Ours ur Magnitude
46.76
53.79
26.08
52.33
24.58
22.95
13,80
34.33
Wanda AlphaPruning w. Wanda
61.69
51.42
25.85
50.24
26.23
20.62
14.00
35.72
0.8
52.75
51.62
26.53
48.54
26.81
20.39
11,80
34.06
Ours wr Wanda
62.14
51.62
26.81
50.20
27.57
20.56
11,40
35.76
SparseGPT
43.55
52.71
27.87
48.86
29.34
18.34
13.40
33.44
AlphaPruning SparseGPT
61.62
52.35
28.29
52.25
30.64
19.28
12.00
36.63
Ours SparseGPT
62.26
53.43
29.29
54.22
30.18
19.71
13.40
37.50
Magnitude
48.75
51.62
25.58
48.46
25.76
22.61
14.60
33.91
AlphaPruning w.
Magnitude
43.18
$2.71
25.87
49.49
25.00
22.35
14.40
33.29
Ours ur. Magnitude
56.64
48.01
25.60
49.49
25.08
22.44
12.60
34.27
Wanda
51.74
45.85
26.12
47.59
25.88
20.48
15.00
33.24
0.85
AlphaPruning w. Wanda
38.23
47.29
25.87
50.12
25.76
22.18
12.40
31.69
Ours w. Wanda
49.91
48.38
25.92
50.12
26.09
20.22
11.00
33.09
SparseGPT
37,89
53.07
26.69
50.36
26.98
19.28
11.40
32.24
AlphaPruning SparseGPT
57.61
52.35
27.08
48.70
26.64
19.45
12.20
34.86
Ours w. SparseGPT
55.69
51.62
27.19
52.88
26.98
19.97
12.40
35.25
FT
FT
Tomp Dalance
Temp Balance
Temp Dulance
Temp Bahmce
Temp Balnnc
Dun
Oues
Curs
Our
Curs
1.0
1.5
2.0
255
1.0
1.3
2.0
2,5
1.0
1.5
2.0
25
1.0
13
2.0
2.5
L.0
1.5
2.0
PL Alpha_HHai Hill
PL AL_Alpha Hill
PL Alpha_ Hill
PL Alpha Hill
PL Alpha Hill
a DPOT-Tinn Tiny, 5%
b DPOT-Tiny. Tiny, 10%
DPOT- Tiny, 25%
d DPOT- Tiny, 50%
DPOT-Timy.
100%
FT
FT
Tomp Dalance
Temp Balance
Tomp Dalance
Temp Bahnce
Temp Balanc
Duns
Curs-
Curs
Oun
Cure
1.0
1.5
2.0
2.5
2.0 PL Alpha Hill
1.0
1.5
2.0
2.
2.0
2.5
2.0 PL Alpha Hill
PL Alpha Hill
PL Alpha Hill
PL Alpha Hill
DPOT-Small, 5%
DPOT-Small, 10%
h DPOl-Small, 25%
DPOTT-mall, 50%
DPOT-Small, 100%
Figure Comparing the distribution of Al Hill 1111oo of DPOT-Tiny DPOT-Small different fine-tuning methods and data ratios.
17

---

Eigenspectrum Analysis of Neural Networks without Aspect Ratio Bias
Table 11, Accuracies (%) of a. for zero-shot tasks with unstructured sparsity from 70% 85%. We compare FARMS with uniform pruning ratios and AlphaP P using Magnitude-based pruning, Wanda and SparseGPT.
Sparsity Ratio
Method
BoolQ
RTE
HellaSwag
WinoGrande
ARC-e
ARC-c
OBQA
Mcan
Magnitude
52.87
50.54
26.57
50.83
28.45
20.56
14,80
34,95
AlphaPruning w. Magnitude
61.28
46.93
30.24
50.43
31.23
26.28
21.20
38.23
Ours w. Magnitude
57.16
43.32
33,90
53.43
34.97
26.88
22.40
38.87
Wanda AlphaPruning Wanda
62.08
52.71
30.38
52.41
40.53
17.49
16,00
38.80
0.7
63.43
53.43
42.16
65.59
57.53
28.67
21.40
47.46
Ours Wanda
66.18
52.71
41.94
64.80
57.87
28.58
22.20
47.75
SparseGPT
69.17
52.71
37.25
63.22
51.98
24.83
21.60
45.82
AlphaPruning u. SparseGPT
72.05
54.15
42.44
68.35
55.60
28.50
22.40
49.07
Ours SparseGPT
73.03
54.51
41.98
69.06
57.45
29.27
22.00
49.61
Magnitude AlphaPruning w. Magnitude
45.05
50.90
25.93
50.43
25.93
20.82
13.60
33.24
60.03
50.90
30.10
52.88
28.20
25.60
19.40
38,16
Ours u. Magnitude
60.06
44.04
28.29
52.57
32,53
23.89
22.40
37.68
Wanda
42.51
52.71
27.88
49.64
29.17
17.83
12.00
33.11
0.75
AlphaPruning Wanda
62.17
52.71
36.09
62.67
44.15
23,29
18,00
42.73
Ours Wanda
62,42
52.71
35.41
61.33
46.93
23,46
16.40
42.66
SparseGPT
62.69
52.71
31.66
57.06
40.74
20.56
15.00
40.06
AlphaPruning SparseGPT
64.19
53.07
37.44
64.88
45.79
26.45
17.40
44.17
Ours SparseGPT
65.69
52.71
36.89
64.72
47.43
26.02
17.80
44.47
Magnitude
38.50
53.43
25.95
48.86
25.51
22.10
13.40
32.53
AlphaPruning w. Magnitude
53.30
51.26
26.67
51.70
26.60
23.38
16.20
35.59
Ours u. Magnitude
59.79
48.38
28.32
54.38
27.10
24.23
16.40
36.94
Wanda AlphaPruning wr Wanda
37.86
$2.71
26.64
48.30
27.15
T9.97
13.20
32.26
0.8
62.17
52.71
29.59
55.09
34.93
20.22
14,20
38.42
Ours Wanda
62.17
52.71
30.45
57.38
36.41
21.08
14.40
39.23
SparseGPT
60.80
52.71
28.71
50.28
30.22
18.26
13.00
36.28
AlphaPruning w. SparseGPT
62.20
52.71
31.46
57.62
35.48
19.62
14.40
39.07
Ours SpurseGPT
62.20
52.71
32.37
59.19
37.16
22.18
15,40
40.18
Magnitude AlphaPruning u. Magnitude
38.29
54.51
25.95
49.90
25.21
23.12
14.00
33.01
39.11
51.26
25.47
$0.83
26.39
22.10
16.60
33.11
Ours we Magnitude
41.22
$1.62
25.73
50.43
26.01
22.01
16.00
33.29
Wanda AlphaPruning Wanda
37.83
$2.71
26.09
47.91
25.88
21.16
12,80
32.05
0.85
37.83
52.71
26.72
49.80
26.09
19.62
11.60
32,05
Ours Wanda
38.65
53.79
26.31
48.46
26.77
20.31
12.60
32.41
SparseGPT
38.41
52.71
27.31
50.04
26.30
17.66
12.60
32.15
AlphaPruning u. SparseGPT
62.17
52.71
28.64
51.70
29.67
18.86
13.40
36.73
Ours SparseGPT
62.17
52.71
28.59
54.14
30.09
18.77
12.80
37.04
optimized using parallel processing techniques. In the LLM pruning task, use 8 L40 GPUs for weight analysis and record the PL_Alphe PIAlphh_Hil 1.1 values for each layer. We select four different sparsity ratios, three pruning methods, and two evaluation approaches, yielding a total of 24 experimental configurations per model. However, since weight analysis is performed only once per model, the additional computational cost per experiment remains minimal. When the number of submatrices becomes particularly large, the additional computational overhead introduced by FARMS increases accordingly. In CV and SciMI. experiments, we use single 1.40 GPU and do weight analysis every epoch during the training and fine-tuning. Compared to previous methods, the additional computational cost introduced by FARMS significant.
particularly
We acknowledge ledge that computational cost can indeed be higher than previous methods. But there are certain ways to mitigate the issue, e.g., by using larger window size (like 4096 for LLaMA 7B/5120 for LLaMA 13B) and a limited number of sampling steps. For example, in Table 6, when we use larger window size(such as 4096) and smaller sampling steps(such as 5), our method can still improve model performance and reduce the computational cost.
C.8. LLM Pruning Stability Analysis
The experiment results from (Yin et al., 2024) show that layer-wise pruning LLMs at high sparsity is a challenging optimization problem. If sparsity ratios are not properly allocated, the performance the model can become very unstable. Therefore, here we aim to demonstrate that the performance improvements observed in our experiments are due more
18

---

Eigenspectrum Analysis of Neural Networks without Aspect Ratio Bias
Uni-
Uni- form
form
form
Alpha-
Alpha- Pruning
Alpha- Praning
Alpba- Proning
Prining
PL PL_Alpha Hill
PL Alpha Hill
PL Alpha Hill
PL Alpha Hill
a LLaMA-7B, SR=0.7
b LLaMA-7B, SR=0.75
c LLaMA-7B, SR=0.8
d LLaMA-7B, SR=0.85
Uni-
Uni-
form
Alpha- Prining
Alpha- Proning
Alpha- Proning
Alpha- Proning
2.0
5.0
2.0
3.0
PL Alpha Hill
PL Alpha Hill
PL Alpha Hill
PL Alpha Hill
LLaMA-13B, SR=0.7
FLLaMA-13B, SR=0.75
g LLaMA-I3B, SR=0.8
LLBMA-13B, SR=0.85
Figure 1.. Comparing the distribution Al Hill LLaMA-7B LLaMA-13B in different layer-wise strategies and sparsity ratios (shown as "SR"). The pruning method is SparseGPT.
Table 12. Computation cost for each experiment.
Model
Method
Weight Analysis Time (seclepoch)
TB
0.887
ResNet
Ours
1.054
Model
Experiment Settings
Weight Amalysis Time (seclexperiment)
TB
1,761
ResNet
AlphaPruning Ours (4096, 5) Ours (2000) 16)
2.65
Ours
1,916
LLaMA-7B
6.08
TB
1.048
VGG 16
134.5
1,200
AlphaPruning (5120, 5) Ours: (2000, 15 15)
6.23
VGG 19
TB
1,421
LLaMA-13B
1.578
163.26
DPOT Tiny
TB_Sig
0.238
DPOT:Small
TB_Sig
1.255
1.340
accurate layer-wise analysis, rather than accidental factors such as random seeds.
We follow the experiment settings from Table 6 and visualize the sparsity ratio assigned to each transformer block in LLaMA-7B under different sampling settings in Figure 13. This visualization indirectly reflects the degree of heavy- tailedness in the ESD of each transformer block's model layer as analyzed by our method. We observe that the differences sparsity ratio assignments across these settings are not significant. However, the assignment from the best FARMS setting still distinct from the worst setting, which explains the performance difference in Table 6.
D. Training Quality Analysis
D.1. Previous work on training quality
Previous work on HT-SS has established that the heavy-tailness ESDs is strongly correlated with the test accuracy of models (Martin & Mahoney, 2021; Martin et al., 2021). While this does not imply that "training quality" is identical to "test accuracy," " the correlation between heavy-tailedness and test accuracy has been used to justify HT-SR metrics. Therefore, improving test accuracy or similar performance metries (e.g., perplexity) remains our primary goal.
Although previous work on HT-SR does not explicitly define "training quality," several related quantities have been mentioned: (1) strong correlation between weight elements (Martin & Mahoney, 2021; Martin et al., 2021) and (2) feature spikes and their alignment with the target teacher model (Ba et al., 2022; Wang et al., 2023). The feature spike, analyzed in the context of a single-index teacher and a two-layer student model (Ba et al., 2022: Wang et al., 2023), is approximately a rank-one update to the model (in the limit infinite matrix size with fixed aspect ratio) and also persists after matrix subsampling. This is because the specific form of the rank-one update makes it cover the whole matrix with probability one.
19

---

Eigenspectrum Analysis of Neural Networks without Aspect Ratio Bias
Training Epochs
50
100
150
200
0.15
Leanring Rate
Leanring Rate
0.15
Leanring Rate
0.15
Leanring Rate
0.15
00.10
10.10
50.10
0.10
0.05
0.05
0.05
0.05
0.00
0.00
0.00
0.00
20
10
20
30
20
30
20
30
Layer index
Layer index
Layer index
Layer index
5.0
5.0
3.0
4.0-
4.0
Alpha
Alpha
Alpha
Alpha
2.0-
2.0
1.0
10
1.0
10
30
1.0
0
20
30
Layer index
Layer index
Layer index
Layer index
ResNet 34, TB(no L.))
ResNet 34, L.+TBB
ResNet 34, Ours(no L.S)
d ResNet 34, LS+Ours
Leanring Rate
0.06
Leanring Rate
0.08
Leanring Rate
0.08
Leunring Rate
0.08
0.06
0.06
0.06
0.04
0.04
0.04
0.02
0.02-
0.00
0.00
0.00
0.00
19
16
1b
Layer index
Layer index
Layer index
Layer index
5.04
4.0-
Alpha
Alpha
Alpha
Alpha
3.0
2.0
2..
1.0
1.0
1.0
10
Layer index
Layer index
Layer index
Layer index
e VGG 16, TB(no LS)
VGG 16, LS+TB
VGG 16, Ours(no LS)
VGG 16, LS+Ours
Figure 12. Visualization of layer-wise Learning Rate and PL Alpha Hill (Alpha) training. The top of each subtigure presents the evolution of layer-wise learning rates and the bottom rows presents PI. A1 Alpha Hil during training for different optimizing configurations of ResNet 34 and VGG on CIFAR 100. The color gradient represents training epochs, transitioning from dark purple (carly epochs) to yellow (later epochs).
D.2. A toy experiment to measure training quality
We designed a toy experiment to test the correlation between "training quality" and the new HT-SR metric measured using FARMS. Following (Ba et al., 2022; Wang et al., 2023), we use a single-index teacher to generate signals to train a two-layer student. The first layer of the student model is a weight matrix, while the second layer a weight vector. We only update the weight of first layer. To measure "training quality", during training, and the ground truth target vector of the teacher model similar to (Ba et al., 2022; Wang et al., 2023), and we define this alignment to be the "training quality" of the student model.
measure the alignment between the weight matrix
Throughout the training process, we select the student network checkpoint with the highest alignment and report both the alignment value and the PLAAphaaHHil value. We then vary the sizes of the student model with different weight matrix aspect ratios on a fixed input dimension 500 to conduct multiple experiments. Each experiment provides one
20

---

Eigenspectrum Analysis of Neural Networks without Aspect Ratio Bias
Farms Best
Farms Worst
Farms Average
0.9
Sparsity Ratio
0.8
0.7
0.6
5
10
15
20
25
30
Transformer Block Index
Figure 13. Block-wise Sparsity Ratio for LLaMA-7B assigned by FARMS. We set FARMS with four different window sizes {500, 1000, 2000, 4096} and four sampling setps (5, 15, 20}. The pruning method SparseGPT and the base sparsity ratio is 0.8. The FARMS Best indicates using this sparsity ratio distribution makes minimal perplexity 39.42:13.86 while the FARMS worst makes largest perplexity 13.53. distribution of block-wise sparsity ratios of FARMS Average represents mean under the x settings, with the standard deviation intervals also illustrated.
PL Al Alaha Hill value and one alignment value. The multiple experiments (conducted using varying sizes) produce one curve for PLAlpha and one curve for the alignment value.
We then plot the two curves using both existing methods for estimating PL.Alpha and our method FARMS. As shown in Figure 14, FARMS reveals clear negative correlation between the two curves: the better the training quality, the larger the alignment, and the smaller the PL.Alpha.Hil Hill. However, for the existing method, due to the aspect ratio bias, the correlation is incorrect.
E. Broader Discussion
In this section, we discuss more related literature as well some explanations regarding aspect ratio bias and issues related to our method.
E.1. Relationship with Matrix Shape
One should recognize that the numerical value of the same property measured in different network layers can naturally have different scales, and thus should not be directly compared without proper normalization. In some real-world application scenarios, such as in LLM Pruning (Lu et al., 2024; Liu et al., 2025a), this factor need to be considered. When the network size changes, hyperparameters also need to be changed accordingly. For example, works such as Tensor Programs IV (Yang & Hu, 2021) and Tensor Programs V (Yang et al., 2021) have explored how layer sizes affect the optimal scaling of weight initializations and learning rates. Tensor Programs IV introduced the Maximal Update Parametrization (uP) to ensure feature learning by carefully choosing parameter scaling rules based on these sizes, addressing how existing parametrizations might otherwise collapse to kernel regimes. Building on this, Tensor Programs V demonstrated that μP can enable zero-shot hyperparameter transfer, where hyperparameters tuned on smaller models remain optimal for significantly larger ones. In this context, "shape" awareness pertains to designing size-dependent hyperparameters that maintain stable and effective (non-kernel) training dynamics.
Nevertheless, our work is fine-grained in that we consider the aspect ratios of different layers (number of rows versus columns). We demonstrate that varying aspect ratios can artificially stretch or compress the ESD. This distortion confounds the interpretation of heavy-tailed (HT) metrics. To mitigate this measurement bias, we propose FARMS to analyze the average ESD of submatrices sampled a fixed aspect ratio. This approach provides a normalized HT metric, enabling more
21

---

Eigenspectrum Analysis of Neural Networks without Aspect Ratio Bias
reliable comparisons of such spectral diagnostics across different layers within given network. Thus, while focusing on shape-aware parametrization for training stability and transfer, our contribution lies in shape-aware analysis technique (FARMS) aimed at correcting measurement bias in spectral diagnostics.
E.2. Why does the FARMS preserve important spectral property about the original matrix?
The goal of measuring heavy-tailness in HT-SR is to evaluate the strength of correlations introduced by training, as established in previous work (Martin & Mahoney, 2021). However, when subsample single submatrix and measure correlations only within that submatrix, some correlations between elements in the subsampled matrix and those outside it are inevitably lost. This motivates our approach of using multiple submatrices to capture a broader range of correlations.
E.3. Does this approach introduce additional bias?
Our approach could be viewed as introducing form of "bias"; however, we interpret this more specifically as achieving partial coverage of the entire matrix. Conceptually, this is similar to the principle behind bootstrap sampling in random forests, where multiple samples, each with potentially limited coverage, are used collectively to mitigate the effects of this partial view and improve overall model robustness.
Further justification for this perspective comes from recent work (Wang al., 2023; Kothapalli et al., 2025) that aims to theoretically quantify heavy-tailedness. These studies interpret heavy-tailedness as the accumulation and evolution of feature spikes in the ESD that align with the teacher model's features. Critically, these feature spikes are characterized as being approximately rank-one updates to the original matrix Because a rank-one one component inherently covers the whole matrix, sampling a submatrix will, with high probability, capture that rank-one component. Therefore, this subsampling process is unlikely to miss the feature spikes, which are identified by previous work as the cause of the heavy-tail structure. We believe this provides substantial evidence that FARMS can preserve important spectral information, specifically as measured by these feature spikes in the ESD.
FARMs
Baseline [PL. Alpha
3.00
Training Quality
0,550
Similarity Scores
FARMs PLL Alpha
2.1
Training Quality
0,550
Similarity Scores
12.75
0.525
0.525
2.50
0.500
2.0
0.500
2.25-
0.475
0.475
1.9
2.00
1.5
1.8
8
1.3
1.5
1.8
2
4
6
Matrix Aspect Ratio
Matrix Aspect Ratio
Bascline
b FARMS
Figure 14. Compare the PL.Alhss.Hill from FARMS and Baseline in measuring the training quality of single layer. The Correlation Coefficient between FARMS and training quality is -0.89 for baseline is -0.51. can find that FARMS can measure the training quality more precisely.
F. Hyperparameter Adjustment
In this section, we report the hyperparameters that we use in the experiments shown in the main paper (Section 4).
First, we report the common hyperparameters shared by Image Classification experiments (Section 4.3): the optimizer SGD, batch size 128. number of total training epochs 200. weight decay 5c-4, and momentum 0.9. For each experiment setting, repeat experiments with three random seeds {43, 37, 13). We also report the mean and standard deviation of the test accuracy across these seeds. In Table 13, we report the details of experiments for each model and method. We use the same learning rate range from (Zhou et al., 2024) and we expand the scaling ratio range into (00.1, 1.9), (0.2, 1.8), (0.3, 1.7). (0.4, 1.6), (0.5, 1.5), (0.6, 1.4), (0.7, 1.3), (0.8, 1.2), (0.9, 1.1)] nine choices.
Second, we provide the hyperparameters used in experiments of LLM pruning and SeiML. We follow the common hyperparameter settings as described in Lu al. (2024); Liu et al. (2024). See more details for other hyperparameters like
22

---

Eigenspectrum Analysis of Neural Networks without Aspect Ratio Bias
in LLM pruning and scaling ratios in SciML in Table 14.
Finally, we report the detailed matrix subsampling settings used in every model in Table 15. We cannot very small window size or sampling steps because doing so may not cover the entire matrix. Conversely, selecting a very large size would result in too much overlap between sampled matrices, For ResNet, VGG and DPOT series models, we use the minimum dimension of the weight matrices to construct the sampled submatrices based on the parameter Q. We also select [m/n] for submatrices number, where m, n is the dimension of weight matrix, m 2 n. But for the final layer of ResNet and VGG models, we select nine submatrices based on experiments results. For LLaMA series models, we apply sliding window sampling using multiple moderately sized submatrices, resulting in smoother PL.Alpha.Hii estimation.
Table 13. Parameter settings of the experiment reported in Section 4.3.
Model
Method
Initial Learning Rate
Scaling Ratio
Test Accuracy
(31,52)
CAL
0.05, 0.1, 0.15
78. 78.23=0.087
TB LS)
0.1
(0.6, 1.4)
78.76=0.111
ResNet 18
LS+TB
(0.2, 1.8)
7 10.10
Ours(no LS)
0.1
(0.1. 1.9)
79.49=0.0e
LS+Ours
0.1
(0.1, 1.9)
79.53=0.177
CAL
0.05, 0.1, 0.15
78.99=0.137
TB(no LS)
0.1
(0.5, 1.5)
79 79.64=0.029
ResNet 34
LS+TB
0.1
(0.3, 1.7)
80.00
Ours(no LS)
0.1
(0.2, 1.8)
80.17=0.213
LS+Ours
0.1
(0.3, 1.7)
80.20
CAL
0.025, 0.05, 0.1
74.30=0.078
TB(no LS)
0.05
(0.6, 1.4)
74.43=0.158
VGG
LS+TB
0.05
(0.3, 1.7)
75.19_0.131
Ours(no LS)
0.05
(0.2, 1.8)
75.36=0.118
LS+Ours
0.05
(0.2. 1.8)
75.15=0247
CAL
0.025, 0.05, 0.1
73.11=0.113
TB(no LS)
0.05
(0.6, 1.4)
73.22_0027
VGG
LS+TB
0.05
(0.2, 1.8)
74.19 20159
Ours(no LS)
0.05
(0.2, 1.8)
74.28= 0.392
LS+Ours 0.05
(0.4. 1.6)
73.99=0.300
Table 14. Hyperparameters for LLaMA and DPOT models. (Left) The range of used for LLM pruning. (Right) Learning rate and scaling ratio settings for DPOT series models different subsampling ratios.
Model
DPOT-Tiny
DPOT-Small
Sparsity
Hyperparameters
Learning
Scaling Ratio
Learning Rate
Scaling Ratio
0.7
0.1, 0.2, 0.3, 0.5,
5%
2.5e-4
(1.0, 1.0)
lc44
(1.0, 1.0)
0.1, 0.3, 0.4, 0.5,
2.5e-4
(1.0, 1.0)
le-4
(1.0, 1.0)
0.8
0.1. 0.2, 0.25, 0.3,
25%
2.5e-4
(1.0, 1.0)
2.5c-4
(1.0, 1.0)
0.1, 0.15, 0.2, 0.25,
50%
5c-4
(1.0, 1.0)
2.5c-
(1.0, 1.0)
100%
5e-4
(1.0, 1.0)
2.5c-4
(1.0, 1.0)
23

---

Eigenspectrum Analysis of Neural Networks without Aspect Ratio Bias
Table 15. Subsampling Hyperparameters for different models.
Model
Aspect Ratio(Q)
Window Size
Submatrices Number
ResNet 18/34, VGG 16/19
1.0
Minimum Dimension
m//J.9 9
DPOT-Tiny/Small
1.0
Minimum Dimension
[m/n]
1.0
2000
15 15
LLaMA-7B
1.0
2000
10 10
1.0
1000
10 10
LLaMA-13B
1.0
2000
15 15
24


Reviews:
[{"review_id": "TyJtZXR50q", "reviewer": "Reviewer_rhqc", "summary": "This paper introduces an approach called FARMS (Fixed Aspect Ratio Matrix Subsampling) that addresses a current deficiency of approaches that rely on quantifying the heavy-tail degree of an empirical spectral density (ESD) without accounting for matrix shape.  In particular, the authors observe that random Gaussian matrices converge to distributions with differing levels of heavytail-ness depending on the ratio $m/n$ for an $m\\times n$ matrix but this same effect is not captured in algorithms that rely on heavy-tail self regularization theory (HTSR) to capture how well trained certain layers are.  FARMS accounts for this by computing a weight matrix's spectral density as the averaged spectral density of subsampled matrices of fixed dimension.  By using a fixed dimension for the subsampling matrix used to compute the ESD for all matrices, the impact of the ratio of $m:n$ on ESD is controlled.  The authors adopt FARMS for computing the ESD for three HTSR-based algorithms and show FARMS improves performance over the standard approach for computing ESD.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful and constructive comments. We have addressed your comments as follows.\n\n### Weakness 2\nWhile we agree that error bars would help show statistical significance, we respectfully point out that several representative works [1-3] on LLM pruning did not provide error bars in main results. Therefore, we did not provide error bars in the submitted paper. However, we also noticed that both SparseGPT [1] and Wanda [2] use calibration data to estimate input statistics, so we sample different calibration sets with 128 samples using six different seeds [0, 1, 2, 3, 4, 5]. We evaluate FARMS(our method) and AlphaPruning and provide the mean value and standard division(STD) of perplexity in [Table 1](https://github.com/Anonymous-For-Submission-code/FARMs-ICML-Rebuttal). We find that FARMS achieves lower perplexity consistently. For example, the perplexity of LLaMA-7B reduces from **96.02 ± 1.59** to **79.42 ± 3.86** using the SparseGPT pruning method at a 0.8 sparsity ratio. We also found that FARMS achieve lower STD in most settings.\n\n[1] Frantar, Elias, and Dan Alistarh. \"Sparsegpt: Massive language models can be accurately pruned in one-shot.\" In International Conference on Machine Learning, pp. 10323-10337. PMLR, 2023.\n\n[2] Sun, Mingjie, Zhuang Liu, Anna Bair, and J. Zico Kolter. \"A simple and effective pruning approach for large language models.\" arXiv preprint arXiv:2306.11695 (2023).\n\n[3] Cheng, Hongrong, Miao Zhang, and Javen Qinfeng Shi. \"A survey on deep neural network pruning: Taxonomy, comparison, analysis, and recommendations.\" IEEE Transactions on Pattern Analysis and Machine Intelligence (2024).\n\n\n### Weakness 3\nFirstly, compared to existing HTSR adaptive methods, our approach demonstrates stronger robustness. Figure 3 in the paper shows that FARMS can achieve better performance without Layer Selection, a heuristic tuning method in TempBalance (TB) to avoid the aspect ratio bias. Secondly, HTSR methods, such as TB and AlphaPruning, are strong baselines. For example, in layer-wise learning rate assignment, TB beats the state-of-the-art optimizers and schedulers like SGDR, SGDP, LARS, and Lookahead.\n\n### Other Comments or suggestions\nPlease refer to our response to Weakness 1 and Question 1 from Reviewer C2rY for more details. We cannot use a very small window size or sampling steps because doing so may not cover the entire matrix. Conversely, selecting a very large size would result in too much overlap between sampled matrices. A precise theoretical characterization may be beyond the scope of this paper, but we will include a detailed discussion in the revised version.\n\n\n### Question 1 & Weakness 1 \n\nFirstly, Layer-wise pruning LLMs at high sparsity is a challenging optimization problem. If sparsity ratios are not properly allocated, the performance of the model can become very unstable[4]. And the sensitivity shown in [Table 1](https://github.com/Anonymous-For-Submission-code/FARMs-ICML-Rebuttal) is mainly from that. \n\nSecondly, we provide the additional results for validating the stability of our method. In  [Table 2](https://github.com/Anonymous-For-Submission-code/FARMs-ICML-Rebuttal), we repeat our experiments with six random seeds [0, 1, 2, 3, 4, 5] and provide the mean and STD of perplexity. We can find that except for cases where a small window size like 500 is used, the results of other settings are better than baseline, and the differences in perplexity corresponding to different parameters are not particularly significant. \n\nThirdly, we show that the FARMS evaluation of the ESD is stable, which demonstrates that most sensitivity is from pruning itself. In [Figure 1](https://github.com/Anonymous-For-Submission-code/FARMs-ICML-Rebuttal), we plot the layer-wise sparsity ratios assigned by FARMS across 4 × 4 different window sizes and sampling steps. We observe that the differences in sparsity ratio assignments across these settings are not significant. However, the assignment from the best FARMS setting is still distinct from the worst setting, which explains the performance difference.\n\n[4] Yin, Lu, You Wu, Zhenyu Zhang, Cheng-Yu Hsieh, Yaqing Wang, Yiling Jia, Gen Li et al. \"Outlier weighed layerwise sparsity (owl): A missing secret sauce for pruning llms to high sparsity.\" arXiv preprint arXiv:2310.05175 (2023).\n\n### Question 2\nWe are unsure what \"parametric adjustment\" means and kindly request the reviewer to clarify. If it refers to the \"change of variable\" method in the Marchenko–Pastur pdf function, this is challenging because the ESD transitions smoothly from Marchenko–Pastur to heavy-tailed in a way that is difficult to quantify precisely. If \"parametric adjustment\" instead refers to other hyperparameter assignment approaches, those are orthogonal to FARMS and can be combined with it.\n\n### Additional experiments\n\nExperiments to answer all reviewers' questions can be found [here](https://github.com/Anonymous-For-Submission-code/FARMs-ICML-Rebuttal).", "claims_and_evidence": "The authors demonstrate sufficient evidence that accounting for the matrix dimension when computing the ESD for subsequent heavy-tail analysis improves the performance and robustness of HTSR algorithms developed for adaptive learning rate optimization and pruning.", "methods_and_evaluation_criteria": "The evaluation is relevant and sufficient for demonstrating the impact of accounting for the shape of a matrix.  The authors evaluate FARMS applied to three different HTSR algorithms in three domains (computer vision, LLM pruning, and SciML) and show improvements for all three.", "theoretical_claims": "There were no substantial theoretical claims.", "experimental_designs_or_analyses": "I found the conclusions drawn from the experiment design and analyses valid in support of FARMS.", "relation_to_broader_scientific_literature": "The contribution of this paper is highly relevant to applications of HTSR and techniques that rely on accurately capturing the ESD.", "essential_references_not_discussed": "I found the discussion of prior work sufficient.", "other_strengths_and_weaknesses": "Strengths:\n- While the idea of accounting for the weight matrix ratio is simple and straightforward, the experiments demonstrate FARMS to offer consistent improvement over prior HTSR approaches across the board.\n- I found the paper clear and well written.  \n- FARMS improves the robustness of TempBalance as shown in Figure 3 where FARMS does not need additional Layer Selection to improve performance.  \n\nWeaknesses:\n- FARMS seems like it can be sensitive to the window size and sampling steps hyperparameter (Table 5) which could make it difficult to apply in practice.  \n- Error bars not provided for the LLM pruning experiments which make it difficult to gauge the statistical significance of the results in Table 1 and Table 5.  \n- The improvement from FARMS is somewhat incremental especially compared to the improvement from using HTSR adaptive learning rate relative to baseline.", "other_comments_or_suggestions": "It seems like there could be more of a theoretically grounded adjustment for this ratio that would preclude the need for window size, sliding window, and subsampling number hyperparameters of FARMS.  Have you given this direction some thought?", "questions_for_authors": "- Why do you think the LLM pruning results were more sensitive to sliding window and submatrices number?\n- As seen in Table 4, it makes sense intuitively that a square matrix would give most ratio-agnostic ESD.  Do you think there's a parametric adjustment that can be made instead of taking a subsampling approach?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5Foqx09keI", "reviewer": "Reviewer_ZdCj", "summary": "The paper studies the problem of eigenspectrum analysis of DNN weight matrices and its relationship with model quality in terms of the training process. The paper proposes Fixed-Aspect-Ratio Matrix Subsampling (FARMS) to address the aspect ratio bias in existing Heavy-Tailed Self-Regularization (HT-SR) eigenspectrum analysis, which enables better computation of HT metrics. The idea is to compute the HT metrics based on the average of a series of subsampled sub weight matrices with fixed aspect ratios. The FARMS further induces improved algorithm for layer-wise hyperparameter tuning, with extensive experiments demonstrating its effectiveness and improvement.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful and constructive comments. We have addressed your comments as follows.\n\n### Weakness 1 and Question 1 \n\n- **Previous work on training quality**\n\nPrevious work on HTSR has established that the heavy-tailness of ESDs is strongly correlated with the test accuracy of models [1][2]. While this does not imply that \"training quality\" is identical to \"test accuracy,\" the correlation between heavy-tailedness and test accuracy has been used to justify HTSR metrics. Therefore, improving test accuracy or similar performance metrics (e.g., perplexity) remains our primary goal.\n\nAlthough previous work on HTSR does not explicitly define \"training quality,\" several related quantities have been mentioned: (1) strong correlation between weight elements [1][2] and (2) feature spikes and their alignment with the target teacher model [3][4]. The feature spike, analyzed in the context of a single-index teacher and a two-layer student model [3][4], is approximately a rank-one update to the model (in the limit of infinite matrix size with fixed aspect ratio) and also persists after matrix subsampling. This is because the specific form of the rank-one update makes it cover the whole matrix with probability one.\n\n- **A toy experiment to measure training quality**\n\nWe designed a toy experiment to test the correlation between \"training quality\" and the new HTSR metric measured using FARMS. Following [3][4], we use a single-index teacher to generate signals to train a two-layer student. The first layer of the student model is a weight matrix, while the second layer is a weight vector. Following [3][4], we only update the weight of first layer. To measure “training quality”, during training, we measure the alignment between the weight matrix and the ground truth target vector of the teacher model similar to [3][4], and we define this alignment to be the \"training quality\" of the student model.\n\nThroughout the training process, we select the student network checkpoint with the highest alignment and report both the alignment value and the PL_Alpha value (the HTSR metric). We then vary the sizes of the student model with different weight matrix aspect ratios on a fixed input dimension 500 to conduct multiple experiments. Each experiment provides one PL_Alpha value and one alignment value. The multiple experiments (conducted using varying sizes) produce one curve for PL_Alpha and one curve for the alignment value.\n\nWe then plot the two curves using both existing methods for estimating PL_Alpha and our method FARMS. As shown in [Figure 2](https://github.com/Anonymous-For-Submission-code/FARMs-ICML-Rebuttal), FARMS reveals a clear negative correlation between the two curves: the better the training quality, the larger the alignment, and the smaller the PL_Alpha. However, for the existing method, due to the aspect ratio bias, the correlation is incorrect.\n\nFurthermore, in Appendix A.1, we compared FARMS with existing ways of measuring heavy-tailedness. For i.i.d. initialized weights of varying sizes, FARMS produces a constant value of heavy-tailedness degree, whereas existing methods change with matrix size, showing an aspect ratio bias. In this setting, all matrices are equally undertrained since they are just initialized.\n\n- **Summary of this question**\n\nWe understand that to formally define \"training quality\" requires substantial novel work. Our goal is not to reinvent the wheel or claim that our method measures training quality better than prior work. Instead, we aim to correct a specific oversight in prior work: the aspect ratio bias. In other words, our goal is to mitigate the aspect ratio bias without sacrificing the ability to measuring heavy-tailedness.\n\n\n[1] Martin and Mahoney. \"Implicit self-regularization in deep neural networks: Evidence from random matrix theory and implications for learning.\" JMLR 2021.\n\n[2] Martin, Peng and Mahoney. \"Predicting trends in the quality of state-of-the-art neural networks without access to training or testing data.\" Nature Communications 2021.\n\n[3] Ba et al. \"High-dimensional asymptotics of feature learning: How one gradient step improves the representation.\" NeurIPS 2022.\n\n[4] Wang et al. \"Spectral evolution and invariance in linear-width neural networks.\" NeurIPS 2023.\n\n### Weakness 2\n\nThe computational cost can indeed be higher than previous methods, and we have acknowledged that. There are certain ways to mitigate the issue, e.g., by using a larger window size (like 4096 for LLaMA 7B/5120 for LLaMA 13B) and a limited number of sampling steps. In [Table 2](https://github.com/Anonymous-For-Submission-code/FARMs-ICML-Rebuttal), when we use larger window size(such as 4096) and smaller sampling steps(such as 5), our method can still improve model performance and reduce the computational cost.\n\n### Additional experiments\n\nExperiments to answer all reviewers' questions can be found [here](https://github.com/Anonymous-For-Submission-code/FARMs-ICML-Rebuttal).", "claims_and_evidence": "Yes, the claims made in the submission are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "Yes, the proposed methods make sense for the problem studied.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The reviewer has checked the experimental designs and analyses.", "relation_to_broader_scientific_literature": "The paper directly contributes to the line of works based on Heavy-Tailed Self-Regularization (HT-SR) [1, 2], which further induces improved algorithm for the task of layer-wise hyperparameter tuning.\n\n**References:**\n\n[1]Mahoney, M. and Martin, C. Traditional and heavy tailed self regularization in neural network models. In Proceedings of the 36th International Conference on Machine Learning\n\n[2] Martin, C. H. and Mahoney, M. W. Implicit self-regularization in deep neural networks: Evidence from random matrix theory and implications for learning. Journal of Machine Learning Research, 22(165):1–73, 2021.", "essential_references_not_discussed": "To the best knowledge of the reviewer, there is no lacking related works.", "other_strengths_and_weaknesses": "**Strengths:**\n1. The idea of matrix subsampling to debias the estimation of the heavy-tailedness metrics is new and interesting. \n2. The layer-wise hyperparameter tuning algorithms based on the new algorithm for HT metric calculation shows promising results on a range of tasks compared with previous SOTAs.\n3. The experiment design is quite extensive and solid.\n\n**Weaknesses:** \n1. To me the main drawback is that it is not clear why in principle the HT statistics of subsampled matrix is a better estimator of the quality of the trained model, despite the fact that the aspect ration of the subsampled matrix has beed fixed. Please also see my question in the **Questions For Authors** part. Is there any theoretical justifications of such a method?\n2. Also, as a minor drawback, the computational cost could be large compared with previous SOTA methods (Appendix C.5).", "other_comments_or_suggestions": "Please see the **Questions For Authors** part.", "questions_for_authors": "1. How to design experiments that can directly evaluate the correlation between the *HT metrics* (calculated by different methods, i.e., existing ones and the FARMS method) and the so called *\"training quality of each layer\"*? How to define the training quality of *different layers*? To me, this seems like the foundation that the newly proposed HT metric calculation algorithm can result in better performance in terms of **layer-wise** hyper-parameter tuning. Since there appears no theory for the design of the algorithm, such an experiment would make the overall methodology more convincing. The authors show in Section 4.3 that by using the FARMS assisted layer-wise hyperparameter tuning, the final testing accuracy improves over using the existing HT metrics calculation method. But this does not directly showcase that the FARMS method gives better indication of how each single layer is trained.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QPuWgtv3yv", "reviewer": "Reviewer_C2rY", "summary": "This paper proposes FARMS, a method for reducing the bias in the estimation of heavytailness (HT) metrics due to the aspect ratio of the weight matrix. FARMS samples submatrices with a fixed aspect ratio and averages the sampled empirical spectral density (ESD). Empirically, using HT metrics estimated by FARMS leads to improved downstream performance in pruning and choosing layerwise learning rates.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful and constructive comments. We have addressed your comments as follows.\n\n### Broader Literature\nThank you for suggesting references [1, 2]. Tensor Programs IV [1] shows that standard parametrizations can collapse to the kernel regimes in the infinite-width limit. It proposes the Maximal Update Parametrization (µP) to ensure feature learning by carefully choosing parameter scaling rules based on layer dimensions. Tensor Programs V [2] further shows that µP enables zero-shot hyperparameter transfer: hyperparameters tuned on smaller models remain optimal for much larger ones. In this context, \"shape\" awareness relates to how fan-in/fan-out dimensions affect optimal scaling of initializations and learning rates to maintain stable training dynamics.\n\nOur paper, on the other hand, focuses on how the aspect ratio (number of rows vs. columns) of an individual weight matrix biases the measurement of empirical spectral density (ESD). We show that different aspect ratios can artificially stretch or compress the ESD, which confounds the interpretation of HT metrics like the Power Law (PL) exponent Alpha as indicators of training quality. FARMS targets this bias. By analyzing the average ESD of submatrices sampled at a fixed aspect ratio, FARMS provides a normalized HT metric for more reliable comparison across layers of diverse shapes within a given network. \n\nThus, while [1][2] use shape-aware parametrization (like µP), our work uses a different shape-aware analysis technique (FARMS) to correct measurement bias in spectral diagnostics. We will discuss this in detail in the revised paper.\n\nReferences\n\n[1] Yang, Greg, and Edward J. Hu. \"Tensor programs iv: Feature learning in infinite-width neural networks.\" International Conference on Machine Learning. PMLR, 2021.\n\n[2] Yang, Greg, et al. \"Tensor programs v: Tuning large neural networks via zero-shot hyperparameter transfer.\" arXiv preprint, 2022.\n\n\n### Weakness 1 and Question 1\n\nFirst, see our previous response to clarify what aspect ratio bias is and explain why FARMS addresses it. Then, we will answer your question in more detail.\n\n- Why does the FARMS subsampling approach preserve important spectral property about the original matrix?\n- Does it introduce additional bias?\n\nThe goal of measuring heavy-tailness in HTSR is to evaluate the strength of correlations introduced by training, as established in previous work [1]. However, when we subsample a single submatrix and measure correlations only within that submatrix, some correlations between elements in the subsampled matrix and those outside it are inevitably lost. This motivates our approach of using multiple submatrices to capture a broader range of correlations.\n\nDoes this approach introduce additional bias? We believe it does, but we view this \"bias\" more as a form of partial coverage of the entire matrix. At a high level, this is conceptually similar to bootstrap sampling in random forests—using multiple samples to mitigate the effects of limited coverage.\n\nRecent work, such as [2][3], aims to theoretically quantify heavy-tailedness by interpreting it as the accumulation and evolution of feature spikes in the ESD that align with the teacher model's features. These feature spikes are approximately rank-one updates to the original matrix, and with high probability, sampling a submatrix will capture that rank-one component because that component covers the whole matrix (so the subsampled matrix contains that rank-one component). Therefore, sampling a submatrix will not miss the feature spikes, which are believed by previous work [2][3] to cause the heavy-tail structure. We believe this provides further justification that FARMS can preserve important spectral information, measured by the feature spikes in the ESD.\n\nFinally, we direct the reviewer’s attention to the experiment in Appendix A.1. In this experiment, we demonstrate that for i.i.d. initialized weights of varying sizes, our method produces a constant measure of heavy-tailedness, whereas existing methods vary with matrix size, exhibiting an aspect ratio bias. While FARMS may introduce partial coverage of the entire matrix, it does so uniformly across matrices of all sizes, preventing any layer-shape-dependent bias.\n\n[1] Martin and Mahoney. \"Implicit self-regularization in deep neural networks: Evidence from random matrix theory and implications for learning.\" JMLR 2021.\n\n[2] Wang et al. \"Spectral evolution and invariance in linear-width neural networks.\" NeurIPS 2023.\n\n[3] Kothapalli et al. \"Crafting heavy-tails in weight matrix spectrum without gradient noise.\" arXiv preprint.\n\n### Other Comments Or Suggestions \n\nThank you for the suggestion. The abbreviation was used in the abstract without a definition. We will include the definition of FARMS in the abstract.\n\n### Additional experiments\n\nExperiments to answer all reviewers' questions can be found [here](https://github.com/Anonymous-For-Submission-code/FARMs-ICML-Rebuttal).", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The experiments in Section 4 are sound.", "relation_to_broader_scientific_literature": "The paper highlights the importance in recognizing that the numerical value of the same property measured in different layers can naturally have different scales and should not be directly compared without proper normalization. This is a basic but important idea that deserves more attention from the community. Similar considerations around the different shapes of the weight matrices in each layer have led to impactful theoretical and practical advancement in areas such as hyperparameter transfer and infinite-width limits of neural networks with feature learning [1, 2].\n\n[1] Yang, Greg, and Edward J. Hu. \"Tensor programs iv: Feature learning in infinite-width neural networks.\" International Conference on Machine Learning. PMLR, 2021.\n\n[2] Yang, Greg, et al. \"Tensor programs v: Tuning large neural networks via zero-shot hyperparameter transfer.\" arXiv preprint arXiv:2203.03466 (2022).", "essential_references_not_discussed": "None that I'm aware of.", "other_strengths_and_weaknesses": "Strength: The paper is well-written, and experimental results are strong and well-presented. The core idea of mitigating the aspect ratio bias makes intuitive sense.\n\nWeakness: While the proposed method empirically improves the downstream performance of the estimated HT metrics, it does not provide a theoretical justification for the approach's soundness. For example, while subsampling the matrices reduces the bias due to its aspect ratio, does it introduce additional biases since we no longer work with the full weight matrix which is what we ultimately care about?", "other_comments_or_suggestions": "The abbreviation FARMS is used in the title without being defined.", "questions_for_authors": "1. Is there any theoretical motivation for the particular subsampling strategy in FARMS? Can you quantify its effect and why it should preserve important spectral properties about the original matrix?\n\n2. If the subsampling strategy is not theoretically well-motivated, have you tried alternative approaches to mitigating the aspect ratio bias?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces a method based on weight matrix subsampling that removes the spectral bias induced by matrix rectangularity, thereby facilitating more robust and comparable measurements of matrices' spectral heavy-tailedness, a property previously shown to correlate with model performance in some circumstances.

The reviewers found the paper to be well-written and the experiments thorough, with the proposed method delivering consistent downstream performance gains relative to the baselines considered; on the other hand, there were some concerns about the theoretical justification. Given the empirical nature of the paper, and the slightly loose motivations regarding the heavy-tailed metrics in the first place, it seems entirely reasonable to focus on the experimental results rather than a precise formulation of the theory behind the subsampling approach. As such, I think the ICML community will find this paper valuable and I recommend acceptance.
Decision: Accept (poster)