ICML 2025

Eigenspectrum Analysis of Neural Networks without Aspect Ratio Bias
Yuanzhe Hu 1 Kinshuk Goel 2 3 Vlad Killiakov 4 Yaoqing Yang 2
Abstract
Diagnosing deep neural networks (DNNs) by an-
alyzing the eigenspectrum of their weights has
been an active area of research in recent years.
One of the main approaches involves measuring
the heavytailness of the empirical spectral densi-
ties (ESDs) of weight matrices. This analysis has
been shown to provide insights to help diagnose
whether a model is well-trained or undertrained,
and has been used to guide training methods in-
volving layer-wise hyperparameter assignment.
In this paper, we address an often-overlooked
challenge in estimating the heavytailness of these
ESDs: the impact of the aspect ratio of weight
matrices. We demonstrate that matrices of vary-
ing sizes (and aspect ratios) introduce a non-
negligible bias in estimating the heavytailness of
ESDs, leading to inaccurate model diagnosis and
layer-wise hyperparameter assignment. To over-
come this challenge, we propose FARMS (Fixed-
Aspect-Ratio Matrix Subsampling), a method that
normalizes the weight matrices by subsampling
submatrices with a fixed aspect ratio. Instead of
measuring the heavytailness of the original ESD,
we measure the average ESD of these subsampled
submatrices. We show that this method effectively
mitigates the aspect ratio bias. We validate our
approach across various optimization techniques
and application domains that involve eigenspec-
trum analysis of weights, including image classifi-
cation in computer vision (CV) models, scientific
machine learning (SciML) model training, and
large language model (LLM) pruning. Our results
show that despite its simplicity, FARMS uniformly
improves the accuracy of eigenspectrum analysis
1Department of Computer Science and Engineering, Univer-
sity of California, San Diego, CA, USA 2Department of Com-
puter Science, Dartmouth College, NH, USA 3Department of
Computer Science & Engineering, SRM Institute of ...