Investigating the Overlooked Hessian Structure: From CNNs to LLMs
Authors: Qian-Yuan Tang
Yufei Gu
Yunfeng Cai
Mingming Sun
Ping Li
zhou Xun
Zeke Xie
Venue: ICML 2025 poster (2025)
Keywords: Hessian
Loss Landscape
Generalization
Large Language Models
Primary area: deep_learning->everything_else

Abstract: It is well-known that the Hessian of deep loss landscape matters to optimization and generalization of deep learning. Previous studies reported a rough Hessian structure in deep learning, which consists of two components, a small number of large eigenvalues and a large number of nearly-zero eigenvalues. To the best of our knowledge, we are the first to report that a simple but overlooked power-law Hessian structure exists in well-trained deep neural networks, including Convolutional Neural Networks (CNNs) and Large Language Models (LLMs). Moreover, we provide a maximum-entropy theoretical interpretation for the power-law Hessian structure and theoretically demonstrate the existence of robust and low-dimensional subspace of deep neural networks. Our extensive experiments using the proposed power-law spectral method demonstrate that the power-law Hessian spectra critically relate to multiple important behaviors of deep learning, including optimization, generalization, and overparameterization. Notably, we discover that the power-law Hessian structure of a given LLM can effectively predict generalization during training, while conventional sharpness-based generalization measures that often works well on CNNs become nearly useless for as a generalization predictor of LLMs.

Investigating the Overlooked Hessian Structure: From CNNs to LLMs
Qian- YuanYuan TPe Tang*1
Yufei Gu*
Yunfeng Cai
Mingming Sun Ping Li Xun Zhou Zeke Xiet 2
Abstract
Yao et al., 2018; Dauphin et al., 2014; Byrd al., 2011 1). Deep learning usually finds flat minima that generalize well (Hochreiter & Schmidhuber, 1995; 1997). The Hessian is one of the most important measures of the minima flatness and directly relates to generalization in deep learning (Hof- fer et al., 2017: Neyshabur et al., 2017; Dinh et al., 2017; Wu et al., 2017). Jiang et al. (2019) reported that minima- flatness- generalization bound is still the most reliable metric in extensive experiments. Wu et al. (2017) reported that the low-complexity solutions that generalize well have small norm of Hessian matrix with respect to model pa- rameters. Yao et al. (2018) 8) reported that the spectrum of the Hessian closely connects to large-batch training and adversarial robustness.
It is well-known that the Hessian of deep loss landscape matters to optimization and general- ization of deep learning. Previous studies re- ported a rough Hessian structure in deep learn- ing, which consists of two components, a small number of large eigenvalues and a large number of nearly-zero eigenvalues. To the best of our knowledge, we are the first to report that sim- ple but overlooked power-law Hessian structure exists in well-trained deep neural networks, in- cluding Convolutional Neural Networks (CNNs) and Large Language Models (LLMs). Moreover, we provide a maximum-entropy theoretical in- terpretation for the power-law Hessian structure and theoretically demonstrate the existence of a robust and cow-dimensional subspace of deep neu- ral networks. Our extensive experiments using the proposed power-law spectral method demonstrate that the power-law Hessian spectra critically re- late to multiple important behaviors of deep Iearn- ing, including optimization, generalization, and overparameterrizatonn. Notably, we discover that the power-law Hessian structure of given LLM can often predict generalization during training in some occasions, while conventional sharpness- based generalization measures which often work well on CNNs largely fail as an effective general- ization predictor of LLMs.
A number of works empirically studied the Hessian structure in Deep Neural Networks (DNNs). Some papers (Sagun et al., 2016 6; 2017; Wu et al., 2017) empirically reported a two-component structure that, in the context of deep learn- ing, most Hessian eigenvalues are nearly zero, while a small number of eigenvalues are large. Sankar et al. (2021) re- vealed that the layer-wise Hessian spectrum is similar to the entire Hessian spectrum. Zhang et al. (2024b) demonstrated that SGD performs worse than Adam for Transformers when Hessian spectra exhibit blockwise heterogeneity. Ormaniec et al. (2024) theoretically studied one-layer Transformers' Hessian in matrix derivatives while comparing them to clas- sical networks in deep learning.
However, quantitative or statistical analysis of the Hessian structure is still largely under-explored for modern neural networks. Does elegant statistical structure hide behind the Hessian spectrum? Does such structure matter to CNNs and LLMs? Our work provides a novel approach to under- standing and analyzing the Hessian structure of deep loss landscape from a statistical perspective. This work mainly made three contributions.
1. Introduction
It is well-known that the Hessian matters to optimization, generalization, and even robustness of deep learning (Li et al., 2020; Ghorbani et al., 2019; Jacot et al., 2019;
First, the best of our knowledge, we the first empir- ically discover and statistically test the power-law Hessian structure of deep loss landscape which has been overlooked by previous studies. Such novel power-law structure widely exist in DNNs, including CNNs and LLMs.
Depparment of Physics, Hong Kong Baptist University Lxaa Lab, The Hong Kong University of Science and Technology (Guangzhou) 'BISSA 'AGI Lab, BIMSA 'Rutgers University Seedd-Fundation-Mooles Team, ByteDance. tributions. Correspondence to: Zeke Xie <zekexie hkust- gz.edu.cn>.
Equal Con-
Second, we propose a framework of power-law spectral analysis for quantitatively analyzing the Hessian structure in deep learning. We not only reveal how the power-law
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Investigating the Overlooked Hessian Structure
10r
10a
10a
10a
SGB
sGn
Eigenvelue
109
Eigenvalue
10
Egenvvaue
10
Egenvalue
10
10"
10
10-1
10
10-
10-
10
10:
10
10:
10-2
10-2
10-1
10-3
100
10t
10a
10
10
10a
10a
10a
10
104
10a
10a
10a
104
10a
10a
Eigenvalue Rank
Eigenvalue Rank
Eigenvalue Rank
Eigenvalue Rank
(a) MNIST
(b) Fashion-MNIST
(c) CIFAR-10
(d) CIFAR-100
Figure/ 1. The power-law structure of the Hessian spectrum in deep learning. Model: LeNet. We may clearly observe that the power-law spectra generally hold for well-trained deep networks on various natural or artificial datasets, while do not hold for random neural networks. We also report that a small number of outlier eigenvalues (~100) slightly deviate from the fitted straight line.
spectra explain the theoretical origin of striking findings but also empirically demonstrate multiple novel insights on optimization, generalization, and scaling.
Table 1. The Kolmogorov-Sminn Smirnov statistics of the Hessian spectra of LeNets on various datasets. The estimated power exponent 3 and slope magnitude & are also displayed.
Third, we report that the plwer-law Hessian spectral anal- ysis can sometimes predict generalization of LLMs during training, particularly when conventional sharpness- based generalization measures that often work well on CNNs be- come nearly useless as a generalization predictor of LLMs. This suggests that generalization measures for LLMs remain to be deeply explored from loss landscape perspective.
Dataset
Model
Training
dos
de
Power: Law
MNIST MNIST
LeNet
Random
0.0796
0.0430
No
LeNet
SGD
0,00900
0,0430
Yes
Fashion MNIST
LeNet
Random
0.0971
0,0430
No
Fashion-MNIST
LeNet
SGD
0.0132
0,0430
Yes
CIFAR-10 CIFAR-10
LeNet
Random
0.0663
0,0430
No
LeNet
SGD
0.0279
0.0430
Yes
CIFAR-100 CIFAR- 100
LeNet
Random
0.0944
0,0430
No
LeNet
SGD
0.0315
0.0430
Yes
2. The Overlooked Hessian Structure
In this section, we demonstrate that the Hessian spectra of well-trained deep neural networks have a simple power- law structure overlooked by previous studies and how to Notations. We denote the training dataset as {(x,)) {(z5.8)) 11)) drawn from the data distribution S, the n model parameters as 0 and the loss function over one data sample {1),,))}} as 1(0, (1..,1))) For sim- plicity, we further denote the training loss as L(0) = N E/=1"(0 I(0, v))) and denote its Hessian as H. We write the descending ordered eigenvalues of the Hessian H
discover an approximately straight line fits the Hessian spee- trum of the well-trained neural network surprisingly well, except that a small number of outliers (~10) slightly deviate from the fitted straight line. To the best of our knowledge, these fitted power-law Hessian spectra were not empirically discovered or theoretically discussed by previous papers for neural networks deep learning.
theoretically derive the power-law structure.
The well-fitted straight line means that the observed distri- bution of the Hessian cigenvalues of trained neural networks approximately obeys power-law distribution,
p(A) = 2 1
as {λ1, ,2, and denote the spectral density as p(A).
where Ze is the normalization factor. The observed eigen- values can be considered as n samples from the power-law distribution p(A). We may also use a corresponding finite- sample power law for describing the observed law as
2.1. Visualizing the Power-Law Structure
Hessian has been studied as a measure of minima flatness Dinh et al. (2017); Xie et al. (2021b) and loss curvature Achille & Soatto (2019), while these works failed to reveal its elegant statistical structure. To better understand the distribution of the Hessian spectrum, we first visualize the Hessian spectrum of well-trained neural network and randomly initialized neural network by using the Lanczos algorithm (Meurant & Strakoš, 2006; Yao et al., 2020) to estimate the eigenvalues and spectral densities. In Figure 1, we display the top 6000 eigenvalues and their corresponding rank order. Both axes are log lcalscale. And we surprisingly
A fa TT(H)
Z k -ー
(2)
where f is the trace- nrace-normalized eigenvalue, I is the rank or- der, the trace Tr(H) ΣE At. and Z. = x =z k Æ is the normalization factor for the finite-sample power law. Note that the finite-sample power law is also called Zipf's law. This can also be approximately written as
(3)
2

---

Investigating the Overlooked Hessian Structure
if we let denote the power exponent of Zipf"s law.
et al., 2008; Tang & Kaneko, 2020). This exactly moti- vates to further verify and study the power-law structure of the Hessian spectrum in the context of deep learning. We discover that the elegant power-law structure indeed exists in well-trained deep neural networks just like bioactive pro- teins. In contrast, random neural networks have no such power-law structure, just like deactivated (denatured or un- folded) proteins. Random neural networks which have no functional ability the given task, break the power-law spectra similarly to deactivated proteins.
p-1
2.2. A Maximum-Entropy Interpretation
In this subsection, we show that the maximum entropy prin- ciple widely used in statistical physies may informally ex- plain the power-law Hessian structure in deep learning.
The maximum entropy principle (Guiasu& & Shenitzer, 1985), also named the maximum entropy prior, states that the prob- ability distribution which best represents the current state of knowledge about a system at equilibrium is the one with the highest entropy. This principle indicates that if we have no prior knowledge for suspecting one state over any other, then all states can be considered equally likely for sys- tem at equilibrium. The logarithmic space volume is often regarded as a kind of entropy in statistical physics ( (Visser, 2013). Note that flat minima have larger space volume reflected by ))). It means maximizing the minima space volume for better generalization may be regarded as a kind of entropy maximization principle. Following Visser (2013), may explicitly write the volume entropy as
2.3. Goodness-of-fi Test
In this subsection, are the first conduct formal statisti- cal tests on the Hessian structure. We also note that recent work (Xie et al., 2023a) follow our statistical spectral analy- sis via KS Tests and only studied the structure of stochastic gradients rather than the Hessian structure.
Following Alstott et al. (2014), we use Maximum Like- lihood Estimation (MLE) for estimating the parameter of the fitted power-law distributions and the Kolmogorov- Smirnov Test (KS Test) (Massey Jr, 1951; Goldstein et al., 2004) for statistically testing the goodness of the fit. The KS test statistic is the KS distance dxs between the hypothesized (fitted) distribution and the empirical data, which measures the goodness of fitting. Mathematically, the KS distance defined as dus = sup, (A) - F()],, where (*) is the hypothesized cumulative distribution function and F()) is the empirical cumulative distribution function based on the sampled data (Goldstein et al., 2004).
Svol = log det (ee((H=)) -) - / p()) log Add
(4)
and the spectral entropy as
Sp sp=- P(A)
P(A) logr(A)dd,
(5)
which is the entropy of the spectral density distribution. Theorem (The Maximum-Entropy Interpretation). Sup- pose we have the volume entropy Syst as Equation (4) and the spectral entropy Sp as Equation (5). To find the optimal distribution p" (A) that maximizes the total entropy, where
The estimated power exponent via MLE (Clauset et al., 2009) can be written 3 1 1+K Σ In Acatoor )
A
Stotal = Sp + Syol and Buorm is a Lagrange multiplier, the optimal distribution p² (A) can be solved as
where K is the number of tested samples and we set Aeutoff = As.- The Powerlaw library (Alstott et al., 2014) provides a convenient tool to compute the KS distance, das- and estimate the power exponent.
p" (A) c Suorm X -Bost
(6)
According to the practice of KS Test, first state the power-law hypothesis that the tested spectrum is power-law. If des is lower than the critical value de at the a = 0.05 significance level, the KS test statistically will support (not reject) the power-law hypothesis. The test results associated with Figure are presented in Table 1. We leave the details and more test results (e.g., ResNet18) in Appendix F.
We leave the proof in Appendix A. We note that the result in Theorem 1 has an amazingly similar form to (I) with Bnorm= log Ze and 3 = Byot-
We may interpret the power-law structure of the Hessian spectrum from two basic maximum entropy principles with the spectral density normalization constraint. It roughly means that simple rules can almost explain the power-law Hessian spectrum in deep learning as well as statistical physics (Visser. 2013). While previous work in the field of deep learning has also interpreted the minima flatness of neural networks from an entropy perspective (Baldassi et al., 2020), the spectra have much simpler structures as shown with our empirical results.
When we say that a spectrum is (approximately) power-law in this paper, we mean that the KS test provides positive evidence to the pawer-law hypothesis instead of rejecting the power-law hypothesis. Our KS test results reject the power- law hypothesis for random neural networks and do not reject the power-law hypothesis for well-trained neural networks. Moreover, when the power-law hypothesis holds, the KS distance is usually significantly smaller than the critical value de. For simplicity, the default a = 0.05 significance
Interestingly, similar well-fitted power laws have been widely discussed in neuroscience and biology (Reuveni
3

---

Investigating the Overlooked Hessian Structure
0,015
Eleenvalue Eipenpap
Rgengapt teonnefats
Egenvaluee Eigengap
Eignppae bijenszues
10
10a
10a
Elgenvalue
10
0.014
10-1
10-
10-1
0.013
10-1
10
10-2
10-1
10-r
10:4
100
10
10
10
1.05
1.10
10a
10a
10 Rank
10
10a
10a
10
10a
Eigenvalue
Rank
Figure 3. The power-law spectra hold across optimizers. Moreover, the slope magnitude is an indicator of minima sharpness and a predictor test performance. Model: LeNet. Dataset: MNIST.
(a) Eigenvalue Rank
(b) Eigengap Rank
The power-law Hessian eigengaps. Model: LeNet. Dat- sets: MNIST. Subfigure (a) displayed the eigengaps by original rank indices sorted by eigenvalues. Subfigure (b) displayed the eigengaps by rank indices re-sorted hy eigengaps. We also present the results of Fashion-MNIST in Figure 21 of Appendix D and GPT2-small Figure 30 Appendix E.
Figure 2.
25
15
200
10
150
100
0
1.00
1.05
1.10
1.15
1.20
1.00
1.05
1.10
1.15
1.20
level is abbreviated in the following.
Figure 4. The slope magnitude closely correlates to the largest Hessian eigenvalue and the Hessian trace. Model: LeNet. Dataset: MNIST.
Following related work on the Hessian of neural networks (Thomas et al., 2020), our empirical analysis and statistical tests mainly focused the top (~ 1000) large eigenval- ues larger than some minimal cutoff value Acutoff for three reasons. First, focusing on relatively large values is very rea- sonable and common in various fields' power-law studies, as roaldworld distributions typically follow power laws only after/larger than some cutoff values (Clauset et al., 2009) to ensure the convergence of the probability distribution. Sec- ond, researchers are usually more interested in significantly large eigenvalues which contribute more to Hessian, minima sharpness, or generalization bound (Thomas et al., 2020). Third, empirically estimating a large number of nearly zero eigenvalues is very inaccurate and expensive.
To quantitatively understand why learning subspace is ro- bust', we may use the angle between the original Hessian eigenvector ug and the perturbed Hessian eigenvector üg. namely (ue, ūx). Suppose the original Hessian is H, the perturbed Hessian is H = + e.M, the i-th eigenvector of H is u, and its corresponding perturbed eigenvector ü, Under the conditions of the Davis-Kahan Theorem and (13),
have
2c| M 1lop
sup supsin ūx) min( Ak-1 AR du - Ab+1)
2e |M l| op (k 1)
A1
2.4. The Power-Law Hessian Eigengaps
where ||M]|lp is the operator norm of the perturbation M. In the derivation details, we applied Theorem 2, useful variant of the Davis-Kahan Theorem (Yu et al., 2015), di- rectly to the Hessian in deep learning and demonstrate that the eigenspace robustness (spanned by the eigenvectors) is relatively tight for the top-learning eigenspace. To the extent of our knowledge, we are the first to theoretically explain the robust and low-dimensional learning subspace using Hessian eigengaps. Formal theoretical analysis and more discussion are available in Appendix B.
In this subsection, we report that the overlooked eigengaps of Hessian are also power-law and how the eigengaps sug- gest a robust and low-dimensional learning subspace.
The empirical investigation of the Hessian eigengaps is missing in previous works. Our experiments have closed this gap. Our experiments show that top eigengaps dominate other tailed eigengaps in deep learning. We visualize and verify the approximate power-law eigengaps in Figure 2.
The phenomenon of low-dimensional learning subspace was empirically reported recently (Gur-Ari al., 2018: Ghor- bani et al., 2019; Xie et al., 2021b) but still lacks theoretical understanding. Does the phenomenon theoretically depend on the Hessian structure? Our answer is yes. In the fol- lowing part, we will demonstrate why the eigengaps of the Hessian H may naturally lead to the phenomenon that learn- ing dynamics mainly take place in ciw-dimensional space during the entire training process.
3. Empirical Analysis of CNNs
In this section, we conduct extensive experiments for ex- ploring the behaviors of deep learning through the lens of power-law spectral analysis.
II this paper, robust space means that the space's 's dimensions are stable during training.

---

Investigating the Overlooked Hessian Structure
10a
Dopthin
Ooptting
10a
10a
Eigenvalue
10a
10
Eigenvatue
10
Bigenvalue
10a
10
10a
10"
#trarng-622
10-1
B-122
10-1
10-1
10-2
B-764
B-493
10-2
0-5009
10-2
10"
Eigenvalue Rank
10'
10'
10
10:2
10a
10:
10a
10
10-2
10a
Bw1224
10'
Eigenvalue Rank
10
10
10-1
10
10'
Eigenvalue Rank
10
10a
Eigenvalue Rank
Figure 5. The power-law spec-
Figure 7. Batch size matters to the spectrum. We discover three phases of the Hessian spectra for large-batch training. LeNet. Dataset: MNIST.
trum holds well in overparam- Figure 6. The spectra of LeNet eterized deep models but disap- on MNIST with respect to vari- pears in the underparameterized ous numbers of training samples. single-layer FCN.
Model:
16
Models: LeNet (LeCun et al., 1998). Fully Connected Net- works (FCN), and 8 (He et al., 2016).
337
eral
10
GoTaenna
Datasets: MNIST (LeCun, 1998), Pashion-MNIST (Xiao al., 2017), CIFAR- 10/100 (Krizhevsky & Hinton, 2009), and non-image Avila (De Stefano et al., 2018).
2PT2 mediun
0PT2 kape
gerziage
16a
10
12
10'
127
201
Eiganvalo
Ejerwalue
(a) Random / Pretrained
(b) Trained / Fine-tuned
1. Optimization and Generalization. Figure 3 discovered that the power-law spectrum consistently holds for vari- ous popular optimizers, such as SGD, Vanilla SGD, Adam, AMSGrad, AdaBound, Yogi, Adam, Adai, PNM, Looka- head, and DiffGrad, long the optimizers can train the network well. We present the KS test results in Table 3.
Figure 8. The power-law Hessian structure exists in well-trained LLMs but disappear in their random initializations (GPT2-nano) pretrained checkpoints (GPT2-small, GPT2- (ee2-mmdium, GPT2-large). The power-law Hessian structure emerged on well-trained LLMs after training / fine-tuning.
It is known that sharpness- based generalization measures are considered the most predictive generalization measures in deep learning (Jiang et al., 2019). We discover that the slope magnitude á of the fitted straight line may serve as nice predictor of minima sharpness and generalization, when the power-law Hessian structure is well fitted. Note that it is common measure minima's sharpness by the largest Hessian eigenvalue or the Hessian trace. A smaller highly correlates to a smaller largest eigenvalue and smaller trace in Figure 4. The similar observation holds on CIFAR-10 10 displayed in Figures and 22 of Appendix D.
large- batch training via the curves in Figure 7 and the KS test results in Table D of the appendix. To our knowledge, we are the first to report the phases and sharp phase transi- tion for large-batch training. When we train CNNs with the same training epochs and let the batch size increase from 640 to 768, the power-law structure suddenly breaks. We see that inadequate training due to a large batch size exhibits Hessian structure similar to one with limited training data. However, in Table D, training CNNs with the same itera- tions, we observe that large-batch training can also lead to power laws the expense of more compute.
Overparameterization. Figure 5 shows that the power- law spectrum holds well in overparameterized models, but
5. Supplementary Results. In Appendix D, we further discussed various interesting empirical results and insights, including studies on linear networks, modern architectures (such as ResNet18) 8), noisy labels, task transferability. and the heavy-tail behavior of SGD.
disappears in underparameteriiee models. Overparameter- ization is necessary for the plwer-law spectrum in deep learning. It will be interesting to study phase transition of under-parametert paadteraraattnizzaiionn to preepeeametiiinitoonn in future.
The size of training data. We evaluate the Hessian structure over various sized training data in Figure The model trained with limited training data would break the power-law structure similarly underparamettrizatton and lead to many sharp directions in the loss landscape. We see that data scaling and model scaling surprisingly exhibit very similar Hessian structures.
4. Empirical Analysis of LLMs
In this section, we empirically studied how the power-law Hessian structure of LLMs behaves differently.
Models: GPT-2 family (Radford et al., 2019): GPT2-nano (11M), GPT2-small (124M), GPT2-medium (355M), and GPT2-large (774M), and TinyLlama (Zhang et al., 2024a) 1.11-Chat-vl..0 1.)) with LoRA adapter (Hu al., 2021).
Batch Size. We discover the three different phases for
5

---

Investigating the Overlooked Hessian Structure
10P
Table 2. The Saamogorov-Smmmonn statistics of the Hessian spectra of various LLMs on various datasets.
Eigenvabue
3.0
Puaraon: 9:523
randemintt
Dataset
Model
Training
di
dc
Power-Law
2.5
train-3
OpenWebText
GPT2-small
Random
0.1168
0.0430
No
10
train-200
train220
WebTett
GPT2 small
Truined
0.0418
0.0430
Yes
Sain-500 wain-1000
Shakespeare
GPTZ-can
Random
0.0969
0,0450
No
Ofticcal st
Shakeespurre
GPT2-cano
Fine-tuned
0,0353
0.0430
Yes
100
1.0
Shakespeare
Pmllllmllll
Preirained
0.1058
0.0430
10.
10'
16a
0.000
0.025
0.050
0.075 0.100 0.125
Shakesppare
GPT2-small
Finc-tuned
0.0259
0.0450
Yes
Eigenvalue Rank
da
Shakespeare
GPT2-medium
Pretrained
0,0787
0.0430
No
4.3
Shakespeare
medium
tuned
0.0184
0.0430
Yes
Shakespeare
GPT2-large
Pretrnined
0.0450
0.0430
No
4.0
4.5
Shakespeare
GPT2-lagg
Fine-tuned
0.0160
0.0430
Yes
3.5
3.5
MathQA
Tayllama (LoRA)
Random
0.0552
0.0430
No
3.0
3.5
MathQA
Tinyllama (LaRA)
Fine-tuned
0.0249
0,0430
Yes
2:5
2.5
2.5
10a
10a
1.5
0.743
1.5
Peamton: 8.889
random-fsit train 50
train220
1.5
1000
250
10"
train 1.00
train 5000
THHM
Esgerwatue
train-200
Elgenvatue
10r
train-50300
Figure 10. The KS distance d&s may serve as an effective predictor to language model's ' generalization abilities with a Pearson corre- lation coefficient up 0.923. Model: GPT2-nano (trained from random initialization). Dataset: Shakespeare.
10a
10f
10-2
10
30a
100
303
107
15a
107
Eigenvalue Rank
Eigenvalue Rank
model architecture or limited to either vision (Figure 14) or text data (Figure 10). Further experiment results regarding different problem setups are provided in the Appendix E.
Figure 9. In the pretraining experiment of GPT2, the power-law Hessian structure emerged as training progressed in a two-stage process. In the first stage, the Hessian cigenvalues decrease in mag- nitude, indicating the discovery of flat minimum. In the second stage, the primal Hessian eigenvalues increase to form a power-law distribution, reflecting transition to sharper minimum. Model: GPT2-small. Dataset: OpenWebTexxt.
2. Generalization Measure. We studied how the general- ization of LLMs closely relates to the power-law Hessian structure across different models and datasets.
We observed that for Figure 10 and Figure 11, minima sharp- ness represented by the largest eigenvalue X and Hessian trace (H) behaves poorly as generalization measure for LLMs, contradicting conventional beliefs in deep learning (Jiang et al., 2019).
Datasets: WebTen (Gokaslan al., 2019), Shake- speare (Karpathy, 2015), and MathQA (Amini et al., 2019). 1. Pre-training and Fine-tuning of LLMs. We empirically investigate the Hessian structure of LLMs with pretraining or fine-tuning. Figure 8 shows that well-trained L.LMs af- ter training/finettuinng can exihibit the plwer-law Hessian structures, similarly to CNNs, while randomly initialized or pretrained models fail. Table presents the KS test results on the power-law Hessian spectra across various LLMs, in- cluding GPT-2 series and TinyLlama- lb with LoRA adapter, in the case of pretraining or fine-tuning on various tasks.
In the pretraining experiment of GPT2-nano, Figure 10 shows that pretraining can decrease the KS distance and test loss effectively. However, surprisingly, we discover that sharpness based generalization measures become nearly useless for predicting generalization. In contrast, the KS distance can serve as effective predictor generalization ability with the Pearson correlation coefficient up to 0.923, Note that the KS distance dis metric quantifies the adher- ence of a model's power-law spectral properties to those expected of a well-trained neural network.
The emergence of the pawer-law Hessian structure is evi- dent throughout both the pretraining experiment (in Figure 9 and 10) and the fine-tuning experiment (in Figure 11). Dur- ing pretraining, the power-law structure gradually develops as we optimize model parameters to capture the underlying structure of the training data. Similarly, during fine-tuning, the pawer-law structure adapts to reflect the specific charac- teristics of the target dataset. These findings, combined with empirical results from CNNs, suggest that the presence of the hessian Power-law structure not confined to a specific
In the fine-tuning experiment of GPT2-small in Figure 11, we notice similar observations. Again, the sharpness-bas bassd generalization measure fails, whereas the power-law good- ness can work well. We can even see that the sharpness increases significantly during iterations 50-5000, whereas the test loss still drops quickly. Similar observation holds in Figure 9, In contrast, during all 5000 iterations, the test loss and the KS distance continuously drop synchronously
6

---

Investigating the Overlooked Hessian Structure
10
4.00
3.5
3.75
3.0
3.50
Eqgenvalus
Lo05
Pearsore 9.783
Elgenvaaes
10
2.25
presain
Finetunt-50
109
2.0
Finetune-300
Fnsttnne-500
10
2.75
Pearson: 3.500
spt2-amal
1.5
2010
2.25
2.50
sp12 mestum
spt2-large Cifical
2.00
10F
10'
10"
10a
0.52
0.04
0.06
0.08 das
0:20
0.12
0.34
10P
10a
30
10
9.00
0.02
0.04
0.06
0.68
0.30
Elganvaluue Rank
Eigenvalue Rank
4.5
4.5
(a) GPT-2 (Pretraining with 5000 steps)
4.0
3.5
3.5
10
4.00
3.0
3.75
3.50
2.-
3.0
Elgene
100
3.00
Pwwr C 716
Fuassrr 0,247
1.0
20a
1.2
10-1
2.75
Poasson: 9,379
spis-amal
TW/4
2.50
spti-mestum
Figure 11. The pawer-law Hessian structure also emerged in fine- tuning tasks. The KS distance des predicts the model's gener- alization abilities with strong Pearson correlation coefficient up to 0.980, while the sharpness- based generalization measures obviously fail in predicting generalization of LLMs. Model: GPT2- small. Dataset: Shakespeare.
spi2-large
Citical
2.00
10P
16
30a
10*
0.00
0.02
0.04
0.06
0.00
0.18
Eigenvalue Rank
(b) GPT-2 (Pretrained Models)
Figure 12. The power-law spectrum holds across GPT-2 models with different model capacity. As we scale model parameters, the Hessian power-law structure goodness and test performance both improve. Subfigure (a) displays the eigenvalues of intermediate pretraining checkpoints at 5000 steps; Subfigure (b) displays the eigenvalues of the official pretrained GPT-2 models. Models: (PP22-small, medium, large). Dataset: OpenWexText.
with the Pearson correlation coefficient up 0.98, while the Pearson correlation coefficients for sharpness- beasee gener- alization measures are even only -0.714 4 and 0.247, which are harmful nearly useless for predicting generalization.
We conjecture that the power-law Hessian structure and the minima's sharpness of the loss landscape can capture a model's generalization ability at different phases. Unlike CNNs, LLMs are often extremely over- pee-parrameteiieed and far from well-trained (e.g., many epochs). In the phase of staying far from minima, the power-law Hessian structure can better predict the generalization of LLMs. In contrast, as people usually train CNNs for many epochs, well-trained CNNs stay very close to minima. In the phase of stay- ing close to minima, gencralization can be captured by the minima's sharpness better, following conventional general- ization theory. We believe a detailed generalization analysis for LLMS remains open for future research.
coefficients are higher than 0.99 for the intermediate check- point and also high for the official pretrained models. This observation further supports that scaling law perspective is reflected in the power-law Hessian structure of the L.IMM
4. Vision Transformer: We also investigate the power-law Hessian structure of Vision Transformer on vision datsets. We analyze the pre-trained and fine-tuned ViT-base models (Dosovitskiy et al., 2021) on the CIFAR- 100 dataset. Figure 14 shows that the power-law Hessian structure is absent in random ViT, and emerges in well-trained VIT models.
We present additional results and discussion, including wise Hessian analysis and LoRA fine-tuning, in Appendix E. (1) In Figure 31, we ob- serve that the power-law Hessian structure exists in various layers LLMs. The plwer-law Hessian structure of the first and middle layers often more pronounced than the last layer. (2) In Figure 33, we discover that while fine-tuning LLMs with LoRA can improve the performance of LLMs specific task, LoRA cannot significantly affect the Hes- sian structure of LLMs as full-parameter fine-tuning, The power-law Hessian structure does not emerge with LoRA fine-tunin mie-tunnig. It suggests that full- full-parameter and LoRA fine- tuning may have significant generalization abilities.
5. Supplementary Results.
3. Model Capacity and Scaling, Scaling law predicts that the performance of LLMs typically follows power laws as we scale model parameters, training data, or computing (Bahri et al., 2024). We further investigate how the power- law Hessian structure depends on model scaling.
Figures 12 and 13 present the results of various pretrain- ing checkpoint (step 5000) and pretrained GPT-2 models trained and evaluated on OpenWehText The results demon- strate that as model parameters increase, the power-law Hes- sian structure also becomes significantly more pronounced for both training and pretrained checkpoints, accompanied by improved model performance. The Pearson correlation

---

Investigating the Overlooked Hessian Structure
(2021) studied the Hessian spectra of more realistic nonlin- ear models. Kaur et al. (2023) studied the maximum Hessian eigenvalue and its relation to generalization. Dauphin et al. (2024) investigated the neglected Nonlinear Modeling Error (NME) matrix part of the Hessian and its influence on gra- dient penalties during training. While a number of works studied the Hessian spectra, they failed to empirically or theoretically discover the simple but important power-law structure, nor reveal its connection generalization.
7 0/14
4 8.43
Oetuumal
CrTz-mal
6/0
841 6972 dep.
suree: 2486
Parian 0.156
Parumeters (rs lan)
Faramsters trd
Figure 1.3. Scaling law the power-law goodness for the Hessian structure. Larger models have more precise power-law Hessian structure. Left: Intermediate GPT-2 pretraining checkpoints at 5000 steps; Right: Official pretrained GPT-2 models. Models; GPT2-(small, medium, large). Dataset: Openwebrrext.
Other power-law phenomena in deep learning. Hestness et al. (2017) studied the power-law relation between model performance and model size as well as data size. Mahoney & Martin (2019) reported that the elements of weight ma- trices may exhibit power-law heavy tails and studied the trends of spectral decay. Lee al. (2020); Velikanov & Yarotsky (2021) reported the power-law decaying eigenval- ues in kemel methods. Agrawal et al. (2022) studied the eigenspectrum decaying of feature covariance with a theoret- ical analysis of linear regression. Xie et al. (2023a) studied the heavy-tailed structure of stochastic gradient covariance. However, none of them reported the overlooked power-law Hessian structure of DNNs.
10a
100
Elgenvalue
10-1
random-init pretrain
6. Conclusion
well-trained
10
10
10
10
10a
While the Hessian of the deep loss landscape matters to optimization and generalization of deep learning, the statis- tical structure of the Hessian is still largely overlooked by previous studies. To the best knowledge, we are the first to report the overlooked power-law Hessian structure in deep learning well formal statistical tests on the power-law Hessian spectra. We provide a novel maximum-entrops interpretation and explain why the learning space may be low-dimensional and robust. While the main limitation of our work is that we cannot include those state-of-the-- LLMSs in our experiments due to the extremely large mem- ory and computational cost of Hessian analysis, our work still goes much further beyond those previous qualitative studies. The power-law Hessian structure provides a useful and novel perspective to reveal and analyze multiple novel behaviors of deep learning on optimization, generalization, and over- parameterization. We particularly discovered that while conventional sharpness-based generalization measures are considered nice generalization predictors of CNNs, they often completely fail predict the generalization of LLMs. Instead, the power-law goodness of the Hessian structure of- ten correlates better with generalization on many deep learn- ing occasions, while we also observe that it is not a robust generalization measure. This suggests that generalization theory and measures of LLMs lacks more exploration and rethinking. We believe that our work will further inspire the- ories and empirical advances toward a deeper understanding of DNNs, loss landscape, and Hessian.
Eigenvalue Rank
Figure 14. The power-law Hessian structure is absent in randomly initialized Vision Transformers and emerges with training. Model: ViT-base. Dataset: CIFAR-100.
5. Related Work
The Hessian structure of DNNs. A number of related works analyzed the spectral distribution of the Hessian in deep learning. Pennington & Bahri (2017) introduced an an- alytical framework from random matrix theory and reported that the shape of the spectrum depends strongly on the en- engy and the over- parameterrrzionn parameter, 6, which measures the ratio of parameters data points. However, Pennington & Bahri (2017) mainly evaluated single-hidden- layer networks, which limits the scope of the conclusion. A followup work (Pennington& & Worah, 2018) focused on neural network with Gaussian data
a Linglehhidden-laee and weights in the limit of infinite width. Obviously, its theoretical and empirical analysis is far from practical deep models. Jacot al. (2019) analyzed the limiting spectrum of the Hessian in neural networks with infinite width. Fort & Scherlis (2019) analyzed the Hessian spectra of initial- ized neural networks. Papyan (2019) studied the three-level hierarchical structure and outliers in Hessian spectra. Singh et al. (2021) proved that the Hessian can be of very low rank for DNNs with linear activations. Liao & Mahoney
8

---

Investigating the Overlooked Hessian Structure
Acknowledgement
Clauset, A., Shalizi, C. R.,, and Newman, M. E. Power- law distributions in empirical data. SIAM review, 51(4): 661-703, 2009.
This work was sponsored by Doubao Large Model Fund of ByteDance, Natural Science Foundation of China (No. 12305052), Research Grants Council of Hong Kong (Nos. 22302723 and SRES2324-2505), Hong Kong Baptist Uni- versity's ' funding support (RC- (RCcoottRRRCRRR..R (RC-FNNA--G/2-22SCCI0)
Dauphin, Y. N., Pascanu, R., Gulcehre, C.,, Cho, K., Gan- guli, S., and Bengio, Y. Identifying and attacking the saddle point problem in high- high-dimensional non-convex op- timization. Advances in Neural Information Processing Systems, 27:2933-2941, 2014.
Impact Statement
Dauphin, Y. N., Agarwala, A., and Mobahi, H. Neglected hessian component explains mysteries in sharpness regu- larization, arXiv preprint arXiv:2401. 108099 2024.
This paper presents work whose goal aims at understanding the foundation of deep learning. While may have many potential societal consequences, we think none of them must be specifically discussed here.
De Stefano, C.,, Maniaci, M., Fontanella, F., and di Freca, A. Reliable writer identification in medieval manuscripts through page layout features: The "avila" bible case. Engineering Applications of Artificial Intelli- gence, 72:99-110, 2018.
References
Achille, A. and Soatto, S. Where is the information in deep neural network? arXiv preprint arXiv: arXiv: /ttp///05.22222 2019.
Dinh, L.,, Pascanu, R., Bengio, S., and Bengio, Y. Sharp minima can generalize for deep nets. In International Conference on Machine Learning, pp. 1019-1028, 2017.
Agrawal, K. K., Mondal, A. K., Ghosh, A., and Richards, B. alpha-req: Assessing representation quality in self- supervised learning by measuring eigenspectrum decay. Advances in Neural Information Processing Systems, 35: 17626-17638 2022.
Dosovitskiy, A., Beyer, L.,, Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image worth 16x16 words: Transformers for image recognition at scale, 2021. URL https: www.xi...og/bbb.. 2010 11929.
Alstott, J., Bullmore, E., and Plenz, D. powerlaw: a python package for analysis of heavy-tailed distributions. PloS one, (1):e85777, 2014.
Dubey, S. R., Chakraborty, S., Roy, S. K., Mukherjee, S., Singh. S. K., and Chaudhuri, B. B. diffgrad: An optimiza- tion method for convolutional neural networks. IEEE transactions on neural networks and learning systems, 31 (1 (1):4500-4551, 2019.
Amini, A., Gabriel, S., Lin, S., Koncel-Kedziorski, R., Choi, Y. and Hajishirzi, H. MathQA: Towards interpretable math word problem solving with operation-bart based for- malisms. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Com- putational Linguisties: Human Language Technologies, Volume (Long and Short Papers). pp. 2357-2367, Min- neapolis, Minnesota, June 2019. Association for Compu- tational Linguistics. doi: 10. 10.18655//1/119--44455 URL ht attpps:
Fort, S. and Scherlis, A. The goldilocks zone: Towards better understanding of neural network loss landscapes. Proceedings of the AAAI Conference on Artificial In- telligence, volume 33, pp. 3574-3581, 2019.
Ghorbani, B.,, Krishnan, S., and Xiao, An investigation into neural net optimization via hessian eigenvalue den- sity. In International Conference on Machine Learning, pp. 2232-22222222 2241. PMLR, 2019.
orr./or////. 9 1245.
Bahri, Y., Dyer, E., Kaplan, J., Lee, J., and Sharma, U. Ex- plaining neural scaling laws. Proceedings of the National Academy of Sciences, 121(27))e23118777 1887781221 2024.
Gokaslan, A., Cohen, V., Pavlick, E.,, and Tellex. S. Open- webtext corpus. Sctp://Skyline githb.. i Loo/ppeneebTextCorpppe 2019.
Baldassi, C., Pittorino, F., and Zecchina, R. Shaping the learning landscape in neural networks around wide flat minima. Proceedings of the National Academy of Sci- ences, 117()):61-177, 2020.
Goldstein, M. I.., Morris, S. A., and Yen, G. G. Problems with fitting to the power-law distribution. The European Physical Journal B-Condensed Matter and Complex Sys- tems, 41(2):255-255 2004.
Byrd, R. H., Chin, G. M., Neveitt, W., and Nocedal, J. On the use of stochastic hessian information in optimization methods for machine learning. SIAM Journal on Opti- mization, 21(3):977-995 2011.
Guiasu, S. and Shenitzer, A. The principle of maximum entropy. The mathematical intelligencer, 7(1):42-4) 48, 1985.
9

---

Investigating the Overlooked Hessian Structure
Gur-Ari, G., Roberts, D. A., and Dyer, E. Gradient descent happens in a tiny subspace. arXiv preprint arXiv:1812.04754, 2018.
International Conference on Learning Representations, 2019.
Jiang, Y., Neyshabur, B., Mobahi, H., Krishnan, D., and
Gurbuzbalaban, M., Simsekli, U., and Zhu, L. The heavy- tail phenomenon in sgd. In International Conference on Machine Learning, pp. 3964-3975. PMLR, 2021.
Bengio, S. Fantastic generalization measures and where find them. In International Conference on Learning Representations, 2019.
Han, B., Yao, Q., Yu, X., Niu, G., Xu, M., Hu, W., Tsang. I., and Sugiyama, M. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In Advances in neural information processing systems, pp. 8527-8537, 2018.
Karpathy, A. char-rnn. https://github.. comm karpathy/ /char-rnn, 2015.
Karpathy, A. NanoGPT.
Kaur, S., Cohen, J., and Lipton, 7. C. On the maximum
Han, B., Yao, Q., Liu, T., Niu, G.. Tsang, I. W., Kwok, J. T., and Sugiyama, M. A survey of label-noise representa- tion learning: Past, present and future. arXiv preprint arXiv:201 .4406, 2020.
hessian eigenvalue and generalization. In Proceedings on, pp. 51-65. PMLR, 2023.
Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. 3rd International Conference on Learning Representations, ICLR 2015, 2015.
He, K., Zhang, X., Ren, S., and S.n, Deep residual learn- ing for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-77877887 2016.
Krizhevsky, A. and Hinton, G. Learning multiple layers of features from tiny images. 2009.
He, Z., Xie, Z., Zhu, Q., and Qin, Z. Sparse double descent: Where network pruning aggravates overfitting. In Interna- tional Conference on Machine Learning. pp. 8635-8659 PMLR, 2022.
LeCun, Y. The mnist database of handwritten digits. http://yanm. lecun. com/exdb/mmist// 1998.
LeCun, Y., Bottou, L.,, Bengio, Y., and Haffner, P. Gradient- based learning applied to document recognition. Proceed- ings of the IEFE, 86(11):222 1):2278 23244, 1998.
Hestness, J., Narang, S., Ardalani, N., Diamos, G., Jun, H., Kianinejad, H., Patwary, M.. Ali, M., Yang, Y., and Zhou, Y. Deep learning scaling is predictable, empirically, arXiv preprint arXiv:1712.004009 2017.
Lee, J., Schoenholz, S., Pennington, J., Adlam, B., Xiao, L., Novak, R., and Sohl-Dickstein, J. Finite versus infinite neural networks: empirical study. Advances in Neural Information Processing Systems, 33: 33:15156-15172 2020.
Hochreiter, S. and Schmidhuber, J. Simplifying neural nets by discovering flat minima. In Advances in neural information processing systems, pp. 529-536, 1995.
Li, X., Gu, Q., Zhou, Y., Chen, T., and Banerjee, A. Hessian based analysis of sgd for deep nets: Dynamics and gener- alization. In Proceedings of the 2020 SIAM International Conference on Data Mining, pp. 190-198. SIAM, 2020.
Hochreiter, S. and Schmidhuber, J. Flat minima. Neural Computation, 9(1):1-42, 1997.
Hodgkinson, L. and Mahoney, M. Multiplicative noise and heavy tails in stochastic optimization. In Interna- tional Conference on Machine Learning, pp. 4262-4274 PMLR, 2021.
Li, Z., Malladi, S., and Arora, S. On the validity of mod- eling SGD with stochastic differential equations (SDEs). In Thirty-Fitth Conference on Neural Information Pro- cessing Systems, 2021. URL https: //ppnrevii............ net / forum? wiw.uumddddoooodoondoddo_a
Hoffer, E., Hubara, I., and Soudry, D. Train longer, general- ize better: closing the generalization gap in large batch training of neural networks. In Advances in Neural Infor- mation Processing Systems, pp. 1729-1739, 2017.
Liao, Z. and Mahoney, M. W. Hessian eigenspectra of more realistic nonlinear models. Advances in Neural Information Processing Systems, 34, 2021.
Hu, J., Shen, Y., Wallis, P., Allen-Zhu, 7.,, Li, Y., Wang,
Liu, L., Jiang, H., He, P., Chen, W., Liu, X., Gao, J., and Han, On the variance of the adaptive learning rate and beyond. In International Conference on Learning Representations, 2019.
S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2/06.0968 09685, 2021.
Jacot, A., Gabriel, F., and Hongler, C. The asymptotic spectrum of the hessian of dnn throughout training. In
Loshchilov, I. Decoupled weight decay regularization. arXiv preprint arXiv: 2017.
10

---

Investigating the Overlooked Hessian Structure
Luo, L., Xiong, Y., Liu, Y., and Sun, X. Adaptive gradient methods with dynamic bound of learning rate. 7th Inter- national Conference on Learning Representations, ICLR 2019, 2019.
Reuveni, S., Granek, R., and Klafter, J. Proteins: coexis-
tence of stability and flexibility. Physical review letters, 100(20):208000) 2008.
Sagun, L.,, Bottou, I.., and LeCun, Y. Eigenvalues of the hessian in deep learning: Singularity and beyond. arXiv preprint arXiv: 1ate:::6111.777770 1.0747,, 2016.
Mahoney, M. and Martin, C. Traditional and heavy tailed self regularization in neural network models. In Interna- tional Conference on Machine Learning, pp. 4284-4293. PMLR, 2019.
Sagun, L.,, Evci, U., Guney, V. U., Dauphin, Y., and Bottou, L. Empirical amalysis of the hessian of wver-parametrized neural networks. arXiv preprint arXiv: 7006:/7060.00 0444444444, 2017.
Massey Jr, F. J. The kolmogorov-smimov test for goodness of fit. Journal of the American statistical Association, 46 (253):68-78) 1951.
Sankar, A. R., Khasbage, Y., Vigneswaran, R., and Bala- subramanian, V. N. A deeper look at the hessian eigen- spectrum of deep neural networks and its applications to regularization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 9481-9488, 2021.
Meurant, G. and Strakoš, Z. The lanczos and conjugate gradient algorithms in finite precision arithmetic. Acta Numerica, 15:471-542, 2006.
Simsekli, U., Sagun, L., and Gurbuzbalaban, M. A luil-index analysis of stochastic gradient noise in deep neural net- works. In International Conference on Machine Learning, pp. 5827-5837, 2019.
Myung, J. Tutorial on maximum likelihood estima- tion. Journal of mathematical Psychology, 47(1):90-100, 2003.
Neyshabur, B., Bhojanapalli, S., McAllester, D., and Sre- bro, N. Exploring generalization in deep learning. In Advances in Neural Information Processing Systems. pp. 5949-5958, 2017.
Singh, S. P., Bachmann, G., and Hofmann, T. Analytic insights into structure and rank of neural network hes- sian maps. Advances in Neural Information Processing Systems, 34:23914-23921 2021.
Tang, Q.-Y. and Kaneko, K. Long-range correlation in protein dynamics: Confirmation by structural data and normal mode analysis. PLoS computational biology, 16 (2):c1007670, 2020.
Ormaniec, W., Dangel, F., and Singh, S. P. What does it mean to be transformer? insights from theoretical hessian analysis. arXiv preprint arXiv:2410.10986, 2024.
Panigrahi, A., Somani, R., Goyal, N., and Netrapalli, P. Non- gaussianity of stochastic gradient noise. arXiv preprint arXiv:1910.09626. 2019.
Thomas, V., Pedregosa, F., Merriënboer, B., Manzagol, P.- A., Bengio, Y., and Le Roux, N. On the interplay between noise and curvature and its effect optimization and generalization. In International Conference on Artificial Intelligence and Statistics, pp. 3503-3513. PMLR, 2020.
Papyan, V. Measurements of three-level hicrarchical struc- ture in the outliers in the spectrum of deepnet hessians. In International Conference Machine Learning, pp. 5012-5021. PMLR, 2019.
Velikanov, M. and Yarotsky, D. Explicit loss asymptotics in the gradient descent training of neural networks. Ad- vances in Neural Information Processing Systems, 34:
Pennington, J. and Bahri, Y. Geometry of neural net- work loss surfaces via random matrix theory. In Interna- tional Conference on Machine Learning. pp. 2798-2806. PMLR, 2017.
2570-2582 2021.
Visser, M. Zipf's law, power laws and maximum entropy. New Journal of Physics, 15(4))043021, 2013.
Wu, L., Zhu, Z., et al. Towards understanding generalization of deep learning: Perspective of loss landscapes. arXiv preprint arXiv: /706. 10239, 2017.
Pennington, J. and Worah, P. The spectrum of the fisher in- formation matrix of a single-hidden-layee neural network. In NeurlPS, pp. 5415-5424, 2018
Xiao, H., Rasul, K., and Vollgraf, R. Fashion-mnist a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv: 170..00.77 2017.
Radford, A.,
Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAl blog, 188); 2019.
Xie, Z., He, F., Fu, S., Sato, I., Tao, D., and Sugiyama, M. Artificial neural variability for deep learning: On over- fitting, noise memorization, and catastrophic forgetting. Neural Computation, 33(8), 2021a.
Reddi, S. J., Kale, S., and Kumar, S. On the convergence of adam and beyond. 6th International Conference on Learning Representations, ICLR 2018, 2019.

---

Investigating the Overlooked Hessian Structure
Xie, Z., Sato, I., and Sugiyama, M. diffusion theory for deep learning dynamics: Stochastic gradient descent exponentially favors flat minima. In International Confer- ence on Learning Representations, 2021b. URL https: //openreview ew. wwweteettoooeeeee forum? diwidddwwwa@o_cc Liiggk_____oo__
Zhang, Y., Chen, C., Ding, T., Li, Z., Sun, R., and Luo, Z.- Q. Why transformers need adam: A hessian perspective. arXiv preprint arXiv:2402. 16788, 2024b.
Xie, Z., Yuan, L., Zhu, Z., and Sugiyama, M. Positive- negative momentum: Manipulating stochastic gradient noise to improve generalization. In International Confer- ence on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 11448-11455 PMLR, 18 28-24.00 Jul 2021c.
Xie, Z., Wang, X., Zhang, H., Sato, I., and Sugiyama, M. Adaptive inertia: Disentangling the effects of adaptive learning rate and momentum. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp- 24430-24459, 2022.
Xie, Z., Tang, Q.. Sun, M., and Li, P. On the overlooked structure of stochastic gradients. In Thirty-seventh Confer- ence on Neural Information Processing Systems, 2023a.
Xie, Z., Xu, Z., Zhang, J., Sato, I., and Sugiyama, M. the overlooked pitfalls of weight decay and how to mitigate them: A gradient-norm perspective. In Thirty-seventh Conference on Neurul Information Processing Systems, 2023b.
Yao, Z., Gholami, A., Lei, Q., Keutzer, K., and Mahoney, M. W. Hessian-based analysis of large batch training and robustness to adversaries. In Advances in Neural Information Processing Systems, pp. 4949-4959, 2018.
Yao, Z., Gholami, A., Keutzer, K., and Mahoney, M. W. Py- hessian: Neural networks through the lens of the hessian. In 2020 IEEE International Conference on Big Data (Big Data), pp. 581-590. IEEE, 2020.
Yu, Y., Wang, T., and Samworth, R. J. useful variant of the davis-kahan theorem for statisticians. Biometrika, 102(2):315-323 2015.
Zaheer, M., Reddi, S., Sachan, D., Kale, S., and Kumar, S. Adaptive methods for nonconvex optimization. In Advances in neural information processing systems, pp. 9793-9803, 2018
Zhang, M., Lucas, J., Ba, J., and Hinton, G. E. Lookahead optimizer: k steps forward, step back. Advances in Neural Information Processing Systems, 32:9597-9608, 2019.
Zhang, P., Zeng. G., Wang. T., and Lu, W. Tinyllama: An ooun-source small language model, 2024a.
12

---

Investigating the Overlooked Hessian Structure
A. Proof of Theorem 1
Proof. Considering the principle of maximum entropy with the two kinds of entropy, we need to maximize the total entropy with the spectral density normalization constraint
Stotal = / p(A) logp(A)ds Bool / p())
log log Ada
(8)
Puorml I p(2)dx 1),
where Stotal Sp + Svol and Pnorm is a Lagrange multiplier. To find the optimal distribution (*(() that maximizes the total entropy, we require the following
oStotal log p(A) - 5ool log À - Snoom = O. 0p(A)
(9)
Thus, the optimal distribution (A) can solved as
p² (A) = -nnorm Arot
(10)
B. Robust and Low-Dimensional Learning Space
At first, denote the ordered eigengap as 6μ λ -a--a -Ax+11 which means the difference between two neighbored eigenvalues. According to (2), we have
8ū = "T((H) (k I4 (A: + 11 =ーx)
(11)
= > k +
(12)
shows that the eigengaps Hessians also approximately exhibit power-law distribution
5% Ir(H) (k + +)+1) 1++++1
(13)
under the approximation s 1. The power exponent s + is larger than the one in (2) by 1.
The phenomenon of low-dimensional learning subspace was empirically reported by Gur-Ari al. (2018). Ghorbani et al. (2019) also investigated and reported that large isolated eigenvalues quickly appear in the spectrum during the optimization process, along with a surprising concentration of the gradient in the corresponding eigenspace. However, they did not theoretically explain this phenomenon. Xie et al. (2021b) theoretically demonstrated that learning dynamics mainly happens along those principal eigenvectors of Hessian corresponding to large eigenvalues. Does the phenomenon theoretically depend on the Hessian structure? Our answer yes. In the following part, we will demonstrate why the eigengaps of the Hessian H may naturally lead to the phenomenon that learning dynamics mainly takes place in a low-dimensional space during the entire training process.
Previous papers only reported that top eigenvalues of Hessian are significantly larger than other tailed ones but did not touch how top Hessian eigengaps dominate other tailed ones in deep learning. However, top large eigenvalues do not imply their eigengaps are relatively large, too. Fortunately, we theoretically and empirically demonstrate that, as the rank index increases, both eigenvalues and eigengaps decay following power laws. Moreover, the eigengaps decay faster than eigenvalues. The theoretical implication behind the power-law eigengaps actually matters to deep learning dynamics.
We directly apply Theorem useful variant of
Theorem (Yu al., 2015), the Hessian deep learning. Theorem 2 (A useful variant of Davis-Kahan Theorem (Yu et al., 2015)). Suppose the true Hessian is H, the perturbed Hessian is H- = H + cM, the i-th eigenvector of H is and its corresponding perturbed eigenvector is ūg Under the conditions of the Davis-Kahan Theorem, we have
2e M llop
sin Ux) < minn(x-
Ae, dn - AR+1)
where M| the operator norm of M.
13

---

Investigating the Overlooked Hessian Structure
Given the power-law eigengaps in Equation (13), the upper bound of eigenvector robustness can be written as
sup (ug, ūx) 2e|1 2
IIo Ip (k
1)
(14)
A1
which is relatively tight for top dimensions and very loose for tailed dimensions. A similar conclusion also holds given Equation (11). This indicates that non-top eigenspace can be highly unstable during training, because Óд can decay to nearly zero for large k. To the best of our knowledge, we are the first to demonstrate that the robustness of low-dimensional learning space directly depends on the eigengaps of the Hessian H.
C. Experimental Settings
Computational environment. The image classification experiments are conducted on computing cluster with NVIDIA R V100/H800 GPUs and Intel Xeon @ CPUs.
C.1. Image Classification:
C.1.1. MODELS, DATASETS, AND OPTIMIZERS
LeNet (LeCun et al., 1998), Fully Connected Networks (FCN), ResNet18 (He et al., 2016) and Vision Transformer (Dosovitskiy et al., 2021). Particularly, we used one-layer FCN, two-layer FCN, four-layer FCN, which have 100 neurons for each hidden layer and use ReLu activations.
Models:
Datasets: MNIST (LeCun, 1998), MNNIST (Xiao et al., 2017), CIFAR-10/100 (Krizhevsky & Hinton, 2009), and non-image Avila (De Stefano et al., 2018).
Optimizers: SGD, Vanilla SGD, Adam (Kingma & Ba, 2015), AMSGrad (Reddi et al., 2019), AdaBound (Luo et al., 2019), Yogi (Zaheer et al., 2018), RAdam (Liu al., 2019), Adai (Xie et al., 2022), PNM (Xie et al., 2021c), Lookahead (Zhang et al., 2019), and DiffGrad (Dubey et al., 2019).
C.1.2. IMAGE CLASSIFICATION ON MNIST AND FASHION-MNIST
Data Preprocessing For MNIST and Fashion-MNIST: We perform the common per-pixel zero-mean unit-variance normalization.
Hyperparameter Settings: We select the optimal learning rate for each experiment from (0.0001, 0.001, 0.01, 0.1, 1, 10) for SGD and use the default learning rate for adaptive gradient methods. In the experiments on MNIST and -NNIT. = 0.1 for SGD, Vanilla SGD, Adai, PNM, and Lookahead; 0.1 for Vanilla SGD:1 0.001 for Adam, AMSGrad, AdaBound, Yogi, RAdam, and DiffGrad.
We train neural networks for 50 epochs on MNIST and 200 epochs on Fashion-MNISS For the learning rate schedule, the learning rate divided by 10 at the epoch of 40% and 80%. The batch size set to 128 for MNIST and Fashion-MNIST; unless we specify it otherwise.
The strength of weight decay defaults to À 0.0005 as the baseline for all optimizers unless specify it otherwise.
We set the momentum hyperparameter 31 = 0.9 for SGD and adaptive gradient methods which involve in Momentum. As for other optimizer hyperparameters, we apply the default settings directly.
C.1.3. IMAGE CLASSIFICATION ON CIFAR-10 10 AND CIFAR-100
Data Preprocessing For CIFAR-10 and CIFAR-100: We perform the common per-pixel zero-mean unit-variance normalization, horizontal random flip, and 32 x 32 random crops after padding with pixels on each side.
Hyperparameter Settings: We select the optimal learning rate for each experiment from (0.0001, 0.001, 0.01, 0.1, 1, 10} for SGD and use the default learning rate for adaptive gradient methods. In the experiments on CIFAF-10 and CIFAR- 100: = 1 for Vanilla SGD, Adai, and PNM; = 0.1 for SGD (with Momentum) and Lookahead; = 0.001 for Adam, AMSGrad, AdaBound, Yogi, RAdam, and DiffGrad. For the learning rate schedule, the learning rate is divided by 10 at the epoch of {80, 160) for CIFAR-10 10 and (100, 150) for CIFAR-100, respectively. The batch size is set to 128 for both CIFAR-10 10 and CIFAR- unless we specify otherwise.
14

---

Investigating the Overlooked Hessian Structure
The strength of weight decay is default À 0.0005 the baseline for all optimizers unless specify it otherwise. Xie al. (2023b) found that popular optimizers with À 0.0005 often yields test results than x 0.0001 for training CNNs on CIFAR-10 and CIFAR-100.
We set the momentum hyperparameter 31 = 0.9 for SGD with Momentum. As for other optimizer hyperparameters, we apply the default hyperparameter settings directly.
C.1.4. LEARNING CNNS WITH NOISY LABELS
We trained LeNet via SGD (with Momentum) on corrupted MNIST with various (asymmetric) label noise. We followed the setting of Han al. (2018) for generating noisy labels for MNIST. The symmetric label noise is generated by flipping every label to other labels with uniform flip rates {40%, 80%]. In this paper, when we talk about label noise, we mean symmetric label noise.
We also randomly shuffle the labels of MNIST to produce MNIST with random labels, which has little knowledge behind the pairs of instances and labels.
C.1.5, IMAGE CLASSIFICATION ON VISION TRANSFORMER
Data Preprocessing For CIFAR-100: We perform resizing of CIFAR-100 images to 224 224 pixels for compatibility during training. We then perform horizontal random flip and the common per-pixel cero-mean unit- normalization.
Hyperparameter Settings: We fine-tuned the pretrained Vision Transformer (ViT-Base) on the CIFAR-100 dataset, following the experimental setup of Dosovitskiy et al. (2021). We trained the pretrained VIT with a total batch size of 4096 for 200 steps using a learning rate of le-4 with linear decay. We employed the Adam optimizer with betas set to (0.9, 0.999) and applied a weight decay strength of 0.1.
Hessian Spectra Computation: We utilized the Stochastic Lanczos Quadrature (SLQ) algorithm implementation from Yao et al. (2020) for computing the Hessian cigenvalues for ViT, incorporating modifications for improved computational and memory efficiency. To further reduce the computational cost of SLQ, we sampled 1000 samples in each run and approximate the model's 's Hessian on the target dataset. For all Hessian experiments on ViT, 3,000 eigenvalues were computed across three repeated SLQ runs to mitigate bias, with only the top 1,000 eigenvalues displayed and used for evaluation, consistent with the experiment setup for LLMs.
C.2. Text Generation:
C.2.1. MODELS, DATASETS AND OPTIMIZERS
Models:
GPT-2 (Radford et al., 2019), Tinyllama (Zhang et al., 2024a). We use the code base of NanoGPT (Karpathy, 2022) for reproducing all GPT-2 models with pretraining and fine-tuning experiments.
Datasets: Web (Gokaslan et al., 2019), Shakespeare (Karpathy, 2015) (processed at the character level), and MathQA (Amini et al., 2019).
Optimizer: AdamW (Loshchilov, 2017) is used for training and fine-tuning of all language models utilized in this study.
C.2.2. IMPLEMENTATION DETAILS ON HESSIAN SPECTRA COMPUTATION
To enable efficient computation of the Hessian spectra for LLMs, we applied the Stochastic Lanezos Quadrature algorithm implementation provided by Yao et al. (2020) with alternations for computational and memory speedups. As computation cost of SLQ is unaffordable on large-scale text dataset as discussed Zhang et al. (2024b), we applied the same batch sampling trick and used fixed batch size of 256 in all of our text generation experiments to approximate model's Hessian evaluated on the target dataset. For all experiments conducted on LLLMs, 3,000 eigenvalues were computed across 3 repeated SLQ runs to minimize bias; Only the top 1,000 eigenvalues were displayed and used for evaluation in all Figures.
C.2.3. TRAINING CONFIGURATIONS
GPT2-nano trained on Shakespeare. We utilized a smaller 'baby GPT' model with 6 layers, attention heads, an embedding size of 384, and 11M parameters provided in NanoGPT. We used the AdamW optimizer with a fixed learning rate 6 x 10-4 We used a batch size 65,536 tokens and weight decay 0.1 for total of 1,000 steps.
15

---

Investigating the Overlooked Hessian Structure
Table 3. The Kolmogorov- statistics of various optimizers for training LeNet on CIFAR-10.
Training
dgs
de
Power-Law
B +
Random
0.0663
0.0430
No
SGD
0.0279
0.0430
1.968 + 0.031
1.033
Vanilla SGD
0.0276
0.0430
Yes
1.935 + 0.030
1.069
Adam
0.0269
0.0430
Yes
1.806 = 0.025
1.241
AMSGrad
0.0232
0.0430
1.786 0.023
1.271
AdaBound
0.0297
0.0430
Yes
1.901 + 0.028
1.110
Yogi
0.0184
0.0430
Yes
1.806 0.025
1.241
RAdam
0.0163
0.0430
1.733 0.023
1.363
Adai
0.0310
0.0430
Yes
1.918 0.029
1.090
PNM
0.0347
0.0430
Yes
1.911 0.029
1.098
Lookahead
0.0358
0.0430
1.964 0.030
1.037
DiffGrad
0.0303
0.0430
Yes
1.803 + 0.024
1.236
Table The Caamoggrrv--mminng statistics of the Hessian spectra various batch sizes.
Dataset
Model
Training
Batch Size
dles
de
Power-Law
B - a
MNIST
LeNet
SGD
B = 128
0.00900
0.0430
Yes
1.991 + 0.031
1.009
MNIST
LeNet
SGD
B 512
0.00787
0.0430
Yes
1.894 + 0.028
1.119
MNIST
LeNet
SGD
B = 640
0.0125
0.0430
Yes
1.838 + 0.027
1.194
MNIST
LeNet
SGD
B 768
0.278
0.0430
No
MNIST
LeNet
SGD
B = 1024
0.129
0.0430
No
MNIST
LeNet
SGD
B 16384
0.249
0.0430
No
MNIST
LeNet
SGD
B = 32768
0.201
0.0430
No
MNIST
LeNet
SGD
B 50000
0.139
0.0430
No
MNIST
LeNet SGD
B = 60000
0.0936
0.0430
No
GPT2-(small, medium, large) pretrained on Open WebTett. We used the Adam optimizer with learning 6 10-4, incorporating 2,000 warmup steps and decay schedule down to 6 10-5, following the experiment setup in NanoGPT. We used a batch size = 65,536 tokens and weight decay = 0.1 for total of 500,000 steps.
GPT2-(small, medium, large) fine-tuned on OpenwebText Shakespeare. We used the Adam optimizer with a learning rate 1 x 10-5 incorporating 1,000 warmup steps and a decay schedule down to 10 6. We used a batch size 32,768 tokens and weight decay 0.1 for total of 5,000 steps.
TimyLlama fine-tuned on MathQA. We fine-tuned the TinyLlama model with LoRA adapters, with rank 16, alpha 32, dropout 0.1 specified in the original LoRA experiment (Hu et al., 2021). We used the Adam optimizer with a learning rate 1 10-4 incorporating 500 warmup steps and decay schedule down to 1 x 10-5, We used batch size = 32 and weight decay 0.1 for 1,000 steps.
D. Supplementary Experiment Results of Convolutional Neural Networks
Three phases in large-batch training. In this section, we will further discuss the three phase changes large-hatch image classification training. We followed the same experiment setup in Table as stated in Appendix C.1.2. First, in Phase (B < 640), moderately large-batch (B 512) training indeed finds sharper minima than small-batch (B 128) training, while the plwer-law spectrum still holds well. Power laws may guarantee that the top eigenvalues of larce-batch trained networks are all larger than the corresponding eigenvalues of small-batol trained networks. The main challenge of large-batch training in Phase I is consistent with the common belief that large- batch training suffers from sharp minima and, thus, leads to bad generalization (Hoffer et al., 2017). The minima sharpness measured by & increases with the batch size.
Second, in Phase II (768 < < 50000), the spectrum large-batch (B = 1024) trained networks does not exhibit power laws but visually similar to the spectrum of underparameterizee models in Figure 5. In Phase II, large-batch trained overparameterized models behave like underparameterizeed models from a spectral perspective, and, thus, can lead bad
16

---

Investigating the Overlooked Hessian Structure
generalization. The phase transition from Phase to Phase II occurs in narrow range of 640 B 768, which is visually observable in Figure 7a and statistically observable in Table D.
Third, in Phase III (B ~ 60000), extremely large-batch training (B 60000) cannot optimize the training loss well or find the Hessian spectra similarly to random initialized neural networks. Phase III indicates that, sometimes, bad convergence rather than sharp minima can become the main performance bottleneck in large-batch training when the batch size is too large.
10
10
Clean Labels
Label Noise 40%
100
Label Noise 80%
100
Eigenvalue
Eigenvalue
10-1 1
10-1
10
10-2
Clean Labels
Label Noise 40%
Label Noise 80%
10-3
10
100
10
10
103
100
10
102
103
Eigenvalue Rank
Eigenvalue Rank
Figure 15, The spectrum in the presence of noisy labels. Top: MNIST Trainset. Bottom: MNIST Testset.
Our Hessian spectra analysis discovery differs from traditional beliefs that different phases exist in training. In phase 2, the Hessian eigenvalue increases and breaks the power-law structure, while the model's performance is much superior compared to untrained neural networks. We may leave a deeper investigation for future work.
10a
Table 5. The Kolmogorov Smimmo statistics the Hessian speetra for various batch sizes under the same number of training iterations.
10a
Dataset
Model
Training
Batch Size
dks
d
Power-Law
10F
MNIST
LeNet
SGD
B
0.0527
0.0430
No
lenet, 0=1200-rain-100
lenet_Be5 Bm511-train-1000
MNIST
LeNet
SGD
B 512
0.0934
0.0430
No
10-1
lenet, 0-000ttriinn00000
MNIST
LeNet
SGD
B 640
0.1138
0.0430
No
lenet_ B0766--rrin--000
MNIST
LeNet
SGD
B 768
0.1242
0.0430
No
lonet, B-1024-rrain-100
MNIST
LeNet
SGD
B 1024
0.0999
0.0430
No
lenet B-16384 train-n00nn 1000
MNIST
LeNet
SGD
B 16384
0.0271
0.0430
Yes
10a
10a
10a
10a
Eigenvalue Rank
Figure The Hessian spectrum for various batch sizes under the same number of training iterations.
Clean and Random Labels. We presented the spectrum of learning with clean labels and random labels in Figure 17. The number of top outliers obviously increases, because random labels make the dataset more complex. However, even if the pairs of instances and labels have little knowledge, we still observe the power-law spectrum after the dozens of top outliers. This may suggest that, even the labels are random, neural networks can still learn useful knowledge from the instances only.
Overfitting and Noisy Labels. As DNNs overfit noisy labels easily, previous papers choose learning with noisy labels an important setting for evaluating overfitting and generalization (Han et al., 2020; Xie et al., 2021a; He et al., 2022). Figure 15 shows that overfitting label noise makes the Hessian spectra less power-law on both the corrupted training dataset and the clean test dataset. In contrast, in the absence of noisy labels, the power-law spectra exist on both the training dataset and the test dataset.
Transferability. transfer learning, people believe that model pretrained on one dataset may learn useful representations
17

---

Investigating the Overlooked Hessian Structure
100
100
Training Test
Gaussian Data
MNIST with Random Labels
10
10
Eigenvalue
Eigenvalue
10-2
10-2
10
10-3
10
10
100
10
102
103
100
10
102
103
Eigenvalue Rank
Eigenvalue Rank
Figure 17. (a) The spectrum of LeNet on (training and test) MNIST with randomly shuffled labels. (b) The spectrum of LeNet on Gaussian data with randomly shuffled labels is highly similar to that MNIST with randomly shuffled labels.
102
10
Eigenvalue
10
10-1
Pretrained an CIFAR-100
Pretrained CIFAR-10
10
100
10
102
103
Eigenvalue Rank
Figure 18. Transferability of the power-law Hessian structure. LeNet which is pretrained on CIFAR-100 may still exhibit power-law Hessian spectra when evaluated on CIFAR-10.
for relevant datasets downstream tasks. When pretrain model CIFAR- 100 and evaluate its Hessian spectrum CIFAR-10, we surprisingly discover that the power-law Hessian structure successfully transfers. This may measures the usefulness of learned representations.
Power Iteration vs. Lanczos Algorithm. We compared the spectra computed via Power Iteration Algorithm and Lanezos Algorithm Figure 19. shows the top eigenvalues estimated via Power Iteration Algorithm are highly consistent with the top eigenvalues via the Lanezos Algorithm. It also demonstrates that the power-lawe spectrum is caused by the properties of deep learning rather than the stochasticity of Lanezos Algorithm.
We presented the power-law eigengaps MaShion-MNIST in Figure 21. It shows that the power-law eigengaps on Fashion-MNIST are highly consistent with the pawer-law eigengaps on MNIST.
We presented the power-law spectrum of the covariance matrix of stochastic gradient noise of FCN on MNIST in Figure 27. As the inverses of the power-law variables are power-law, the covariance spectrum shows heavy-tail properties. demonstrates that the heavy-tail property belongs to deep neural networks rather than SGD itself.
Avila Dataset We presented the power-law spectrum of two-layer FCN Avila Dataset in Figure 20. It shows that the power-law spectrum of neural networks may also generally exist in non-convolution neural networks trained non-image dataset. Particularly, we note that the Avila Dataset has only ten attributes, including intercolumnar distance, upper margin, lower margin, exploitation, row number, modular ratio, interlinear spacing, weight, peak number, and modular ratio/ interlinear spacing. These attributes are essentially different from the pixels in image datasets.
Supplementary Empirical Results on CIFAR-10. The spectra of LeNet on CIFAR-10 10 trained via various optimizers
18

---

Investigating the Overlooked Hessian Structure
10
Lanczos Algorithm Power Iteration
10a
Avila
Eigenvalue
10
Eigenvalue
10
10
10-1
10
100
10
102
10-2
109
10 Eigenvalue Rank
102
Eigenvalue Rank
Figure 19. The spectrum via Power Iteration Algorithm is highly consistent with the spectrum via Lanezos Algorithm. It also shows that the power-law spectrum caused by the properties of deep learning rather than the stochasticity of Lanezos Algorithm.
Figure 20. The spectrum of FCN on Avila Dataset. It shows that the power-law spectrum of neural networks may also exist in non-image datasets.
102
102
Eigenvalue or Eigengap
Eigenvalue or Eigengap
Eigengaps Eigenvallues
Eigengaps Eigenvalues
100
100
10
10-2
10
10-4
10-6
10-6
100
10
102 Rank
10
10
10
102 Rank
103
(a) Eigenvalue Rank
(b) Eigengap Rank
Figure 21. The power-law Hessian eigengaps in deep learning. Model: LeNet. Datset: Fashion-MNIST. Subfigure (a) displayed the cigengaps by original rank indices sorted by cigenvalues. Subfigure (b) displayed the eigengaps by rank indices re-sorted by eigengaps.
19

---

Investigating the Overlooked Hessian Structure
103
Pearson 0.7
0.27
SGp Adam AMf-trred AdaBound Yogl
102
AM55rrd
Eigenvalue
Test Error
10
0.26
Adul
100
0.25
10
0.24
10
100
10a
102
103
1.1
1.2
1.3
Eigenvalue Rank
Figure 22. The power-law spectra hold across optimizers. Moreover, the slope magnitude 8 is indicator of minima sharpness and predictor of test performance. Model:LeNet. Dataset: CIFAR-10.
Pearsor-0.985
3000
Opt
Peanson: 972 Ostmmeer
600
2500
400
Tr(H)
2000
200
1500
1000
0
1.1
1.2
1.3
1.1
1.2
1.3
S
S
Figure 23. The slope magnitude å closely correlates with the largest Hessian eigenvalue and the Hessian trace. Model:LeNet. Dataset: CIFAR-10.
showed in Figure 22. Figures 4 and 23 shows that the slope magnitude closely correlates with the largest Hessian eigenvalue and the Hessian trace.
We presented the power-law spectra of ResNer18 on CIFAR-10 in Figure 24, It shows that the power-law spectra hold for ResNet, a representative of the modern neural network architectures, as well as simple CNNs/FCNs. Due to the GPU memory limit, we may only display the top 50 eigenvalues for ResNet18, However, the KS test still supports accepting the power-law hypothesis.
We report the spectra of large-batch trained ResNet18 CIFAR-10 in Figure 25. It indicates that the phase transition behaviors of the spectra with respect to batch size generally exist. However, it seems that Phase II and Phase III merge into one phase for ResNet18 8 on CIFAR-10. 10.
Figure 26 shows that the small width neural networks may also break the power-law spectrum like small depth. This also supports that overparameterization or large model capacity is necessary for the power-law spectrum.
Rethinking the heavy-tail phenomenon SGD. The heavy-tail property of SGD has been hot and arguable topic recently (Simsekli al., 2019; Panigrahi al., 2019; Gurbuzbalaban al., 2021; Hodgkinson & Mahoney, 2021; Xie et al., 2021b; bb; Li et al., 2021). Note that the power-law distribution is one of the most common heavy-tail distributions in the real world. We argue that the arguable heavy-tail property of SGD may depend on the power-law Hessian spectrum rather than SGD itself, as gradient noise covariance critically depends on the Hessian. We present the power-law spectra of gradient noise covariance in Figure 27.
We specifically study the Hessian spectrum Linear Neural Networks (LNNss) which has no ReLu and BatchNorm. Figure
20

---

Investigating the Overlooked Hessian Structure
10
10a
Eigenvalue
Eigenvalue
100
10
Vanilla
B=120
Random
B-1024
B-1152
10
R-1280
B-2048
0=16384
10
100
10 Eigenvalue Rank
10
100
104
Eigenvalue Rank
Figure 24. The power-law spectra of ResNet18 on CIFAR-IO. shows that the power-law spectrum of neural networks may also exist in modern neural network architectures (ResNet) as well as simple CNNs/FCNs.
Figure 25. Batch size matters to the spectrum. Model:ResNet-Il 18.
Dataset: CIFAR-10. The sharp phase transition occurs 1152 B < 1280.
104
102
112
B=100
10
B-200
Eigenvalue
B-300
Eigenvalue
10
10-1
Width= 10
Width-20
30
10
W0
10 -2
Width=70
Width=100
10 10-3
100
101
102
10
10 -4
Eigenvalue Rank
100
102 Eigenvalue Rank
101
Figure 26. The spectra are not power-law for neural networks with a small width(~~ 10), but gradually become more power-law (more straight in the log-log plot) the width increases. This may also suggest the plwer-law spectrum depends on model capacity. Model: Two-layer FCN. Dataset: MNIST.
Figure 27. The power-law spectrum of gradient noise covariance exists in deep learning for various batch sizes. Model: Fully Con- nected Network(FCN), Dataset: MNIST.
21

---

Investigating the Overlooked Hessian Structure
10
10a
104
S6D ECDDEppch
5CID-Ppach
Eigenvalue
10a
SGD EpDDh 10
10
5CDD-Ppoch 30
Elgenvalue
5CD SCD.Epach 50
10-1 -
10"1
10
LNN LNN+BatchNorm
10"
LNN+ReLU
10-3
100
10a
102
10a
10
Eigenvalue Rank
10a
10a
10
10a
Eigenvalue
Figure 28. The spectrum of LNN with or without BatchNorm and ReLu.
Figure 29. The spectrum of LeNets trained with various epochs.
29 shows that the spectrum of LNNs is not power-law but more like spectra of underparameterizze models. It may indicate that nonlinearity and BatchNorm both help improve model capacity. LNNs equipped with ReLu or BatchNorm immediately recover the power-law spectra again.
E. Supplementary Experiment Results of Large Language Models
E.1. Power-law Hessian Eigengaps in LLM
In this subsection, we report the power-law spectrum in Hessian eigengaps of fine-tuned GPT2-small. As previous results of LeNet (Figure 2, 21), GPT2-small as a language model, demonstrated that learning dynamics similarly takes place in low-dimensional learning subspace as proved in Appendix B.
10a
10a
Eigenvalus or Eigengap
101
Elgenvalue or Elgengap
10'
100
10a
10-1
10-1
10-2
10-2
10
Eigenvalues Elgengap
10
Eigenvalues Eigengap
10
10
10'
101
102
10a
109
10a
10'
10'
Eigenvalue
Eigengap Rank
Figure 30. The power-law Hessian eigengaps. Model: GPT2-small. Dataset: Shakespeare, Subfigure (a) displayed the eigengaps by original rank indices sorted by eigenvalues. Subfigure (b) displayed the eigengaps by rank indices re-sorted by eigengaps.
E.2. Layer-wise Power-law Spectra
Due to computational constraints, we reported the Hessian spectra only using the parameters of the last layer in previous sections. In this section, we will present and discuss the power-law spectrum at different layers of the fine-tuned GPT2-small model. Figure 31 presented the power-law spectrum at 2, 5, 5,8, 11 layers of GPT2-small checkpoints fine-tuned on the Shakespeare dataset. The phenomenon is clear: the power-law spectrum forms early in training, particularly in the earlier layers, and the eigenvalue magnitudes decrease across layers.
E.3. Fine-tume tuning with Low-dimensional Adaptations (LoRA)
In this section, we report the Hessian eiganvalues of the low dimensional adapters (LoRA (Hu et al., 2021)) for efficient fine-tuni the-tuning of the pretrained TinyLlama- 1B model on the MathQA dataset. The top 1000 eigenvalues are computed for
22

---

Investigating the Overlooked Hessian Structure
301
301
201
Eige Eigenaatiz
Eigenaluu
Ihaura-US
patan
Insture:300
(hstur-22055
#robire:001
Frebuee5000
109
10
109
201
200
10'
107
201
Egensstic
Egenstto
Egenaalu
(a) 2d Layer
(b) 5th Layer
(c) 8 Layer
(d) 12th Layer
Figure:3 31. Power-law spectrum at different layers and checkpoints of the GPT2-small model. Dataset: Shakespeare.
the Hessian of all LoRA layers displayed, and may observe that a hessian Power-law structure is formed after 500 fine-tuning steps.
We then discuss how generalization of the fine-tuned LoRA adapters relate to the power-law Hessian structure. In Figure 32, the KS distance (d&a) calculated from the Hessian spectra of the adapters, initially increases during the first 100 steps of fine-tuning before decreasing alongside the test loss. Although the largest eigenvalue A increases at step 50 compared its random initialization, the Hessian trace Tr(H) consistently decreases throughout fine-tuning. This behavior suggests that for pretrained models with good proficiency, the power-law spectra may diverge from expected patterns, and the Hessian trace Tr(H) may be more reliable indicator of generalization capability according to Figure 32.
lora
lonerrandom
3.5
lora-30
lora-50
3.0
lera-100
lora-100
Eiganvalue
10a
Iora-300
lora 200
Ione-200
2.0
lora-1000
2.0
104
1.0
10
10
107 Exgevvauue Rank
10
0.02
0.00
0.08
0.10
0.0
0.0
Figure 32. The behaviour of the Hessian spectrum of all LoRA adapters aligns similarity the original model. LoRA adapters randomly initialized before fine-tuning. Model: LoRA Layers (adapted on TinyLlama). Dataset: Mat.
We further computed the Hessian spectrum of the pretrained last layer of L'mnaalama-br merged with the fine-tuned LoRA Adapter in Figure 33. Due to the consistency in most parameters, there are significant changes in behaviours of the Hessian spectrum at different checkpoints of fine-tuning, nor the minima sharpness.
pretran merge 30
merge 100
200
Eigenvalus
Test Loss
merge 1000
merge 500
Pest LO5S
Loss
Criica
pretrein
pretrein
10"
merge-50
marge-130
merge: 100
mee mee
merge-230
500
merge
merge:
10
10a
10>
0.000 6.025
8,050
0.300 du.
0.075
0.125
420
425
450
435
500
520
552
575
500
5
6x 3p4 7x
6x30
2 10'
Eigenvalue Pank
7X(4)
Figure 33. Due to consistency in most parameters, the Hcssian spectrum does not change significantly at different fine-tuning checkpoints of the merged TlmyL with LoRA adapter. The observation demonstrated the minimal impact the original model when tuning with Row-Rank Adaptations. Model: TinyLlama 1.IB 1--Chat-v1.0 1.0 (merged with LoRA adapter). Dataset: MathOA.
23

---

Investigating the Overlooked Hessian Structure
F. Kolmogorov-Smirnov Goodness-of-Fi Test
In this section, we introduce how to conduct the Kolmogorov-Smimoo Goodness-of-Fit Test for the self-containedness purpose.
As mentioned above, work used Maximum Likelihood Estimation (MLE) (Myung, 2003; Clauset al., 2009) for estimating the parameter of the fitted power-law distributions and the Kolmogorox-Smirnov Test (KS Test) (Massey Jr, 1951; Goldstein et al., 2004) for statistically testing the goodness of the fit. The KS test statistic is the KS distance des between the hypothesized (fitted) distribution and the empirical data, which measures the goodness of fit. Mathematically, the KS distance defined as
dks sup s F(A)I.
(15)
where F" (A) is the hypothesized cumulative distribution function and F()) is the empirical cumulative distribution function based the sampled data (Goldstein al., 2004). The estimated power exponent via MLE (Clauset et al., 2009) be written
A Acutoff
1=1+ K
Σ In
(16)
where K is the number of tested samples and set Acutoff Ap. We note that the Powerlaw library (Alstott al., 2014) provides a convenient tool to compute the KS distance, dks. and estimate the power exponent.
According to the practice of KS Test (Massey Jr, 1951), we first state the power-law hypothesis that the tested spectrum is power-law. If des higher than the critical value de at the a = 0.05 significance level, the KS test statistically will support the power-law hypothesis (we cannot reject the power-law hypothesis). We display the critical values Table
We conducted the KS tests for all of our studied spectra. We display the KS test statistics and the estimated power exponents 3 with standard errors well the corresponding á in Tables 7, 8, 9, and 10. In the tables, we take the base hyperparameter setting in Appendix C as the default setting. For better visualization, we color accepting the power-law hypothesis in blue and color rejecting the power-law hypothesis (and the cause) in red.
Table The Table of Kolmogorov- -miinov Test Critical Values (Significance Level), which was first reported in Massey Jr (1951). If the KS distance du, lower than a critical value, such as the = 0.05 significance level. Note that K is the number of tested eigenvalues.
would reject the null hypothesis and accept the power-law hypothesis
Sample size
a
0.2
a = 0.15
a =
0.1
a = 0.05
a = 0.01
K 35
1.02
1.14
1.22
1.36
1.63
vk"
vk
v/
v/
vi
K 50
0.151
0.161
0.173
0.192
0.231
K 1000
0.0338
0.0360
0.0386
0.0430
0.0515
24

---

Investigating the Overlooked Hessian Structure
Table The Kolmogorov- Smirnov statistics of LeNet on MNIST and Fashion MNIST. The estimated power exponent / and slope magnitude 8 are also displayed.
Dataset
Model
Training
Sample size
Setting
da.
d
Power-Law
B 8
MNIST
LeNet
Random
1000
0.0796
0.0430
No
MNIST
LeNet
SGD
1000
0.00900
0.0430
Yes
1.991 + 0.031
1.009
MNIST
LeNet
Vanilla SGD
1000
0.0103
0.0430
Yes
1.914 0.029
1.094
MNIST
LeNet
Adam
1000
0.00962
0.0430
Yes
1.873 0.028
1.145
MNIST
LeNet
AMSGrad
1000
0.00987
0.0430
1.845 0.027
1.184
MNIST
LeNet
AdaBound
1000
0.00889
0.0430
1,904 + 0.029
1,106
MNIST
LeNet
Yogi
1000
0.00966
0.0430
Yes
1.834 + 0.026
1.198
MNIST
LeNet
RAdam
1000
0.0164
0.0430
1.889 0.028
1.125
MNIST
LeNet
Adai
1000
0.0101
0.0430
1.892 0.028
1.122
MNIST
LeNet
PNM
1000
0.0127
0.0430
1.846 0.027
1.181
LeNet
Lookahead
1000
0.0101
0,0430
1.982 0.031
1.018
MNIST
LeNet
DiffGrad
1000
0.0105
0.0430
1.834 0.026
1.198
MNIST
LeNet
SGD
1000
B 128
0.00900
0.0430
Yes
1.991 + 0.031
1.009
MNIST
LeNet
SGD
1000
B 512
0.00787
0.0430
Yes
1.894 +: 0.028
1.119
MNIST
LeNet
SGD
1000
B 640
0.0125
0.0430
Yes
1.838 + 0.027
1.194
MNIST
LeNet
SGD
1000
B 768
0.278
0.0430
No
MNIST
LeNet
SGD
1000
B 1024
0.129
0.0430
No
MNIST
LeNet
SGD
1000
B 8192
0.240
0.0430
No
MNIST
LeNet
SGD
1000
B 16384
0.249
0.0430
No
MNIST
LeNet
SGD
1000
B 32768
0.201
0.0430
No
MNIST
LeNet
SGD
1000
B 50000
0.139
0.0430
No
MNIST
LeNet
SGD
1000
B = 60000
0.0936
0.0430
No
MNIST
LeNet
SGD
1000
N 600
0.205
0.0430
No
MNIST
LeNet
SGD
1000
N 800
0.0399
0.0430
1.995
0.031
1.004
MNIST
LeNet
SGD
1000
N 1000
0.0198
0.0430
Yes
2.128
0.036
0.886
MNIST
LeNet
SGD
1000
N 3000
0.0159
0.0430
Yes
2.091
0.034
0.917
MNIST
LeNet
SGD
1000
N 6000
0.0151
0.0430
2.001
0.032
0.999
MNIST
LeNet
SGD
1000
40% Label Noise
0.180
0.0430
No
MNIST
LeNet
SGD
1000
80% Label Noise
0.157
0.0430
No
MNIST
LeNet
SGD
1000
Random Labels
0.0482
0.0430
No
MNIST
LeNet
Random
1000
0.0971
0.0430
No
Fashion-MNIST
LeNet
SGD
1000
0.0132
0.0430
Yes
1.939
0.030
1.065
MNIST
LeNet
SGD
1000
Eigengap Eigengap
0.0153
0.0430
Yes
1.550
0.017
1.817
Sishion-MNIST
LeNet
SGD
1000
0.0240
0.0430
Yes
1.520
0.017
1,922
MNIST
LeNet
SGD
1000
Epoch=
0.0321
0.0430
Yes
1.908 0.029
1.102
MNIST
LeNet
SGD
1000
Epoch= 2
0.0298
0.0430
Yes
1.920 0.009
1.087
MNIST
LeNet
SGD
1000
Epoch=
0.0354
0.0430
Yes
1.916
1.092
MNIST
LeNet
SGD
1000
Epoch= 10
0.0291
0.0430
Yes
2.029 + 0.033
0.972
MNIST
LeNet
SGD
1000
Epoch= 20
0.0268
0.0430
Yes
2.081 + 0.034
0.925
MNIST
LeNet
SGD
1000
Epoch= 30
0.0068
0.0430
Yes
2.074 + 0.034
0.930
25

---

Investigating the Overlooked Hessian Structure
Table Kolmogorov-S -Sminnov statistics LeNet on CIFAR-10 CIFAR-100. The estimated power exponent / and slope
magnitude 8 are also displayed.
Dataset
Model
Training
Sample size
Setting
dls
d.
Power-Law
CIFAR-10
LeNet
Random
1000
0.0663
0.0430
No
CIFAR-10
LeNet
SGD
1000
0,0279
0,0430
Yes
1.968 1 0.031
1.033
CIFAR-10
LeNet
Vanilla SGD
1000
0,0276
0,0430
Yes
1.935 +: 0.030
1.069
CIFAR-I0
LeNet
Adam
1000
0.0269
0.0430
Yes
1.806 + 0.025
1,241
CIFAR-I0
LeNet
AMSGrad
1000
0.0232
0.0430
Yes
1.786 + 0.025
1.271
CIFAR-10
LeNet
AdaBound
1000
0.0297
0.0430
Yes
1.901 = 0.028
1.110
CIFAR-10
LeNet
Yogi
1000
0.0184
0.0430
Yes
1.806 = 0.025
1,241
CIFAR-10
LeNet
RAdam
1000
0.0163
0.0430
Yes
1.733 0.023
1.363
CIFAR-10
LeNet
Adai
1000
0,0310
0.0430
Yes
1.918 + 0.029
1.090
CIFAR-10
LeNet
PNM
1000
0.0347
0.0430
Yes
1.911 0.029
1.098
CIFAR-10
LeNet
Lookahead
1000
0.0358
0.0430
Yes
1.964 0.030
1.037
CIFAR-10
LeNet
DiffGrad
1000
0.0303
0.0430
Yes
1.803 0.024
1.236
CIFAR-100
LeNet
Random
1000
0.0944
0.0430
No
CIFAR-100
LeNet
SGD
1000
0,0315
0.0430
Yes
1.908 = 0.029
1.101
CIFAR-100
LeNet
Vanilla SGD
1000
0.0379
0.0430
Yes
1.903 0.029
1.108
CIFAR-100
LeNet
1000
Evaluated on CIFAR-10
0.0306
0.0430
Yes
1.913
0.029
1.095
Table 9. The Kolmogorov Smirnov statistics of FCN. The estimated power exponent 3 and slope magnitude á are also displayed.
Dataset
Model
Training
Sample size
Setting
duss
de
Power-Law
B a
Avila
2Layer-FCN
SGD
50
0.0683
0.176
Yes
1.604 1 0.085
1.656
MNIST
ILayer-FCN
Random
1000
0.185
0.0430
No
MNIST
ILayer-FCN
SGD
1000
0.241
0.0430
No
MNIST
2Layer-FCN
Random
1000
0.129
0.0430
No
MNIST
2Laycr-FCN
SGD
1000
0,0112
0.0430
Yes
2.209
0.038
0.827
MNIST
4Layer-FCN
Random
1000
0.0628
0.0430
No
MNIST
4Layer-FCN
SGD
1000
0.0141
0.0430
Yes
2.201 0.833
1: 0.038
MNIST
2Layer-FCN
SGD
1000
1i
0.149
0.0430
No
MNIST
2La
SGD
1000
Width=20
0.185
0.0430
MNIST
2Layer-FCN
SGD
1000
Width=30
0.0656
0.0430
No
MNIST
2Ca
SGD
1000
Width=50
0.0187
0.0430
Yes
2.138 + 0.028
0.879
MNIST
2La
SGD
1000
Width=70
0.0376
0.0430
Yes
2.271
0.030
0.787
MNIST
2Layer-FCN
SGD
1000
Width=100
0.0112
0.0430
Yes
2.209 +: 0.038
0.827
26

---

Investigating the Overlooked Hessian Structure
Table 10. The Kolmogorov-Smirov statisties of ResNet18. The estimated power exponent and slope magnitude are also displayed.
Dataset
Model
Training
Sample size
Setting
diex
Power-Law
B +
CIFAR-10
ResNec18
Random
50
0.334
0.176
No
CIFAR-10
ResNet18
SGD
50
0.0803
0.176
2 = 0.162
0.873
CIFAR-10
ResNet18
Vanilla SGD
0.0891
0.176
2.193 = 0.169
0.838
CIFAR 10
ResNet18
Adam
50
0.0478
0.176
Yes
2.062 = 0.149
0,950
CIFAR-10
ResNet18
AMSGrad
50
0,0542
0.176
Yes
2.011 = 0.117
0.961
CIFAR-10
ResNet18
AdaBound
50
0.0588
0.176
Yes
2.029 = 0..16
0.971
CIFAR 10
ResNet18
Yogi
50
0.116
0.176
Yes
1.915 0.129
1.092
CIFAR-10
ResNet18
RAdam
50
0.168
0.176
Yes
1.794 0.1112
1.259
CIFAR 10
ResNet18
Adai
0.103
0.176
2.183 = 0.167
0.845
10
ResNet18
PNM
50
0.138
0.176
2.132 = 0.160
0.884
CI
ResNet18
Lookahead
50
0,110
0.176
Yes
2.098 = 0.155
0.911
CIFAR-10
ResNet18
DifIGrad
50
0.068
0.176
Yes
2.055 = 0.149
0.948
CIFAR 10
ResNet18
SGD
50
512
0.0561
0.176
Yes
2.146 = 0.151
0.936
CIFAR- 10
ResNet18
SGD
50
B 1024
0.0647
0.176
Yes
2.076 = 0.152
0.929
CIFAR-10
ResNet18
SGD
50
B 1152
0.0598
0.176
Yes
2.060 + 0.150
0.944
CIFAR- 10
ResNer18
SGD
50
1280
0.331
0.176
No
CIFAR-10
ResNet18
SGI)
50
2048
0.334
0.176
No
CIFAR-10
ResNet18
0.334
0.176
No
CIFAR-10
ResNet18
SGD
50
16384
0.343
0.176
No
CIFAR- 100
ResNet18
Random
50
0.373
0.176
No
CIFAR-100
ResNet18
SGD
0.108
0.176
2.299
0.184
0.770
27


Reviews:
[{"review_id": "f9C6t6nD8E", "reviewer": "Reviewer_tKCL", "summary": "This paper investigates the Hessian structure in CNNs and LLMs from the point of view of the power laws in the Hessian spectrum. It is shown that across a range of settings the power law like trend holds for the Hessian eigenvalues of a trained network, which however is not the case at random initialization. There is some rough theoretical interpretation that is provided based on the maximum-entropy principle. In addition to power laws in the Hessian spectrum, similar thing is shown for eigengaps. Alongside, there is a thorough empirical exploration of the effects of power law in regards to training dataset size, batch size, and some interesting trend is shown for its ability to correlate with the generalization during the course of pre-training/fine-tuning.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your support of our work. We try best to structure your concerns and address them as follows:\n\nQ1: Normalization by the Hessian trace in Eqn.2 may not be suitable when Hessian is indefinite.\n\nA1: Eqn.2 attempts to describe the power-law from a frequency perspective, and we reported true eigenvalues in experiments following Eqn.3 instead of being trace-normalized. Further KS tests on the absolute eigenvalues also suggest robust power-law structure.\n\nQ2: Measuring minima flatness with the Hessian inverse determinant is ad-hoc.\n\nA2: Mathematically, the absolute value of the determinant quantifies how the transformation scales n-dimensional volume, which naturally make Hessian a measure of minima flatness. It corresponds to the volume scaling factor of parallelepipeds in 3D, which implies sharp or flat curvature. We also draw parallels with research works that the Hessian determinant is a kind of sharpness measure [1] and Empirical Fisher determinant is an approximation of loss curvature [2]. We will improve the clarification in the revision.\n\nQ3: I am curious how this behavior holds at all eigenvalues. \n\nA3: Experiments on LeNet show the eigenvalues quickly decay to near zero beyond certain threshold. Researchers commonly focus on the top eigenvalues since they are significant and critically relate to the network's performance compared to smaller ones.\n\nQ4: Could use some measurements to see if the connection to generalization during training is robust / What are the numbers for GPT2 medium/large?\n\nA4: In addition to Pearson's correlation (p_corr), we have re-evaluated with Spearman's Rank Correlation (s_corr) and OLS's statistical significance (p_ols<0.05). For GPT2-{nano,small} finetuned on Shakespeare (Figure 10-11), s_corr={0.886,1.0} and p_ols={0.009,0.0}; For GPT2-{medium,large} pretrained on OpenWebText, p_corr={0.766,0.803}, s_corr={0.855,0.786} and p_ols={0.01,0.03}; the trend in the emergence of the Hessian power-law structure over training seems robust.\n\nQ5: When showing the trend with different batch sizes, the underlying setting is unclear.\n\nA5: We set epochs=50 with hyperparameters stated in Appendix C.1.2 in Table 4, thus all models are trained with the same total images. We follow your advice and use fixed total training steps, early experiments suggest that with more computation/same iterations, dKS for large-batch indeed decreases, while it is still significantly higher than small-batch training.\n\nQ6: The statement `model trained with limited training data finds minima ... like underparameterized models' is a bit strange.\n\nA6: We mean to convey that both data and parameter scaling could cause the power-law to emerge, and both scenarios of limited data and underparameterization would similarly break the power law and lead to many sharp directions in the loss landscape. We will clarify this more clearly in the revision.\n\nQ7: It seems like the power law test might fail if setting is more complicated in Table 1.\n\nA7: Thanks for this interesting question. We actually agree with your point. Too simple neural networks like LeNet may not well learn complex tasks like ImageNet, thus training loss will be very high and the power-law structure may break due to the poorly learned representation. Training LeNet however depend on ImageNet-size problems and we may supply it in the revision.\n\nQ8: How many data points are used to estimate the Hessian in Table 1?\n\nA8: All Table 1 experiments utilize the full training dataset of MNIST, CIFAR-10 etc. The same experiment on 1/10 dataset (5000 data points) has no noticeable difference.\n\nQ9: What do you mean by the networks being close to equilibrium above section 2.3?\n\nA9: Equilibrium often indicates that a system/network reaches a stationary distribution, which can help interpret NN optimization dynamics in theoretical ML works. We would remove the contents as it is not necessary in our work.\n\nQ10: It would have been interesting to read off some trends in the use of different optimizers.\n\nA10: Our findings primarily demonstrate the power-law existence, and the slope correlates to generalization and minima sharpness learned by different optimizers. A more fine relation to optimizer settings is an open question and beyond the main scope of this work.\n\nIn addition to addressing the primary question, we have incorporated the reviewers’ suggestions to correct typo errors, clarify ambiguities in Eqn 1-3, change 'random' labels to 'random-init' in Figures (9,10,14), and discuss the work of Mahoney on heavy tails. We sincerely thank the reviewer's feedback and constructive suggestions. It definitely inspires us to further improve our work.\n\nWe would highly appreciate it if you could kindly reconsider the rating, given the enhanced work quality under your kind help.\n\nRefs:\n\n[1] A Diffusion Theory For Deep Learning Dynamics: Stochastic Gradient Descent Exponentially Favors Flat Minima. ICLR 2021.\n\n[2] Where is the Information in a Deep Neural Network? arxiv:2019", "claims_and_evidence": "- In Eqn 2, when discussing the finite-sample power law, normalization is carried out by the Hessian trace. However, this may not a be suitable normalization when the Hessian is indefinite, which is definitely the case at a random initialization. A more appropriate choice would have been to consider the absolute value of the eigenvalue and then see when they are ranked, if the power law trend is present or not. Currently, I'm concerned how this would change the trend for the spectrum at random initialization. \n\n- The link to maximum entropy is interesting, but conceiving of measure flatness of the minimum with the determinant of **Hessian inverse** is ad-hoc. Normally, you would use a spectral quantity based on the Hessian, so det(H) would be fine. I believe this is soley being done for the purposes of their interpretation, else the power law has positive exponents, which would be a mismatch. Can the authors comment on this?", "methods_and_evaluation_criteria": "- I am curious if you were to look at all the eigenvalues, and not just the top 6000, how much of this behaviour holds. Of course, I understand the difficulty with such an endeavour, but this can still be done with a network having parameters in the range of 20K and see what happens.", "theoretical_claims": "yeah theorem 1, and I think the use of Hessian inverse for determinant is a bit ad-hoc as detailed elsewhere", "experimental_designs_or_analyses": "- The connection to generalization during training is interesting, but could use some measurements to see if the correlation is really robust or not.\n\n- When showing the trend with different batch sizes, it is unclear what the underlying setting is. Is the number of training steps kept the same across batch sizes? How is the learning rate adjusted? These things could make a big difference\n\n- The statement that \"the model trained with limited training data finds minima with many sharp directions like under parameterized models\" and their finding therein is a bit strange. Since, underparametrized is relative to number of parameters (which I believe are fixed), so actually when training set is 600, it is the most overparameterized setting. So I don't quite understand this statement in the paper and the trend. \n\nAlso, see claims and evidence section", "relation_to_broader_scientific_literature": "This is a complementary perspective to the study of Hessian in deep learning, and the initial results like the correlation trend with generalization seem promising.", "essential_references_not_discussed": "I think the work of Mahoney and co-authors on heavy tails could be discussed and compared to, since essentially both these works are effectively comparing the trends of spectral decay.", "other_strengths_and_weaknesses": "The paper contains some very interesting results, on the power-law structure. However, often a lot of the results are thrown at the reader, and the overall narrative is not as cohesive. Some of the theoretical discussion is not well justified. The potential link to generalization offers an interesting avenue, perhaps that could be elaborated and studied further, to result in a more tightly-knit paper. Also, I'd move some of the other marginal analyses to the appendix.", "other_comments_or_suggestions": "- Table 1 caption: no \\beta or slopes are mentioned\n\n- It would be worth elaborating on the link between Eqns 1, 2, 3 to reach a wider readership. \n\n- The 'random' in the column label 'Training' is confusing at first sight, and rather say something like, 'random-init'; else it could be thought of something else.", "questions_for_authors": "- Looking at table 1, it seems that the more complex the setting, the higher is the value of d_KS. So it seems like the power law test might fail if you go to a setting which is more complicated. Could the authors comment on the possible reason for this? Also, it would be nice to see the results on ImageNet and see what the trend is like!\n\n- How many datapoints are used to estimate the Hessian in each of these cases in Table 1? Ideally, the power law trend should not be too sensitive to this choice of datapoints, assuming a sufficient number of samples is used. Can the authors verify this aspect?\n\n- What do you mean by the networks being close to equilibrium, just above section 2.3? It is very vague at the moment. \n\n- What are the numbers for GPT-2 medium and large on OpenWebText?\n\n- It would have been interesting to read off some trends from looking at the use of different optimizer. But right now it's unclear if something can be said, except that this power law occurs for various optimizer choices.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ez9Y1GYKyI", "reviewer": "Reviewer_sJoB", "summary": "This paper investigates the power-law structure of the Hessian matrix in deep neural networks, including Convolutional Neural Networks (CNNs) and Large Language Models (LLMs). Key contributions include: A maximum-entropy principle from statistical physics is proposed to explain the emergence of the power-law structure, linking it to flat minima and generalization. The authors identify a power-law distribution in the Hessian spectra of well-trained NNs, contrasting with random or under-trained models. This structure is consistent across CNNs, LLMs, and Vision Transformers (ViTs). This paper indicate this power-law structure helps predict the generalization of LLMs.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your support of our work and constructive suggestions. We try our best to structure your concerns and duly address them as follows:\n\nQ1: The author cannot provide a rigorous explanation of the maximum entropy principle. However, this article still provides a new perspective on how to predict the generalization of LMMs.\n\nA1: Thanks for the suggestion. First, to recognize our works, we would like to convey that we are the first to provide an interpretation of the power-law hypothesis while following the maximum entropy principle in statistical physics. We refer you to the works (Bahri et al., 2020; Torlai & Melko, 2016) we referenced in line 117 for better understanding. To ground our approach, we draw parallels with research works in the machine learning field that adopt similar theoretical frameworks: Dinh et al. studied minima flatness with similar Hessian-based measures [1] and Baldassi et al. also interpreted the minima flatness of neural networks from an entropy perspective [2]. However, we understand that this work lacks a strong justification for using the maximum entropy principle for power-law interpretation. We are willing to improve the theoretical analysis and justify the relationship between the maximum entropy principle and minima flatness of neural networks in future revisions. \n\nQ2: The author should mention the computational cost and time required to obtain the Hessian matrix and its spectrum.\n\nA2: Thanks for your suggestion. We employed the Stochastic Lanczos Quadrature (SLQ) algorithm with optimizations tailored for large neural networks in our work. SLQ consists of K Lanczos iterations and M integral steps to reduce estimation error. The primary computational expense arises from repeated Hessian-vector product (HvP) computations in each Lanczos step, requiring two backpropagation passes, which we implemented using PyTorch's auto-differentiation framework. To manage memory overhead, we stored intermediate SLQ computations as temporary tensors on disk, reducing space requirements to a level comparable to training the network via SGD with momentum.\n\nAssuming the time complexity of a single backpropagation step is O(N), the overall SLQ complexity is O(N^2 \\cdot K \\cdot M). For all LLM experiments presented in our paper, we set M = 3, K = 1000 and a sampled batch size of 256 as in Zhang et al. (2024b). In our implementation, computing 3000 Hessian eigenvalues for the last layer of GPT2-small takes around 12-14 GPU hours on a standard RTX 4090. While PyTorch's current implementation does not support explicit storage of the computation graph, in theory, the time complexity could be further reduced to O(N \\cdot K \\cdot M) by pre-saving gradients and the computation graph before SLQ execution. Consequently, computing the top-N eigenvalues has a time complexity equivalent to training the network for N steps.\n\nQ3: Is the small KS distance of untrained networks widely recognized criteria, or could it be that many untrained models themselves also satisfy the power-law hypothesis?\n\nA3: We do not believe that many untrained models could satisfy the power-law hypothesis. We would like to convey that the critical values (d_C) of the KS Tests are derived from a=0.05 significance level (Massey Jr, 1951) and we would reject the null hypothesis for d_KS below the critical values. As we have conducted a large number of experiments on models from CNNs to LLMs, we have not observed the d_KS of any untrained networks are below d_C. Some untrained models may have a close d_KS but they are still far from accepting the power-law hypothesis.\n\nFinally, we sincerely thank the reviewer's feedback. It definitely inspires us to further improve our work and clarification for more readers. We would highly appreciate it if you could kindly reconsider the rating of our work given the initial score of 3 and the addressed concerns. Your suggestion and action will both make an addition to our community.\n\nRefs:\n\n[1] Dinh, Laurent, et al. \"Sharp minima can generalize for deep nets.\" ICML, 2017.\n\n[2] Baldassi, Carlo, Fabrizio Pittorino, and Riccardo Zecchina. \"Shaping the learning landscape in neural networks around wide flat minima.\" PNAS, 2020.", "claims_and_evidence": "The claims in the submission are generally supported by a theoretical interpretation and experimental discovery. However, the theoretical analysis here is merely an intuitive explanation and does not provide a rigorous proof.", "methods_and_evaluation_criteria": "The evaluation criteria for generalization are widely recognized and reasonable standards.", "theoretical_claims": "The proof of the theory are correct but somehow straightforward.", "experimental_designs_or_analyses": "From an experimental perspective, the experimental design of this paper can illustrate the claims.", "relation_to_broader_scientific_literature": "This article adopts the well-known maximum entropy principle to intuitively explain the power-law distribution of the eigenvalues of the network Hessian matrix. The main contribution lies in extensive experimental observations, which is not very related to prior works.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "The theoretical analysis is the weakness of this article; the author only somewhat tenuously accepts the maximum entropy principle and can not\nprovide a rigorous explanation of the relationship between the two sides. However, this article still provides a new perspective on how to predict\nthe generalization of LMMs.", "other_comments_or_suggestions": "See the following Questions.", "questions_for_authors": "1. The author should mention the computational cost and time required to obtain the Hessian matrix and its spectrum. This could become unaffordable when applied to advanced LMMs?\n\n2. In the evaluation criteria of this article, when the KS distance exceeds 0.05, it is considered as not satisfying the power-law hypothesis. However, as shown in Table 1, the DS distance of untrained networks is also quite small. Is this value a widely recognized criteria, or could it be that many untrained models themselves also satisfy the power-law hypothesis?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "No ethical concerns identified. This work focuses on the empirical performance of existing networks."}}, {"review_id": "Sfufu1nw8J", "reviewer": "Reviewer_qcZb", "summary": "In this work, the authors report a previously overlooked power-law Hessian structure in well-trained deep neural networks, which includes CNNs and LLMs. The authors show that both the top few thousand eigenvalues as well as the eigengaps follow approximately a power-law $p(\\lambda) = Z_c^{-1} \\lambda^{-\\beta}$, which they test statistically using the Kolmogorov-Smirnov Test (KS Test). This phenomenon is observed across multiple optimizers, datasets, and architectures and differs from the eigenspectrum of randomly initialed networks. \nIn addition, the authors provide a theoretical interpretation via the maximum-entropy principle.\nThe authors also provide additional ablations, including different optimizers, overparametrizing the network, the size of training data, and the batch size, and show how the fitted slope magnitude correlates well with the largest Hessian eigenvalue, the Hessian trace, and the test performance and how the KS distance $d_{KS}$ predicts the generalization performance of LLMs models, which conventional sharpness measures ($Tr(H)$ and $\\lambda_{\\max}$) fail to capture.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your insightful feedback. Below, we duly address your concerns through additional experiments and careful responses.\n\nQ1: Is it possible to reproduce some of the results for instance on ResNet18 trained on Cifar10?\n\nA1. Yes, it is possible. The Kolmogorov-Smirnov distance decreases from d_KS = 0.198 in randomly initialized ResNet18 to d_KS = 0.031 (< d_c) in the well-trained model, providing further empirical support for our paper's central conclusion. We expect to supply these results in the revision.\n\nQ2: The experimental designs and resulting analyses are sound, as it would be nice to provide further figures in the setting of image classification, similar to the section on LLMs.\n\nA2. Thank you for your complementation and suggestion. We will supply these results in the revision.\n\nQ3: It is also interesting to see how the slope magnitude correlates with other sharpness measures and correlates well with generalization in LLMs.\n\nA3. Thank you for your suggestion. However, we are unable to identify significant correlations between slope and other metrics on LLMs while it works well on CNNs. In particular, for GPT2-small fine-tuned on the Shakespeare dataset, the Pearson correlation coefficient of the slope magnitude to test loss and hessian trace are -0.173 and 0.414, drawing a negative conclusion on the validity of using the Hessian spectrum slope as a generalization metric.\n\nQ4: What is meant by the so-called equilibrium of DNNs, which is mentioned in line 124?\n\nA4: Thank you for raising this question. We have followed the concept from statistical physics (e.g. Boltzmann Machine) we referenced in line 117. Equilibrium often indicates that a system or a network reaches a stationary distribution, which can help in understanding the optimization dynamics of NNs in some ML works. However, we would like to remove the contents of equilibrium, as it is not a necessary concept in our work.\n\nQ5: Can the authors explain the eigenvalue distribution of sharp minima (e.g. large-batch training with B=768) intuitively seems to obey the power-law distribution less than a randomly initialized network?\n\nA5: Thank you for your question. We have discussed the observation that large-batch trained networks (B > 768) do not exhibit power laws, resembling under-parameterized networks in Appendix D, and summarized as the three phases in large-batch training. Our Hessian spectra analysis discovery differs from traditional beliefs that different phases exist in training. In phase 2, the Hessian eigenvalue increases and breaks the power-law structure, while the model's performance is much superior compared to untrained neural networks. We may leave a deeper investigation for future works.\n\nQ6: Can the authors elaborate how the work by Wu et al. [2017] (Figure 2 right) fits into the analysis?\n\nA6: Thank you for your suggestion. Wu et al. utilize an attack set to control the model's performance which differs from the principle experiment setup in our main paper. We believe the behaviour cannot be guaranteed to be consistent across different data recipes and setups. We have instead discussed a similar setting with noisy labels presented in Figure 15, Appendix D. The red line refers to the model with the best generalization performance as trained on the fully clean dataset, corresponds to the red line of the model with 92.92% accuracy in Figure 2 (right) of Wu et al. (2017). We may observe that the power-law behaviour of the Hessian spectrum computed on the Test set is disturbed by the noisy labels introduced. We may also observe that the magnitude of eigenvalues increases for networks trained with adversarial labels and poorer generalization performance.\n\nFinally, we sincerely thank the reviewer's feedback. It definitely inspires us to further improve our work and clarification for more readers. We respectfully hope that the reviewer can reevaluate our work given the responses addressing your main concerns.", "claims_and_evidence": "Generally, I think that all claims are supported by clear and convincing evidence. For some of the experiments, I believe that one needs to extend the figures to fully make the claims of the authors. For instance, in the experiment on varying Batch Size, where the authors describe three different phases of large-batch training, it would be beneficial to also include the Test Error to Figure 8 to more clearly different phases apart from the qualitative shape of the eigenvalue distribution. Actually, it would be interesting to see if one could have a similar plot as e.g. Fig. 10 or Fig. 11, where the $d_{KS}$ for each batch size is plotted against the Test Loss/Error.", "methods_and_evaluation_criteria": "The methods and evaluation criteria seem reasonable to me. The authors chose a reasonable range of different models, including FCN, LeNet, ResNet18, and ViTs for some image classification datasets, including MNIST, Fashion-MNIST, Cifar-10/100, and non-image Avila as well as GPT2-models on OpenWebText, Shakespeare, and MathQA.\nAlso, the ablation studies were quite thorough, including training data size, model size, batch size, and optimizers.\nOf course, it would be beneficial if the results could also be reproduced in some other settings, as Figure 3, Figure 4 and Figure 7 are all conducted on LeNet trained on MNIST. Is it possible to reproduce some of the results for instance on ResNet18 trained on Cifar10?", "theoretical_claims": "I only checked the proof for Theorem 1 in Appendix A and think that it is fine as is.", "experimental_designs_or_analyses": "The experimental designs and resulting analyses are sound as I have already elaborated in the section above on \"Claims and Evidence\". \nAs elaborated above, it would be nice to provide further figures in the setting of image classification, similar to the section on LLMs. e.g. plotting $d_{KS}$ against the test loss for the ablation study on large-batch training and training-data size.", "relation_to_broader_scientific_literature": "The analysis of $d_{KS}$ as a generalization measure in LLMs relates to previous results by Jiang et al. [2019] and Kaur et al. [2023], which uses sharpness measures such as the Hessian trace $\\text{Tr}(H)$ and the largest eigenvalue $\\lambda_{\\max}(H)$ as a measure of generalization. \n\nThe power-law Hessian structure relates to previous work analyzing the Hessian structure of DNNs, including work by Pennington & Bahri [2017], who study the eigenvalue distribution through random matrix theory; Singh et al. [2021], who study the rank of DNNs; Liao & Mahoney [2021], who study the Hessian spectra of nonlinear models and Dauphin et al. [2024], who investigate the Nonlinear Modeling Error (NME) matrix part of the Hessian, which has been neglected in previous analysis.", "essential_references_not_discussed": "-", "other_strengths_and_weaknesses": "I think the observation of the power-law distribution of the eigenvalues that the authors report is quite intriguing and it's exciting to see it occur in such various settings. It is also interesting to see how the slope magnitude correlates with other sharpness measures and $d_{KS}$ correlates well with generalization in LLMs. \n\nI believe that this is very interesting work and am open to raising my score if the authors address my comments and questions.", "other_comments_or_suggestions": "- some minor typos: line 258: considered, line 389: maximum, line 429: considered\n\n- it was not very clear to me what the three phases of large-batch training were until I read the section in the appendix. I would recommend the authors either to refer to this section in the main paper or to move this section from the appendix to the main paper to avoid confusion.", "questions_for_authors": "- What is meant by the so-called equilibrium of DNNs, which is mentioned in line 124? Is it equivalent to a DNN at convergence?\n\n- I am not entirely sure, but the eigenvalue distribution of sharp minima (e.g. large-batch training with B=768) intuitively seems to obey the power-law distribution less than a randomly initialized network, which to me implies a larger $d_{KS}$. However, I would assume that the network trained with $B=768$ will still perform better than random. (This is also why I asked for additional plots.) Can the authors explain this? Or for which cases $d_{KS}$ is a good predictor of generalization? \n\n- I checked the work by Wu et al. [2017] and noticed therein in Figure 2 (right) that models with quite different generalization behavior (21% vs. 92.9%) have a similar shape of the spectrum (at least by eyeballing), but that the spectrum is simply shifted. Can the authors elaborate how this result fits into the analysis that the authors have conducted? (e.g. Figures 3 and 4 in this work?)", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper reports a previously unnoticed power-law structure in the Hessian top eigenvalues of trained neural networks that is not present in randomly initialized nets.
It provides a theoretical motivation for this structure in terms of the maximum entropy principle.
The main contribution is an empirical study of the emergence and properties of the power law relative to relative to overparameterization and generalization on CNNs and LLMs.

This paper is a borderline case:
- The reviewers found the theoretical interpretation provided by the paper interesting but rather vague, and the authors could not substantially strengthen the intuition in the rebuttal.
- They found the empirical results presented by this work, specifically using the power law's exponent to predict generalization, to be intriguing and promising, and the experimental design to be solid.
  Some reviewers found the presentation of results non-cohesive due to the sheer amount of experiments presented in the main text.
  After reading the paper, I agree with this point.

My overall impression is that the paper contains findings to consider acceptance, but that their presentation and the story line could be improved significantly to make the main findings clearer.
Decision: Accept (poster)