SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zeke Xie $^{1}$ , Qian-Yuan Tang $^{2}$ , Mingming Sun $^{1}$  and Ping Li $^{1}$
$^{1}$ Cognitive Computing Lab, Baidu Research Department of Physics, Hong Kong Baptist University Correspondence: xiezeké@baidu.com,tangqy@hkbu.edu.hk
Stochastic gradients closely relate to both optimization and generalization of deep neural networks (DNNs). Some works attempted to explain the success of stochastic optimization for deep learning by the arguably heavy-tail properties of gradient noise, while other works presented theoretical and empirical evidence against the heavy-tail hypothesis on gradient noise. Unfortunately, formal statistical tests for analyzing the structure and heavy tails of stochastic gradients in deep learning are still under-explored. In this paper, we mainly make two contributions. First, we conduct formal statistical tests on the distribution of stochastic gradients and gradient noise across both parameters and iterations. Our statistical tests reveal that dimension-wise gradients usually exhibit power-law heavy tails, while iteration-wise gradients and stochastic gradient noise caused by minibatch training usually do not exhibit power-law heavy tails. Second, we further discover that the covariance spectra of stochastic gradients have the power-law structures overlooked by previous studies and present its theoretical implications for training of DNNs. While previous studies believed that the anisotropic structure of stochastic gradients matters to deep learning, they did not expect the gradient covariance can have such an elegant mathematical structure. Our work challenges the existing belief and provides novel insights on the structure of stochastic gradients in deep learning.
Stochastic optimization methods, such as Stochastic Gradient Descent (SGD), have been highly successful and even necessary in the training of deep neural networks (LeCun et al., 2015). It is widely believed that stochastic gradients as well as stochastic gradient noise (SGN) significantly improve both optimization and generalization of deep neural networks (DNNs) (Hochreiter and Schmidhuber, 1995, 1997; Hardt et al., 2016; Wu et al., 2021; Smith et al., 2020; Wu et al., 2020; Xie et al., 2021a; Sekhari et al., 2021; Amir et al., 2021). SGN, defined as the difference between full-batch gradient and stochastic gradient, has attracted much attention in recent years. People studied its type (Simsekli et al., 2019; Panigrahi et al., 2019; Hodgkinson and Mahoney, 2021; Li et al., 2021a), its magnitude (Mandt et al., 2017; Liu et al., 2021), its structure (Daneshmand et al., 2018; Zhu et al., 2019; Chmiel et al., 2020; Xie et al., 2021b; Wen et al., 2020), and its manipulation (Xie et al., 2021c). Among them, the noise type and the noise covariance structure are two core topics.
Topic 1. The arguments on the type and the heavy-tailed property of SGN. Recently, a line of research (Simsekli et al., 2019; Panigrahi et al., 2019; Gurbuzbalaban et al., 2021; Hodgkinson and Mahoney, 2021) argued that SGN has the heavy-tail property due to Generalized Central Limit Theorem (Gnedenko et al., 1954). Simsekli et al. (2019) presented statistical evidence showing that SGN looks closer to an  $\alpha$ -stable distribution that has power-law heavy tails rather than a Gaussian distribution. (Panigrahi et al., 2019) also presented the Gaussianity tests. However, their statistical tests were actually not applied to the true SGN that is caused by minibatch sampling. Because, in this line of research, the abused notation "SGN" is studied as stochastic gradient at some iteration
rather than the difference between full-batch gradient and stochastic gradient. Another line of research (Xie et al., 2021b, 2022b; Li et al., 2021a) pointed out this issue and suggested that the arguments in Simsekli et al. (2019) rely on a hidden strict assumption that SGN must be isotropic and does not hold for parameter-dependent and anisotropic Gaussian noise. This is why one tail-index for all parameters was studied in Simsekli et al. (2019). In contrast, SGN could be well approximated as an multi-variant Gaussian distribution in experiments at least when batch size is not too small, such as  $B \geq 128$  (Xie et al., 2021b; Panigrahi et al., 2019). Another work (Li et al., 2021a) further provided theoretical evidence for supporting the anisotropic Gaussian approximation of SGN. Nevertheless, none of these works conducted statistical tests on the Gaussianity or heavy tails of the true SGN.
Contribution 1. To our knowledge, we are the first to conduct formal statistical tests on the distribution of stochastic gradients/SGN across parameters and iterations. Our statistical tests reveal that dimension-wise gradients (due to anisotropy) exhibit power-law heavy tails, while iteration-wise gradient noise (which is the true SGN due to minibatch sampling) often has Gaussian-like light tails. Our statistical tests and notations help reconcile recent conflicting arguments on Topic 1.
Topic 2. The covariance structure of stochastic gradients/SGN. A number of works (Zhu et al., 2019; Xie et al., 2021b; HaoChen et al., 2021; Liu et al., 2021; Ziyin et al., 2022) demonstrated that the anisotropic structure and sharpness-dependent magnitude of SGN can help escape sharp minima efficiently. Moreover, some works theoretically demonstrated (Jastrzkebski et al., 2017; Zhu et al., 2019) and empirically verified (Xie et al., 2021b, 2022b; Daneshmand et al., 2018) that the covariance of SGN is approximately equivalent to the Hessian near minima. However, this approximation is only applied to minima and along flat directions corresponding to nearly-zero Hessian eigenvalues. The quantitative structure of stochastic gradients itself is still largely overlooked by previous studies.
Contribution 2. We surprisingly discover that the covariance of stochastic gradients has the power-law spectra in deep learning, which is overlooked by previous studies. While previous studies believed that the anisotropic structure of stochastic gradients matters to deep learning, they did not expect the gradient covariance can have such an elegant power-law structure. The power-law gradient covariance may help understand the success of stochastic optimization for deep learning.
Notations. Suppose a neural network  $f_{\theta}$  has  $n$  model parameters as  $\theta$ . We denote the training dataset as  $\{(x,y)\} = \{(x_j,y_j)\}_{j=1}^N$  drawn from the data distribution  $S$  and the loss function over one data sample  $\{(x_j,y_j)\}$  as  $l(\theta, (x_j,y_j))$ . We denote the training loss as  $L(\theta) = \frac{1}{N}\sum_{j=1}^{N}l(\theta, (x_j,y_j))$ .
We compute the gradients of the training loss with the batch size  $B$  and the learning rate  $\eta$  for  $T$  iterations. We let  $g^{(t)}$  represent the stochastic gradient at the  $t$ -th iteration. We denote the Gradient History Matrix as  $G = [g^{(1)}, g^{(2)}, \dots, g^{(T)}]$  an  $n \times T$  matrix where the column vector  $G_{.,t}$  represents the dimension-wise gradients  $g^{(t)}$  for  $n$  model parameters, the row vector  $G_{i,t}$  represents the iteration-wise gradients  $g_{i}$  for  $T$  iterations, and the element  $G_{i,t}$  is  $g_{i}^{(t)}$  at the  $t$ -th iteration for the parameter  $\theta_{i}$ . We analyze  $G$  for a given model without updating the model parameter  $\theta$ . The Gradient History Matrix  $G$  plays a key role in reconciling the conflicting arguments on Topic 1. Because the defined dimension-wise SGN (due to anisotropy) is the abused "SGN" in one line of research (Simsekli et al., 2019; Panigrahi et al., 2019; Gurbuzbalaban et al., 2021; Hodgkinson and Mahoney, 2021), while iteration-wise SGN (due to minibatch sampling) is the true SGN as another line of research (Xie et al., 2021b, 2022b; Li et al., 2021a) suggested. Our notation mitigates the abused "SGN".
We further denote the second moment as  $C_m = \mathbb{E}[gg^\top]$  for stochastic gradients and the covariance as  $C = \mathbb{E}[(g - \bar{g})(g - \bar{g})^\top]$  for SGN, where  $\bar{g} = \mathbb{E}[g]$  is the full-batch gradient. We denote the descending ordered eigenvalues of a matrix, such as the Hessian  $H$  and the covariance  $C$ , as  $\{\lambda_1, \lambda_2, \dots, \lambda_n\}$  and denote the corresponding spectral density function as  $p(\lambda)$ .
Goodness-of-Fit Test. In statistics, various Goodness-of-Fit Tests have been proposed for measuring the goodness of empirical data fitting to some distribution. In this subsection, we introduce how to conduct the Kolmogorov-Smirnov (KS) Test (Massey Jr, 1951; Goldstein et al., 2004) for measuring the goodness of fitting a power-law distribution and the Pearson's  $\chi^2$  Test (Plackett, 1983) for measuring the goodness of fitting a Gaussian distribution. We present more details in Appendix B.
Table 1: The KS and  $\chi^2$  statistics and the hypothesis acceptance rates of the gradients over dimensions and iterations, respectively. Model: LeNet. Batch Size: 100. In the second column "random" means randomly initialized models, while "pretrain" means pretrained models.
Dataset	Training	SG Type	dks	dc	Power-Law Rate	p-value	Gaussian Rate
MNIST	Random	Dimension	0.0355	0.0430	76.6%	4.11 × 10-4	0.18%
MNIST	Random	Iteration	0.191	0.0430	0.0788%	0.321	65.1%
MNIST	Pretrain	Dimension	0.0401	0.0430	60.6%	5.77 × 10-4	0.24%
MNIST	Pretrain	Iteration	0.179	0.0430	0.052%	0.306	62.0%
CIFAR-10	Random	Dimension	0.0330	0.0430	81.8%	1.03 × 10-3	0.3%
CIFAR-10	Random	Iteration	0.574	0.0430	0%	0.337	66.8%
CIFAR-10	Pretrain	Dimension	0.0381	0.0430	69.2%	4.01 × 10-5	0%
CIFAR-10	Pretrain	Iteration	0.654	0.0430	0%	0.275	56.6%
Figure 1: Magnitude of gradients w.r.t. magnitude rank. The dimension-wise gradients (one column vector of  $G$ ) have power-law heavy tails, while the iteration-wise gradients (one row vector of  $G$ ) have no power-law heavy tails. Model: (Randomly Initialized) LeNet. Dataset: MNIST and CIFAR-10.
Figure 2: The power-law rates and Gaussian rates w.r.t. batch size. Increasing batch size significantly improves the Gaussianity of SGN. Model: LeNet.
When we say a set of random variables (the elements or eigenvalues) is approximately power-law/Gaussian in this paper, we mean the tested set of data points can pass KS Tests for power-law distributions or  $\chi^2$  Test for Gaussian distributions at the Significance Level 0.05. We note that, in all statistical tests of this paper, we set the Significance Level as 0.05.
In KS Test, we state the power-law hypothesis that the tested set of elements is power-law. If the KS distance  $d_{ks}$  is larger than the critical distance  $d_c$ , the KS test will reject the power-law hypothesis. In contrast, if the KS distance  $d_{ks}$  is less than the critical distance  $d_c$ , the KS test will support (not reject) the power-law hypothesis. The smaller  $d_{ks}$  is, the better the goodness-of-power-law is.
In  $\chi^2$  Test, we state the Gaussian hypothesis that the tested set of elements is Gaussian. If the estimated  $p$ -value is larger than 0.05, the  $\chi^2$  test will reject the Gaussian hypothesis. If the estimated  $p$ -value is less than 0.05, the  $\chi^2$  test will support (not reject) the Gaussian hypothesis. The smaller  $p$ -value is, the better the goodness-of-Gaussianity is.
The Gaussianity test consists of Skewness Test and Kurtosis Test (Cain et al., 2017) (Pleas see Appendix B) for more details. Skewness is a measure of symmetry. A distribution or dataset is symmetric if the distribution on either side of the mean is roughly the mirror image of the other. Kurtosis is a measure of whether the data are heavy-tailed or light-tailed relative to a normal distribution. Empirical data with high (respectively, low) kurtosis tend to have heavy (respectively, light) tails. Thus,  $\chi^2$  Test can reflect both Gaussianity and heavy tails.
3 Rethink Heavy Tails in Stochastic Gradients
In this section, we try to reconcile the conflicting arguments on Topic 1 by formal statistical tests.
The power-law distribution. Suppose that we have the set of  $k$  random variables  $\{\lambda_1, \lambda_2, \dots, \lambda_k\}$  that obeys a power-law distribution. We may write the probability density function of a power-law distribution as
$$
p (\lambda) = Z ^ {- 1} \lambda^ {- \beta}, \tag {1}
$$
(a) Random FCN on MNIST
(b) Random ResNet18 on CIFAR-10
(c) Random ResNet18 on CIFAR-100
(d) Pretrained FCN on MNIST
(e) Pretrained ResNet18 on CIFAR-(f) Pretrained ResNet18 on CIFAR-10 100
Figure 3: We plot the magnitude of gradients with respect to the magnitude rank for FCN and ResNet18 on MNIST, CIFAR-10, and CIFAR-100. The dimension-wise gradients have power-law heavy tails, while the iteration-wise gradients have no power-law heavy tails.
where  $Z$  is the normalization factor. The finite-sample power law, also known as Zipf's law, can also be approximately written as
$$
\lambda_ {k} = \lambda_ {1} k ^ {- s}, \tag {2}
$$
if we let  $s = \frac{1}{\beta - 1}$  denote the power exponent of Zipf's law (Visser, 2013). A well-known property of power laws is that, when the power-law variables and their corresponding rank orders are scattered in log-log scaled plot, a straight line may fit the points well (Clauset et al., 2009).
Dimension-wise gradients are usually power-law, while iteration-wise gradients are usually Gaussian. In Figure 1, we plot dimension-wise gradients and iteration-wise gradients of LeNet on MNIST and CIFAR over 5000 iterations with fixing the model parameters. We leave experimental details in Appendix A and the extensive statistical test results in Appendix C. In Figure 1, while some points slightly deviate from the fitted straight line, we may easily observe the straight lines approximately fit the red points (dimension-wise gradients) but fail to fit the blue points (iteration-wise gradients). The observations indicate that dimension-wise gradients have power-law heavy tails while iteration-wise gradients have no such heavy tails.
Table 1 shows the mean KS distance and the mean  $p$ -value over dimensions and iterations as well as the power-law rates and the Gaussian rates. We note that the power-law/Gaussian rate means the percentage of the tested points that are not rejected for the power-law/Gaussian hypothesis via  $\mathrm{KS} / \chi^2$  tests. Dimension-wise gradients and iteration-wise gradients show significantly different preferences for the power-law rate and the Gaussian rate. For example, a LeNet on CIFAR-10 has 62006 model parameters. Dimension-wise gradients of the model are power-law for  $81.8\%$  iterations and are Gaussian for only  $0.3\%$  iterations. In contrast, iteration-wise gradients of the model are Gaussian for  $66.8\%$  dimensions (parameters) and are power-law for no dimension.
The observation and the statistical test results of Table 1 both indicate that dimension-wise gradients usually have power-law heavy tails while iteration-wise gradients are usually approximately Gaussian (with light tails) for most dimensions. The conclusion holds for both pretrained models and random models on various datasets. Similarly, we also observe power-law dimension-wise gradients and non-power-law iteration-wise gradients for FCN and ResNet18 in Figure 3, as well as Table 5 in Appendix C.
According to Central Limit Theorem, the Gaussianity of iteration-wise gradients should depend on the batch size. We empirically studied how the Gaussian rate of iteration-wise gradients depends on the batch size. The results in Figure 2 and Table 4 (see Appendix) support that the Gaussianity of iteration-wise gradients indeed positively correlates to the batch size, which is consistent with the Central Limit Theorem. In the common setting that  $B \geq 30$ , the Gaussianity of SGN can be statistically more significant than heavy tails for most parameters of DNNs, according to  $\chi^2$  Tests.
(b) CIFAR-10
Figure 4: The gradient spectra are highly similar and exhibit power laws for both random models and pretrained models. Model: LeNet and 2-Layer FCN. Dataset: MNIST and CIFAR-10.
Reconciling the conflicting arguments on Topic 1. We argue that the power-law tails of dimension-wise gradients and the Gaussianity of iteration-wise gradients may well explain the conflicting arguments on Topic 1. On the one hand, the evidences proposed by the first line of research are mainly for describing the elements of one column vector of  $G$  which represent the dimension-wise gradient at a given iteration. Thus, the works in the first line of research can only support that the distribution of (dimension-wise) stochastic gradients has a power-law heavy tail, where heavy tails are mainly caused by the gradient covariance (See Section 4) instead of minibatch training.
On the other hand, the works in the second line of research pointed out that the type of SGN is actually decided by the distribution of (iteration-wise) stochastic gradients due to minibatch sampling, which is usually Gaussian for a common batch size  $B \geq 30$ . Researchers care more about the true SGN, the difference between full-batch gradients and stochastic gradients, mainly because SGN essentially matters to implicit regularization of SGD and deep learning dynamics (Jastrzkebski et al., 2017; Zhu et al., 2019; Xie et al., 2021b; Li et al., 2021b). While previous works in the second line of research did not conduct statistical tests, our work fills the gap.
In summary, while it seems that the two lines of research have conflicting arguments on Topic 1, their evidences are actually not contradicted. We may easily reconcile the conflicts as long as the first line of research clarifies that the heavy-tail property describes dimension-wise gradients (not SGN), which corresponds to the column vector of  $G$  instead of the row vector of  $G$ .
We notice that the Gaussian rates (the rates of not rejecting the Gaussian hypothesis) do not approach to a very high level (e.g.  $95\%$ ) even under relatively large batch sizes (e.g.,  $B = 1000$ ), while they have nearly zero power-law rates (the rates of not rejecting the power-law hypothesis). While the Gaussian rate is low under small batch sizes, the power-law rate is still nearly zero. This may indicate that SGN of a small number of model parameters or under small batch sizes may have novel properties beyond the Gaussianity and the power-law heavy tails that previous works expected.
4 The Overlooked Power-Law Structure
In this section, we study the covariance/second-moment structure of stochastic gradients. Despite the reconciled conflicts on Topic 1, another question arises that why dimension-wise stochastic gradients may exhibit power laws in deep learning. We show that the covariance not only explains why power-law gradients arise but also surprisingly challenges conventional knowledge on the relation between the covariance (of SGN) and the Hessian (of the training loss).
The power-law covariance spectrum. We display the covariance spectra for various models on MNIST and CIFAR-10. Figure 4 shows the covariance spectra for pretrained models and random models are both power-law despite several slightly deviated top eigenvalues. The KS test results are shown in Table 2. To our knowledge, we are the first to discover that the covariance spectra are usually power-law for deep learning with formal empirical & statistical evidences.
Table 2: KS statistics of the covariance spectra of LeNet and FCN.
Dataset	Model	Training	dsk	dc	Power-Law	s
MNIST	LeNet	Pretrain	0.0206	0.043	Yes	1.302
MNIST	LeNet	Random	0.0220	0.043	Yes	1.428
CIFAR-10	LeNet	Pretrain	0.0201	0.043	Yes	1.257
CIFAR-10	LeNet	Random	0.0214	0.043	Yes	1.300
MNIST	FCN	Pretrain	0.0415	0.043	Yes	0.866
MNIST	FCN	Random	0.0418	0.043	Yes	0.864
The relation between gradient covariances and Hessians. Both SGN and Hessian essentially matter to optimization and generalization of deep learning (Li et al., 2020; Ghorbani et al., 2019; Zhao et al., 2019; Jacot et al., 2019; Yao et al., 2018; Dauphin et al., 2014; Byrd et al., 2011). A conventional belief is that the covariance is approximately proportional to the Hessian near minima, namely  $C(\theta) \propto H(\theta)$  (Jastrzkebski et al., 2017; Zhu et al., 2019; Xie et al., 2021b, 2022b; Daneshmand et al., 2018; Liu et al., 2021). Near a critical point, we have
$$
C (\theta) \approx \frac {1}{B} \left[ \frac {1}{N} \sum_ {j = 1} ^ {N} \nabla l \left(\theta , \left(x _ {j}, y _ {j}\right)\right) \nabla l \left(\theta , \left(x _ {j}, y _ {j}\right)\right) ^ {\top} \right] \frac {1}{B} \operatorname {F I M} (\theta) \approx \frac {1}{B} [ H (\theta) ], \tag {3}
$$
where  $\mathrm{FIM}(\theta)$  is the observed Fisher Information matrix, referring to Chapter 8 of Pawitan (2001) and Zhu et al. (2019). The first approximation holds when the expected gradient is small near minima, and the second approximation hold because FIM is approximately equal to the Hessian near minima.
Some works (Xie et al., 2021b, 2022b) empirically verified Eq. (3) and further argued that Eq. (3) approximately holds even for random models (which are far from minima) in terms of the flat directions corresponding to small eigenvalues of the Hessian. Note that the most eigenvalues of the Hessian are nearly zero. The gradients along these flat directions are nearly zero as the approximation in Eq. (3) is particularly mild along these directions.
The common PCA method, as well as the related low-rank matrix approximation, actually prefers to remove or ignore the components corresponding to small eigenvalues. Because the top eigenvalues and their corresponding eigenvectors can reflect the main properties of a matrix. Unfortunately, previous works (Xie et al., 2021b, 2022b) only empirically studied the small eigenvalues of the covariance and the Hessian and missed the most important top eigenvalues. The missing evidence for verifying the top eigenvalues of the covariance and the Hessian can be a serious flaw for the well-known approximation Eq. (3). A number of works (Sagun et al., 2016, 2017; Wu et al., 2017; Pennington and Bahri, 2017; Pennington and Worah, 2018; Papyan, 2018, 2019; Jacot et al., 2019; Fort and Scherlis, 2019; Singh et al., 2021; Liao and Mahoney, 2021; Xie et al., 2022a) tried to analyze the Hessian structure of deep loss landscape. However, they did not formally touch the structure of stochastic gradients. In this paper, we particularly compute the top thousands of eigenvalues of the Hessian and compare them to the corresponding top eigenvalues of the covariance.
In Figure 5, we discover that, surprisingly, the top eigenvalues of the covariance can significantly deviate from the corresponding eigenvalues of the Hessian sometimes by more than one order of magnitude near or far from minima. This challenges the conventional belief on the proportional relation between the covariance and the Hessian near minima.
We also note that the covariance and the second-moment matrix have highly similar spectra in the log-scale plots. For simplicity of expressions, when we say the spectra of gradient noise/gradients in the following analysis, we mean the spectra of the covariance/the second moment, respectively.
For pretrained models, especially pretrained FCN, while the magnitudes of the Hessian and the corresponding covariance are not even close, the straight lines fit the Hessian spectra and the covariance spectra well. Moreover, the fitted straight lines have similar slopes. Our results also support a very recent finding (Xie et al., 2022a) that the Hessians have power-law spectra for well-trained DNNs but significantly deviate from power laws for random DNNs.
For random models, while the Hessian spectra are not power-law, the covariance spectra surprisingly still exhibit power-law distributions. This is beyond the existing work expected. It is not surprising that the Hessian and the covariance have no close relation without pretraining. However, we report
(a) Pretrained LeNet
(b) Random LeNet
(c) Pretrained FCN
(d) Random FCN
Figure 5: The spectra of gradients (Uncentered Covariance), gradient noise (Centered Covariance), and Hessians for random models and pretrained models. Model: LeNet and FCN. Dataset: MNIST.
that the power-law covariance spectrum seems to be a universal property, and it is more general than the power-law Hessian spectrum for DNNs.