SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Robust and low-dimensional learning subspace. May power-law covariance spectra theoretically imply any novel insight? The answer is affirmative. A number of papers (Gur-Ari et al., 2018; Ghorbani et al., 2019; Xie et al., 2021b) reported that deep learning (via SGD) mainly happens in a low-dimensional space spanned by the eigenvectors corresponding to large eigenvalues during the whole training process. Note that the low-dimensional learning space implicitly reduces deep models' complexity. These studies indicated that the intrinsic dimension of learning space is much lower than the original model dimensionality, resulting in improved generalization and low complexity of deep models. However, existing work cannot explain why the learning space is low-dimensional and robust during training. In this paper, robust space means that the space's dimensions are stable during training. We argue that the power-law structure can mathematically explain why low-dimensional and robust learning space exists.
We try to mathematically answer this question by studying the gradient covariance eigengaps. We define the  $k$ -th eigengap as  $\delta_k = \lambda_k - \lambda_{k+1}$ . According to Eq. (2), we have  $\delta_k$  approximately meeting
$$
\delta_ {k} = \operatorname {T r} (C) Z _ {d} ^ {- 1} \left(k ^ {- \frac {1}{\beta - 1}} - (k + 1) ^ {- \frac {1}{\beta - 1}}\right) = \lambda_ {k} \left[ 1 - \left(\frac {k}{k + 1}\right) ^ {s} \right], \tag {4}
$$
where  $Z_{d} = \sum_{k=1}^{n} k^{-\frac{1}{\beta - 1}}$  is the normalization factor. Interestingly, it demonstrates that eigengaps also approximately exhibit a power-law distribution. We present the empirical evidences of the power-law eigengaps in Figure 16 and Table 3 (of Appendix C). We observe that the eigengaps decay faster than the eigenvalue. This is not surprising. For example, under the approximation  $s \approx 1$ , we have an approximate power-law decaying
$$
\delta_ {k} = \operatorname {T r} (C) Z _ {d} ^ {- 1} (k + 1) ^ {- (s + 1)} \tag {5}
$$
with a larger power exponent as  $s + 1$ .
Based on the Davis-Kahan  $\sin (\Theta)$  Theorem (Davis and Kahan, 1970), we use the angle of the original eigenvector  $u_{k}$  and the perturbed eigenvector  $\tilde{u}_k$ , namely  $\langle u_k,\tilde{u}_k\rangle$ , to measure the robustness of
space's dimensions. We apply Theorem 1, a useful variant of Davis-Kahan Theorem (Yu et al., 2015), to the gradient covariance in deep learning, which states that the eigenspace (spanned by eigenvectors) robustness can be well bounded by the corresponding eigengap.
Theorem 1 (Eigengaps Bound Eigenspace Robustness (Yu et al., 2015)). Suppose the true gradient covariance is  $C$ , the perturbed gradient covariance is  $\tilde{C} = C + \epsilon M$ , the  $i$ -th eigenvector of  $C$  is  $u_{i}$ , and its corresponding perturbed eigenvector is  $\tilde{u}_{i}$ . Under the conditions of the Davis-Kahan Theorem, we have
$$
\sin \langle u _ {k}, \tilde {u} _ {k} \rangle \leq \frac {2 \epsilon \| M \| _ {o p}}{\min \left(\lambda_ {k - 1} - \lambda_ {k} , \lambda_ {k} - \lambda_ {k + 1}\right)},
$$
where  $\| M\|_{op}$  is the operator norm of  $M$ .
As we have a small number of large eigengaps corresponding to the large eigenvalues, the corresponding learning space robustness has a tight upper bound. For example, given the power-law eigengaps in Eq. (5), the upper bound of eigenvector robustness can be approximately written as
$$
\sup  \sin \langle u _ {k}, \tilde {u} _ {k} \rangle \approx \frac {2 \epsilon \| M \| _ {o p} (k + 1) ^ {s + 1}}{\lambda_ {1}}, \tag {6}
$$
when  $s$  is close to one. Obviously, the bound is relatively tight for top dimensions (small  $k$ ) but becomes very loose for tailed dimensions (large  $k$ ). A similar conclusion also holds given Eq. (4) without  $s \approx 1$ . This indicates that non-top eigenspace can be highly unstable during training, because  $\delta_{k}$  can decay to nearly zero for a large  $k$ . To the best of our knowledge, we are the first to demonstrate that the robustness of low-dimensional learning space directly depends on the eigengaps.
Note that the existence of top large eigenvalues does not necessarily indicate their gaps are also statistically large. Previous papers failed to reveal that top eigengaps also dominate tailed eigengaps in deep learning. Fortunately, we numerically demonstrate that, as rank order increases, both eigenvalues and eigengaps decay. Eigengaps even decay faster than eigenvalues due to the larger magnitude of the power exponent. Our analysis partially explains the foundation of learning space robustness in deep learning.
5 Empirical Analysis and Discussion
In this section, we empirically studied the covariance spectrum for DNNs in extensive experiments. We particularly reveal that when the power-law covariance for DNNs appears or disappears. We leave experimental details in Appendix A
1. Batch Size. Figure 6 shows that the power-law covariance exists in deep learning for various batch sizes. Moreover, the top eigenvalues are indeed approximately inverse to the batch size as Eq. (3) suggests, while the proportional relation between the Hessian and the covariance is weak.
Figure 6: Power-law covariance is approximately inverse to the batch size in deep learning. Model: LeNet. Dataset: MNIST.
Figure 7: Comparison of the gradient spectra on MNIST with clean labels, noisy labels, and random labels. Model: LeNet.
2. Learning with Noisy Labels and Random Labels. Recently, people usually regarded learning with noisy labels (Han et al., 2020) as an important setting for exploring the overfitting and generalization of DNNs. Previous papers (Martin and Mahoney, 2017; Han et al., 2018) demonstrated that DNNs may easily overfit noisy labels and even have completely random labels during training, while
Figure 8: Large enough width (e.g., Width  $\geq 70$ ) matters to the goodness of power-law covariance, while the depth does not. Left: FCN with various depths. Right: FCN with various widths.
convergence speed is slower compared with learning with clean labels. Is this caused by the structure of stochastic gradients? It seems no. We compared the covariance spectrum under clean labels,  $40\%$  noisy labels,  $80\%$  noisy labels, and completely random labels in Figure 7. We surprisingly discovered that memorization of noisy labels matters little to the power-law structure of stochastic gradients.
3. Depth and Width. In this paper, we also study how the depth and the width of neural networks affect the power-law covariance. Figure 8 and the KS tests in Table 8 (Appendix C) support that certain width (e.g.,  $\mathrm{Width}\geq 70$ ) is often required for supporting the power-law hypothesis, while the depth seems unnecessary. Even one-layer FCN may still exhibit the power-law covariance.
(a) LNN
(b) LNN+BatchNorm
Figure 9: The power-law gradients appear in LNNs with BatchNorm or ReLU, but disappear in fully LNNs. Dataset: MNIST.
(c) LNN+ReLU
4. Linear Neural Networks (LNNs) and Nonlinearity. What is the simplest model that shows the power-law covariance? We study covariance spectra for fully LNNs, LNNs with BatchNorm, and LNNs with ReLU (FCN w/o BatchNorm) in Figure 9. Obviously, fully LNNs may not learn minima with power-law Hessians. Layer-wise nonlinearity seems necessary for the power-law Hessian spectra (Xie et al., 2022a). However, even the simplest two-layer LNN with no nonlinearity still exhibits power-law covariance.
5. The Outliers, BatchNorm, and Data Classes. We also report that there sometimes exist a few top covariance eigenvalues that significantly deviate from power laws or the fitted straight lines. Figure 11 shows that, the outliers are especially significant for LeNet, a Convolution Neural Network, but less significant for FCN. We also note that LeNet does not apply BatchNorm, while the used FCN applies BatchNorm. What is the real factor that determines whether top outliers are significant or not? Figures 9 and 10 support that it is BatchNorm that makes top outliers less significant rather than the convolution layers. Because even the simplest LNNs, which have no convolution layers and nonlinear activations, still exhibit significant top outliers. This may explain why BatchNorm helps training of DNNs.
Suppose there are  $c$  classes in the dataset, where  $c = 10$  for CIFAR-10 and MNIST. We observe that the number of outliers is usually  $c - 1$  in Figures 11 and 12. It supports that the gradients of DNNs indeed usually concentrate in a tiny top space as previous work suggested (Gur-Ari et al., 2018), because the ninth eigenvalue may be larger than the tenth eigenvalue by one order of magnitude. However, this conclusion may not hold similarly well without BatchNorm.
Figure 10: BatchNorm can make the outliers less significant. Model: FCN with/without BatchNorm.
Figure 13: Momentum and Weight Decay does not significantly affect the power-law covariance, while Adam does.
Figure 11: The number of outliers is usually  $c - 1$ . The outliers of the FCN gradient spectrum is much less significant than that of LeNet. Dataset: MNIST.
Figure 14: The power-law covariance holds with Gradient Clipping. Model: LeNet. Dataset: CIFAR-10.
Figure 15: The power-law gradients appear on non-image natural datasets. Dataset: Avila. Model: FCN.
Figure 12: The outliers in the power-law spectrum mainly depends on the the number of data classes rather than the number of model outputs (logits).
Is it possible that the number of outliers depends on the number of model outputs (logits) rather than the number of data classes? In Figure 12, we eliminate the possibility by training a LeNet with 100 logits on CIFAR-10, denoted by CIFAR-  $10^{\star}$ . The number of outliers will be constant even if we increase the model logits.
6. Optimization Techniques. Previous papers report that Weight Decay, Momentum, and Adam may significantly affect gradient noise (Sutskever et al., 2013; Daneshmand et al., 2018; Xie et al., 2023). In Figure 13, we discover that Weight Decay and Momentum do not affect the power-law structure, while Adam obviously breaks the power-law structure due to adaptive gradients. Gradient Clipping is a popular method for stabilizing and accelerating the training of language models (Zhang et al., 2019). Figure 14 shows Gradient Clipping does not break the power-law structure.
7. Non-image Data. Natural images have special statistical properties (Torralba and Oliva, 2003). May the power-law covariance be caused by the statistical properties of natural images? In Figure 15 on a non-image UCI dataset, Avila, which is a simple classification dataset with only ten input features, the power-law gradients of DNNs are more general than natural image statistics.
In this paper, we revisit two essentially important topics about stochastic gradients in deep learning. First, we reconciled recent conflicting arguments on the heavy-tail properties of SGN. We demonstrated that dimension-wise gradients usually have power-law heavy tails, while iteration-wise gradients or SGN have relatively high Gaussianity. Second, to our knowledge, we are the first to report that the covariance of gradients usually has a surprising power-law structure in even simple neural networks. The heavy tails of dimension-wise gradients could be explained as a natural result of the power-law covariance. We further analyze the theoretical implications and how various settings affect the power-law gradient structure in deep learning. The main limitations of our work lie in lacking (1) theoretically explaining why the power-law covariance generally exists in deep learning and (2) scaling the experiments to larger Transformer-based models. We leave the theoretical explanation and the results of large Transformer-based models as future work.
We thank Ms. Zheng He for reproducing core experimental results. This work is supported by the Early Career Scheme (ECS) from the Research Grants Council (RGC) of Hong Kong (No. 22302723).
Alstott, J., Bullmore, E., and Plenz, D. (2014). powerlaw: a python package for analysis of heavy-tailed distributions. PloS one, 9(1):e85777.
Amir, I., Koren, T., and Livni, R. (2021). Sgd generalizes better than gd (and regularization doesn't help). In Conference on Learning Theory, pages 63-92. PMLR.
Byrd, R. H., Chin, G. M., Neveitt, W., and Nocedal, J. (2011). On the use of stochastic hessian information in optimization methods for machine learning. SIAM Journal on Optimization, 21(3):977-995.
Cain, M. K., Zhang, Z., and Yuan, K.-H. (2017). Univariate and multivariate skewness and kurtosis for measuring nonnormality: Prevalence, influence and estimation. Behavior research methods, 49(5):1716-1735.
Chmiel, B., Ben-Uri, L., Shkolnik, M., Hoffer, E., Banner, R., and Soudry, D. (2020). Neural gradients are near-lognormal: improved quantized and sparse training. In International Conference on Learning Representations.
Clauset, A., Shalizi, C. R., and Newman, M. E. (2009). Power-law distributions in empirical data. SIAM review, 51(4):661-703.
Daneshmand, H., Kohler, J., Lucchi, A., and Hofmann, T. (2018). Escaping saddles with stochastic gradients. In International Conference on Machine Learning, pages 1155-1164.
Dauphin, Y. N., Pascanu, R., Gulcehre, C., Cho, K., Ganguli, S., and Bengio, Y. (2014). Identifying and attacking the saddle point problem in high-dimensional non-convex optimization. Advances in Neural Information Processing Systems, 27:2933-2941.
Davis, C. and Kahan, W. M. (1970). The rotation of eigenvectors by a perturbation. iii. SIAM Journal on Numerical Analysis, 7(1):1-46.
De Stefano, C., Maniaci, M., Fontanella, F., and di Freca, A. S. (2018). Reliable writer identification in medieval manuscripts through page layout features: The "avila" bible case. Engineering Applications of Artificial Intelligence, 72:99-110.
Fort, S. and Scherlis, A. (2019). The goldilocks zone: Towards better understanding of neural network loss landscapes. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 3574-3581.
Ghorbani, B., Krishnan, S., and Xiao, Y. (2019). An investigation into neural net optimization via hessian eigenvalue density. In International Conference on Machine Learning, pages 2232-2241. PMLR.
Gnedenko, B., Kolmogorov, A., Gnedenko, B., and Kolmogorov, A. (1954). Limit distributions for sums of independent. Am. J. Math, 105.
Goldstein, M. L., Morris, S. A., and Yen, G. G. (2004). Problems with fitting to the power-law distribution. The European Physical Journal B-Condensed Matter and Complex Systems, 41(2):255-258.
Gur-Ari, G., Roberts, D. A., and Dyer, E. (2018). Gradient descent happens in a tiny subspace. arXiv preprint arXiv:1812.04754.
Gurbuzbalaban, M., Simsekli, U., and Zhu, L. (2021). The heavy-tail phenomenon in sgd. In International Conference on Machine Learning, pages 3964-3975. PMLR.
Computational environment. The experiments are conducted on a computing cluster with NVIDIA® V100 GPUs and Intel® Xeon® CPUs. We can produce/reproduce the main experiments using PaddlePaddle (Ma et al., 2019) and PyTorch (Paszke et al., 2019).
Model: LeNet (LeCun et al., 1998), Fully Connected Networks (FCN), and ResNet18 (He et al., 2016). Dataset: MNIST (LeCun, 1998), CIFAR-10/100 (Krizhevsky and Hinton, 2009), and Avila (De Stefano et al., 2018). Avila is a non-image dataset.
A.1 Gradient History Matrices
In this paper, we compute the Gradient History Matrices and the covariance for multiple models on multiple datasets. Then, we use the elements in Gradient History Matrices and the eigenvalues of the covariance/second-moment to evaluate the goodness of fitting Gaussian distributions or power-law distributions via  $\chi^2$  tests and KS tests.
The Gradient History Matrix is an  $n \times T$  matrix. For the experiment of LeNet and FCN, we compute the gradients for  $T = 5000$  iterations at a fixed randomly initialized position  $\theta^{(0)}$  or a pretrained position  $\theta^{\star}$ . Due to limit of memory capacity, for the experiment of ResNet18, we compute the gradients for  $T = 200$  iterations at  $\theta^{(0)}$  or  $\theta^{\star}$ .
A Gradient History Matrix can be used to compute the covariance or the second moment of stochastic gradients for a neural network. Note that a covariance matrix is an  $n \times n$  matrix, which is extremely large for modern neural networks. Thus, we mainly analyze the gradient structures of LeNet and FCN at an affordable computational cost.
A.2 Models and Datasets
Models: LeNet (LeCun et al., 1998), Fully Connected Networks (FCN), and ResNet18 (He et al., 2016). We mainly used two-layer FCN which has 70 neurons for each hidden layer, ReLu activations, and BatchNorm layers, unless we specify otherwise.
Datasets: MNIST (LeCun, 1998) and CIFAR-10/100 (Krizhevsky and Hinton, 2009), and non-image Avila (De Stefano et al., 2018).
Optimizers: SGD, SGD with Momentum, and Adam (Kingma and Ba, 2015).
A.3 Image classification on MNIST
We perform the common per-pixel zero-mean unit-variance normalization as data preprocessing for MNIST.
Pretraining Hyperparameter Settings: We train neural networks for 50 epochs on MNIST for obtaining pretrained models. For the learning rate schedule, the learning rate is divided by 10 at the epoch of  $40\%$  and  $80\%$ . We use  $\eta = 0.1$  for SGD/Momentum and  $\eta = 0.001$  for Adam. The batch size is set to 128. The strength of weight decay defaults to  $\lambda = 0.0005$  for pretrained models. We set the momentum hyperparameter  $\beta_{1} = 0.9$  for SGD Momentum. As for other optimizer hyperparameters, we apply the default settings directly.
Hyperparameter Settings for  $G$ : We use  $\eta = 0.1$  for SGD/Momentum and  $\eta = 0.001$  for Adam. The batch size is set to 1 and no weight decay is used, unless we specify them otherwise.
A.4 Image classification on CIFAR-10 and CIFAR-100
Pretraining Hyperparameter Settings: In the experiments on CIFAR-10 and CIFAR-100:  $\eta = 1$  for Vanilla SGD;  $\eta = 0.1$  for SGD (with Momentum);  $\eta = 0.001$  for Adam. For the learning rate schedule, the learning rate is divided by 10 at the epoch of  $\{80, 160\}$  for CIFAR-10 and  $\{100, 150\}$  for CIFAR-100, respectively. The batch size is set to 128 for both CIFAR-10 and CIFAR-100. The
batch size is set to 128 for MNIST, unless we specify it otherwise. The strength of weight decay defaults to  $\lambda = 0.0005$  as the baseline for all optimizers unless we specify it otherwise. We set the momentum hyperparameter  $\beta_{1} = 0.9$  for SGD and adaptive gradient methods which involve in Momentum. As for other optimizer hyperparameters, we apply the default settings directly.
Hyperparameter Settings for  $G$ : We use  $\eta = 1$  for SGD,  $\eta = 0.1$  for SGD with Momentum, and  $\eta = 0.001$  for Adam. The batch size is set to 1 and no weight decay is used, unless we specify them otherwise.
A.5 Learning with noisy labels
We trained LeNet via SGD (with Momentum) on corrupted MNIST with various (asymmetric) label noise. We followed the setting of Han et al. (2018) for generating noisy labels for MNIST. The symmetric label noise is generated by flipping every label to other labels with uniform flip rates  $\{40\%, 80\% \}$ . In this paper, we used symmetric label noise.
For obtaining datasets with random labels which have little knowledge behind the pairs of instances and labels, we also randomly shuffle the labels of MNIST to produce Random MNIST.
B Goodness-of-Fit Tests
B.1 Kolmogorov-Smirnov Test
In this section, we introduce how to conduct the Kolmogorov-Smirnov Goodness-of-Fit Test.
We used Maximum Likelihood Estimation (MLE) (Myung, 2003; Clauset et al., 2009) for estimating the parameter  $\beta$  of the fitted power-law distributions and the Kolmogorov-Smirnov Test (KS Test) (Massey Jr, 1951; Goldstein et al., 2004) for statistically testing the goodness of fitting power-law distributions. The KS test statistic is the KS distance  $d_{\mathrm{ks}}$  between the hypothesized (fitted) distribution and the empirical data, which measures the goodness of fit. It is defined as
$$
d _ {\mathrm {k s}} = \sup  _ {\lambda} | F ^ {\star} (\lambda) - \hat {F} (\lambda) |, \tag {7}
$$
where  $F^{\star}(\lambda)$  is the hypothesized cumulative distribution function and  $\hat{F} (\lambda)$  is the empirical cumulative distribution function based on the sampled data (Goldstein et al., 2004). The estimated power exponent via MLE (Clauset et al., 2009) can be written as
$$
\hat {\beta} = 1 + K \left[ \sum_ {i = 1} ^ {K} \ln \left(\frac {\lambda_ {i}}{\lambda_ {\operatorname* {m i n}}}\right) \right] ^ {- 1}, \tag {8}
$$
where  $K$  is the number of tested samples and we set  $\lambda_{\mathrm{min}} = \lambda_k$ . In this paper, we choose the top  $K = 1000$  data points for the power-law hypothesis tests, unless we specify it otherwise. We note that the Powerlaw library (Alstott et al., 2014) provides a convenient tool to compute the KS distance,  $d_{\mathrm{ks}}$ , and estimate the power exponent.
According to the practice of Kolmogorov-Smirnov Test (Massey Jr, 1951), we state the null hypothesis that the tested spectrum is not power-law. We state the alternative hypothesis, called the power-law hypothesis, that the tested spectrum is power-law. If  $d_{\mathrm{ks}}$  is higher than the critical value  $d_{\mathrm{c}}$  at the  $\alpha = 0.05$  significance level, we would accept the null hypothesis. In contrast, if  $d_{\mathrm{ks}}$  is lower than the critical value  $d_{\mathrm{c}}$  at the  $\alpha = 0.05$  significance level, we would reject the null hypothesis and accept the power-law hypothesis.
For each KS test in this paper, we select top  $k = 1000$  data points from dimension-wise gradients and iteration-wise gradients and top  $k = 1000$  covariance eigenvalues as the tested sets to measure the goodness of power laws. We choose the largest data points for two reasons. First, focusing on relatively large values is very reasonable and common in various fields' power-law studies (Stringer et al., 2019; Reuveni et al., 2008; Tang and Kaneko, 2020), as real-world distributions typically follow power laws only after/large than some cutoff values (Clauset et al., 2009) for ensuring the convergence of the probability distribution. Second, researchers are usually more interested in significantly large eigenvalues due to the low-rank matrix approximation.
B.2  $\chi^2$  Test
In this section, we introduce how we conduct  $\chi^2$  Test to evaluate the Gaussianity.
We directly used the  $\chi^2$  Normal Test implemented by the classical Python-based scientific computing package, Scipy (Virtanen et al., 2020), to evaluate the Gaussianity of empirical data. Note that we need to normalize the empirical data via whitening (zero-mean and unit-variance) before the tests.
The Gaussianity test statistic,  $p$ -value, is returned by the squared sum of the statistics of Skewness Test and Kurtosis Test (Cain et al., 2017). Skewness is a measure of symmetry. A distribution or dataset is symmetric if it looks the same to the left and right of the center. Kurtosis is a measure of whether the data are heavy-tailed or light-tailed relative to a normal distribution. Empirical data with high kurtosis tend to have heavy tails. Empirical data with low kurtosis tend to have light tails. Thus,  $\chi^2$  Test can reflect both Gaussianity and heavy tails. In this paper, we randomly choose  $K = 100$  data points for the Gaussian hypothesis tests, unless we specify it otherwise.
We may write the  $p$ -value return by  $\chi^2$  Test as
$$
p = z _ {S} ^ {2} + z _ {K} ^ {2}, \tag {9}
$$
where  $z_{S}$  is the Skewness Test statistic and  $z_{K}$  is the Kurtosis Test statistic. There are a number of ways to compute  $z_{S}$  and  $z_{K}$  in practice. It is convenient to use the default two-sided setting in Virtanen et al. (2020). Please refer to Virtanen et al. (2020) and the source code of stats.skewtest and stats.kurtosistest for the detailed implementation.
For each  $\chi^2$  test in this paper, we randomly select  $k = 100$  data points from both dimension-wise gradients and iteration-wise gradients as the tested set to measure the Gaussianity. The returned test statistic,  $p$ -value, is a classical indicator of the relative goodness of Gaussianity for two types of gradients.
C Statistical Test Results