SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We present the statistical test results of the eigengaps of the gradient covariances in Table 3 and the visualized results in Figure 16.
Figure 16: The eigengaps of stochastic gradient covariances are also approximately power-laws. Dataset: MNIST. Model: FCN. Left figure displays the eigengaps by original rank indices sorted by eigenvalues. Right figure displays the eigengaps by rank indices re-sorted by eigengaps.
Table 3: KS statistics of the covariance eigengaps of LeNet and FCN.
Dataset	Model	Training	dfs	dc	Power-Law	s
MNIST	LeNet	Pretrain	0.0205	0.043	Yes	5.111
MNIST	LeNet	Random	0.0221	0.043	Yes	2.232
MNIST	FCN	Pretrain	0.0219	0.043	Yes	1.668
MNIST	FCN	Random	0.0231	0.043	Yes	1.668
We present the statistical test results of dimension-wise gradients and iteration-wise gradients of LeNet and ResNet18 on various datasets in Tables 4 and 5.
We conducted the KS Tests for all of our studied covariance spectra. We display the KS test statistics and the estimated power exponents  $\hat{s}$  in the tables. For better visualization, we color accepting the power-law hypothesis in blue and color accepting the null hypothesis (and the cause) in red. The KS Test statistics of the covariance spectra are shown in Tables 6, 7, 8, 9, 10, and 11.
Table 4: The KS and  $\chi^2$  statistics and the hypothesis acceptance rates of iteration-wise gradients with respect to the batch size. Model: LeNet. Dataset: MNIST
Type	Training	Setting	dks	dc	Power-Law Rate	p-value	Gaussian Rate
Iteration	Random	B=1	0.428	0.0430	0.067%	0.047	12.6%
Iteration	Random	B=3	0.385	0.0430	0.17%	0.089	21.5%
Iteration	Random	B=10	0.267	0.0430	0.25%	0.173	28.5%
Iteration	Random	B=30	0.249	0.0430	0.16%	0.240	50.9%
Iteration	Random	B=100	0.191	0.0430	0.079%	0.321	65.1%
Iteration	Random	B=300	0.119	0.0430	0.033%	0.382	74.5%
Iteration	Random	B=1000	0.120	0.0430	0.041%	0.388	75.5%
Dimension	Random	B=1	0.0306	0.0430	90.6%	4.51 × 10-5	0%
Dimension	Random	B=3	0.0358	0.0430	74.5%	9.07 × 10-5	0.02%
Dimension	Random	B=10	0.0392	0.0430	65.3%	1.78 × 10-4	0%
Dimension	Random	B=30	0.0379	0.0430	68.9%	2.29 × 10-4	0.14%
Dimension	Random	B=100	0.0355	0.0430	76.6%	4.11 × 10-4	0.18%
Dimension	Random	B=300	0.0269	0.0430	97.5%	1.21 × 10-3	0.48%
Dimension	Random	B=1000	0.0309	0.0430	90.6%	1.43 × 10-4	0%
Table 5: The KS and  $\chi^2$  statistics and the hypothesis acceptance rates of the gradients over dimensions and iterations, respectively. Model: ResNet18. Batch Size: 100. In the second column "random" means randomly initialized models, while "pretrain" means pretrained models.
Dataset	Training	SG Type	d̅ks	dc	Power-Law Rate	p-value	Gaussian Rate
CIFAR-10	Random	Dimension	0.0924	0.0962	54.3%	1.73 × 10-2	6.4%
CIFAR-10	Random	Iteration	0.141	0.0962	1.32%	0.495	93.4%
CIFAR-10	Pretrain	Dimension	0.0717	0.0962	82.6%	1.1 × 10-2	3.2%
CIFAR-10	Pretrain	Iteration	0.140	0.0962	1.38%	0.497	93.5%
CIFAR-100	Random	Dimension	0.0631	0.0962	92.4%	8.55 × 10-3	3%
CIFAR-100	Random	Iteration	0.141	0.0962	1.36%	0.496	93.2%
CIFAR-100	Pretrain	Dimension	0.0637	0.0962	88.5%	8.11 × 10-3	3.4%
CIFAR-100	Pretrain	Iteration	0.140	0.0962	1.37%	0.496	93.1%
Table 6: The KS statistics of the second-moment spectra of dimension-wise gradients for LeNet on MNIST.
Dataset	Model	Training	Batch	Sample size	Setting	dks	dc	Power-Law	δ
MNIST	LeNet	Pretrain	1	1000	-	0.0206	0.0430	Yes	1.302
MNIST	LeNet	Pretrain	10	1000	-	0.0244	0.0430	Yes	1.313
MNIST	LeNet	Pretrain	100	1000	-	0.0171	0.0430	Yes	1.390
MNIST	LeNet	Pretrain	1000	1000	-	0.0173	0.0430	Yes	1.314
MNIST	LeNet	Pretrain	10000	1000	-	0.0204	0.0430	Yes	1.290
MNIST	LeNet	Pretrain	60000	1000	-	0.106	0.0430	No	0.206
MNIST	LeNet	Random	1	1000	-	0.0220	0.0430	Yes	1.428
MNIST	LeNet	Random	10	1000	-	0.0223	0.0430	Yes	1.334
MNIST	LeNet	Random	100	1000	-	0.0228	0.0430	Yes	1.313
MNIST	LeNet	Random	1000	1000	-	0.0198	0.0430	Yes	1.423
MNIST	LeNet	Random	10000	1000	-	0.0213	0.0430	Yes	1.284
MNIST	LeNet	Random	60000	1000	-	0.203	0.0430	No	0.271
Table 7: The KS statistics of the covariance spectra of dimension-wise gradients for LeNet on MNIST.
Dataset	Model	Training	Batch	Sample size	Setting	\(d_{\mathrm {ks}}\)	\(d_{\mathrm {c}}\)	Power-Law	\(\hat{s}\)
MNIST	LeNet	Random	1	1000	-	0.0226	0.0430	Yes	1.425
MNIST	LeNet	Random	10	1000	-	0.0227	0.0430	Yes	1.331
MNIST	LeNet	Random	100	1000	-	0.0230	0.0430	Yes	1.311
MNIST	LeNet	Random	1000	1000	-	0.0200	0.0430	Yes	1.423
MNIST	LeNet	Random	10000	1000	-	0.0287	0.0430	Yes	1.320
MNIST	LeNet	Pretrain	1	1000	-	0.0206	0.0430	Yes	1.299
MNIST	LeNet	Pretrain	10	1000	-	0.0247	0.0430	Yes	1.310
MNIST	LeNet	Pretrain	100	1000	-	0.0171	0.0430	Yes	1.386
MNIST	LeNet	Pretrain	1000	1000	-	0.0174	0.0430	Yes	1.312
MNIST	LeNet	Pretrain	10000	1000	-	0.0223	0.0430	Yes	1.331
MNIST	LeNet	Pretrain	1	1000	Label Noise 40%	0.0289	0.0430	Yes	1.453
MNIST	LeNet	Pretrain	1	1000	Label Noise 80%	0.0138	0.0430	Yes	11.442
MNIST	LeNet	Pretrain	1	1000	Random Label	0.0129	0.0430	Yes	1.374
MNIST	LeNet	Pretrain	1	1000	GradClip=1	0.0226	0.0430	Yes	1.323
MNIST	LeNet	Pretrain	1	1000	GradClip=0.1	0.0261	0.0430	Yes	1.343
Table 8: The KS statistics of the second-moment spectra of dimension-wise gradients for FCN on MNIST.
Dataset	Model	Training	Batch	Sample size	Setting	dks	dc	Power-Law	s
MNIST	2Layer-FCN	Pretrain	10	1000	-	0.0415	0.0430	Yes	0.866
MNIST	2Layer-FCN	Random	10	1000	-	0.0418	0.0430	Yes	0.864
MNIST	2Layer-FCN	Random	10	1000	Noise	0.0427	0.0430	Yes	0.862
MNIST	2Layer-FCN	Pretrain	10	1000	Width=70	0.0415	0.0430	Yes	0.866
MNIST	2Layer-FCN	Pretrain	10	1000	Width=30	0.0425	0.0430	Yes	0.869
MNIST	2Layer-FCN	Pretrain	10	1000	Width=10	0.0486	0.0430	No
MNIST	2Layer-FCN	Random	10	1000	Width=70	0.0418	0.0430	Yes	0.864
MNIST	2Layer-FCN	Random	10	1000	Width=30	0.0488	0.0430	No
MNIST	2Layer-FCN	Random	10	1000	Width=10	0.0491	0.0430	No
MNIST	1Layer-FCN	Random	10	1000	-	0.0384	0.0430	Yes	1.357
MNIST	1Layer-FCN	Pretrain	10	1000	-	0.0384	0.0430	Yes	1.355
Table 9: The KS statistics of the second-moment spectra of dimension-wise gradients for LNN on MNIST.
Dataset	Model	Training	Batch	Sample size	Setting	dks	dc	Power-Law	s
MNIST	4Layer-LNN	Pretrain	10	1000	-	0.0445	0.0430	No	1.629
MNIST	4Layer-LNN	Pretrain	10	1000	BatchNorm	0.0268	0.0430	Yes	0.955
MNIST	4Layer-LNN	Pretrain	10	1000	ReLU	0.0154	0.0430	Yes	1.074
Table 10: The KS statistics of the covariance spectra of dimension-wise gradients for LeNet on CIFAR-10.
Dataset	Model	Training	Batch	Sample size	Setting	dks	dc	Power-Law	s
CIFAR-10	LeNet	Pretrain	1	1000	-	0.0201	0.0430	Yes	1.257
CIFAR-10	LeNet	Random	1	1000	-	0.0214	0.0430	Yes	1.300
CIFAR-10	LeNet	Pretrain	1	1000	GradClip=0.1	0.0244	0.0430	Yes	1.348
CIFAR-10	LeNet	Random	100	1000	SGD	0.00818	0.0430	Yes	1.305
CIFAR-10	LeNet	Random	100	1000	Weight Decay	0.0107	0.0430	Yes	1.300
CIFAR-10	LeNet	Random	100	1000	Momentum	0.00806	0.0430	Yes	1.262
CIFAR-10	LeNet	Random	100	1000	Adam	0.0634	0.0430	No
Table 11: The KS statistics of the covariance spectra of LeNet on CIFAR-100.
Dataset	Model	Training	Batch	Sample size	Setting	dks	dc	Power-Law	s
CIFAR-100	LeNet	Pretrain	1	1000	-	0.0287	0.0430	Yes	1.276
CIFAR-100	LeNet	Random	1	1000	-	0.0307	0.0430	Yes	1.229
CIFAR-100	LeNet	Pretrain	1	1000	-	0.0197	0.0430	Yes	1.076