SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 5: Parameter settings of the experiment reported in Section 4.3 Figure 7. The five random seeds for each setting are {43, 37, 13, 51, 71}, and the means and standard deviations of the test accuracy among the five seeds are reported.
Index	Dataset	Model	Method	HT-SR Metric	Initial learning rate η0	Test Acc	scaling ratio (s1,s2)
0	ResNet18	TB	SpectralNorm	0.05, 0.1, 0.15	77.83 ± 0.21, 78.30 ± 0.32, 78.27 ± 0.25	(0.5, 1.5)
1	ResNet18	TB	AlphaWeighted	0.05, 0.1, 0.15	78.18 ± 0.27, 78.67 ± 0.17, 78.48 ± 0.24	(0.5, 1.5)
1	ResNet18	TB	PL_Alpha_Hill	0.05, 0.1, 0.15	78.48 ± 0.27, 78.97 ± 0.29, 78.69 ± 0.11	(0.5, 1.5)
2	ResNet34	TB	SpectralNorm	0.05, 0.1, 0.15	78.25 ± 0.16, 78.71 ± 0.15, 78.92 ± 0.28	(0.5, 1.5)
3	ResNet34	TB	AlphaWeighted	0.05, 0.1, 0.15	78.36 ± 0.39, 78.87 ± 0.34, 78.83 ± 0.23	(0.5, 1.5)
3	CIFAR100	ResNet34	TB	PL_Alpha_Hill	0.05, 0.1, 0.15	79.36 ± 0.18, 79.89 ± 0.15, 79.09 ± 0.64	(0.5, 1.5)
4	VGG16	TB	SpectralNorm	0.025, 0.05, 0.1	73.58 ± 0.19, 74.29 ± 0.16, 74.17 ± 0.28	(0.5, 1.5)
5	VGG16	TB	AlphaWeighted	0.025, 0.05, 0.1	73.97 ± 0.22, 74.19 ± 0.11, 74.42 ± 0.31	(0.5, 1.5)
5	VGG16	TB	PL_Alpha_Hill	0.025, 0.05, 0.1	74.40 ± 0.31, 74.96 ± 0.15, 74.94 ± 0.16	(0.5, 1.5)
6	VGG19	TB	SpectralNorm	0.025, 0.05, 0.1	72.34 ± 0.26, 72.91 ± 0.35, 73.04 ± 0.39	(0.5, 1.5)
7	VGG19	TB	AlphaWeighted	0.025, 0.05, 0.1	72.85 ± 0.16, 73.41 ± 0.17, 73.33 ± 0.21	(0.5, 1.5)
7	VGG19	TB	PL_Alpha_Hill	0.025, 0.05, 0.1	73.47 ± 0.16, 73.77 ± 0.43, 73.40 ± 0.38	(0.5, 1.5)
Table 6: Parameter settings of the experiment reported in Section 4.3 Figure 8. The five random seeds for each setting are {43, 37, 13, 51, 71}, and the means and standard deviations of the test accuracy among the five seeds, the means and standard deviations of the computation time of using TB among the 10 times are reported.
Index	Dataset	Model	Method	PL fitting method	Initial learning rate η0	Test Acc	Computation Time (sec)	scaling ratio (s1, s2)
0	ResNet18	TB	Goodness-of-fit	0.1	78.59 ± 0.21	8.20 ± 0.53	(0.5, 1.5)
1	ResNet18	TB	Fix-finger	0.1	79.06 ± 0.22	7.24 ± 0.74	(0.5, 1.5)
1	ResNet18	TB	Median	0.1	78.97 ± 0.29	1.14 ± 0.04	(0.5, 1.5)
2	ResNet34	TB	Goodness-of-fit	0.1	79.13 ± 0.21	16.45 ± 0.48	(0.5, 1.5)
3	ResNet34	TB	Fix-finger	0.1	79.64 ± 0.22	15.13 ± 1.05	(0.5, 1.5)
3	ResNet34	TB	Median	0.1	79.89 ± 0.15	2.27 ± 0.06	(0.5, 1.5)
4	CIFAR100	VGG16	TB	Goodness-of-fit	0.05	74.46 ± 0.24	8.54 ± 0.10	(0.5, 1.5)
5	VGG16	TB	Fix-finger	0.05	74.48 ± 0.20	8.45 ± 0.59	(0.5, 1.5)
5	VGG16	TB	Median	0.05	74.96 ± 0.15	1.37 ± 0.05	(0.5, 1.5)
6	VGG19	TB	Goodness-of-fit	0.05	73.36 ± 0.16	11.48 ± 0.15	(0.5, 1.5)
7	VGG19	TB	Fix-finger	0.05	73.52 ± 0.16	11.15 ± 0.79	(0.5, 1.5)
7	VGG19	TB	Median	0.05	73.77 ± 0.43	1.85 ± 0.05	(0.5, 1.5)
E Comparison with more baselines
In Figure 17, we provide additional results by comparing TempBalance with LAMB and Adam. We found that our method outperforms both baseline methods. Furthermore, we also found that the Adam-based methods do not provide better results than the SGD baseline with cosine annealing (CAL) in our experiment setting, which was mentioned in Section 4.2. For Adam, we searched the initial learning rate over  $\{0.00005, 0.0001, 0.001, 0.01, 0.1\}$ , and we used  $\epsilon = 10^{-8}$ . For LAMB, we searched the initial learning rate over  $\{0.005, 0.01, 0.02\}$ , and we used  $\epsilon = 10^{-6}$ . Both methods used weight decay  $5.0 \times 10^{-4}$ ,  $\beta_{1} = 0.9$ ,  $\beta_{2} = 0.999$ , learning rate decay with cosine annealing. Each experiment was conducted with five random seeds.
We also discuss the difference between TempBalance and these two types of learning rate scheduling.
- Compared to layer-wise learning rate scheduling (e.g., LARS): TempBalance uses a more precise model quality metric, PL_Alpha_Hill from HT-SR Theory, to enhance the performance of deep models during training. This "shape-based" metric estimates the shape of the eigenspectrum of weight matrices. In contrast, LARS uses a "norm-based" metric, such as the layer-wise gradient norm. A recent study in HT-SR [22] has shown that the shape-based metrics surpasses norm-based ones in assessing model quality and performance. Figure 3 confirms that our method outperforms the layer-wise scheduler LARS in test accuracy.
- Compared to parameter-wise learning rate scheduling (e.g., Adam): Similarly, our method employs the "shape-based" metric PL_Alpha_Hi11 to improve the generalization, an approach not incorporated in traditional parameter-wise methods.
Figure 17: (Comparison with additional baselines). Comparing our method, TempBalance (TB), with other baselines such as parameter-wise learning rate schedulers Adam and LAMB, using ResNet18/34 trained on CIFAR100. Each cross represents the mean test accuracy of five random seeds.
(a) Maximum
(b) Minimum
Figure 18: (Layerwise gradient norm during training). From left to right: maximum, minimum, and mean of the layerwise gradient norm at every 30 iterations for the first 10 epochs. ResNet18 on CIFAR-100.
(c) Mean
F Does addressing other training issues lead to TempBalance's improvement?
We discuss whether the improvement from the proposed method, TempBalance, is due to indirectly addressing another fundamental training issue that could distort the ESD, specifically the gradient
Figure 19: (Histogram of gradient norm distribution during first epoch). ResNet18 on CIFAR-100.
Figure 20: (Impact of large rank-1 updates on ESD). Large rank-1 updates result in the spikes of ESD, observed exclusively during the first epoch. From the second epoch onward, the ESD exhibits a heavy-tailed distribution. ResNet18 on CIFAR-100.
Figure 21: (Varying the starting epoch of applying TempBalance (TB)). Postponing the usage of TempBalance to Epochs 2, 5, and 10 doesn't affect the performance of TempBalance (originally starting from Epoch 1).
magnitude excursions [80] (explosion/vanishing). This discussion further strengthens the connection between our method and the HT structure, as discussed in the Sections 1, A, and B.
We first summarize the questions and the corresponding primary findings, with subsequent detailing of our experiment and supporting results.
- Does gradient excursion exist? We discovered that gradient explosion does exist, but it is confined to the first epoch out of a total of 200 training epochs, leading us to believe it does not significantly impact the test accuracy. We observed no gradient vanishing.
- Does the observed gradient explosion impact the estimation of PL_Alpha_Hill? We discovered that the large rank-1 updates resulting from the gradient explosion do indeed
affect the ESD as well as the PL_Alpha_Hill estimation. However, this effect is again restricted to the first epoch.
- Does TempBalance boil down to addressing gradient explosion? We found that postponing the use of TempBalance until the epoch when neither the gradient explosion nor the PL_Alpha_Hill estimation is affected does not compromise the test accuracy.
To support the above answers, we conducted three experiments. We discuss the setup of these experiments first and then analyze the results.
- (Figures 18, 19) We aim to detect gradient excursion by tracking the gradient norm across layers during training. We examine the model every 30 iterations over the first 10 epochs, calculating the  $L_{2}$  norm of each gradient update across layers using the training batches of size 128. This produces an empirical gradient norm distribution with a total sample size of update numbers  $\times$  layer numbers. Figure 18 presents the maximum/minimum/mean of the distribution, while Figure 19 visualizes these distributions for several iterations of Epoch 1.
- (Figure 20) We aim to assess the impact of gradient explosion on PL_Alpha_Hi11 estimation by monitoring the ESD. Figure 20 examines the change of the ESD of a single weight matrix over several iterations, tracked in the experiment depicted in Figures 18 and 19.
- (Figure 21) We aim to see if TempBalance enhances generalization by implicitly addressing gradient explosion. Since the gradient explosion and its effect on PL_Alpha_Hill estimation only transpire in the first epoch, we postpone the starting epoch of TempBalance to Epochs 2, 5, and 10 and see if it affects the test accuracy.
Our answers to the above questions are supported by the results obtained from the three experiments:
- First question (Figure 18 and 19): We observed that the notable exploding gradients only occur in the initial 200 iterations of the first epoch. In Figure 18, we pinpoint a singular peak of maximum gradient norm within the first epoch. This aligns with the abnormal distribution with a large gradient norm in the subfigure of Figure 19 titled "Epoch 1, iteration 30."
- Second question (Figure 20): Note that large rank-one updates have been studied in random matrix theory, which manifests as a "bulk+spike" pattern. This has been analyzed in, e.g., Theorem 2.13 of [94]. Figure 20 shows this "bulk+spike" pattern, but only in the first epoch. The ESD exhibits a heavy-tail distribution in subsequent epochs, suggesting the influence of rank-one updates is limited.
- Third question (Figure 21): Delaying the application of TempBalance until after the first epoch does not adversely affect the test accuracy. Figure 21 illustrates that applying TempBalance from Epochs 2, 5, and 10 results in test performance comparable to when TempBalance is applied from Epoch 1. Since the gradient explosion only occurs in the first epoch and its effect on PL_Alpha_Hill estimation diminishes after this, the effectiveness of TempBalance does not rely on addressing gradient explosion or biased PL_Alpha_Hill estimation from large rank-one updates.
- Third question: We compare TempBalance with the baseline method LARS, which uses gradient norms to determine layer-wise learning rates in combating gradient vanishing/explosion issues. As illustrated in Figure 4, TempBalance outperforms LARS in terms of generalization performance.
G Corroborating results on other tasks
We provide corroborating results of applying TempBalance to two different tasks: object detection (OD) and language modeling (LM). In both tasks, TempBalance consistently improves generalization, outperforming the baseline scheduler cosine annealing (CAL) when both are combined with Adam/AdamW optimizers.
For OD, we studied the PASCAL VOC2007 [95] dataset with YOLO series [96] pre-trained model. We compared TempBalance with the baseline scheduler CAL with both applied to Adam/AdamW optimizer. For both scheduler methods, we trained for 200 epochs with batch size 64, and we set the same hyperparameter for the optimizers:  $\beta_{1} = 0.9$ ,  $\beta_{2} = 0.999$ ,  $\epsilon = 10^{-8}$ , weight decay  $= 5.0 \times 10^{-4}$ . We searched the initial learning rate for all methods among  $\{7.5 \times 10^{-6}, 1 \times 10^{-5}, 2.5 \times 10^{-5}\}$ . For metrics we use the COCO [97] version mean Average Precision (mAP, higher is better), which is calculated for 10 IOUs varying in a range of 0.5 to 0.95 with steps of 0.05. We
report the mean of mAP over five random seeds on the test set. We set the scaling factors  $(s_1, s_2)$  of TempBalance to be (0.6, 1.4).
Here are the experimental settings for LM. We studied the Penn Treebank (PTB) dataset [98] using a three-layer "tensorized transformer core-1" [99]. We compared TempBalance with the baseline scheduler CAL with both applied to Adam optimizer. For both scheduler methods, we trained the models for 40K iterations with a batch size of 120, and a dropout rate of 0.3. We searched the initial learning rate for baseline methods among  $\{1.25 \times 10^{-4}, 2.5 \times 10^{-4}, 5 \times 10^{-4}, 1 \times 10^{-3}, 1.25 \times 10^{-3}, 2.5 \times 10^{-3}, 5 \times 10^{-3}\}$  for baseline CAL. The hyperparameters for Adam are  $\beta_{1} = 0.9$ ,  $\beta_{2} = 0.999$ ,  $\epsilon = 10^{-8}$ . The mean of perplexity (PPL, lower is better) across five random seeds on the test set is reported. We observed improved performance of TempBalance in this task when extending our hyperparameter search to include the scaling factors  $(s_{1}, s_{2}) \in \{(0.5, 1.5), (1.0, 2.0)\}$ , the power-law fitting hyperparameter  $\lambda_{\mathrm{min}}$  index  $k \in \{\frac{n}{2}, \frac{n}{1.25}\}$ , and the TempBalance update interval over  $\{10, 25, 50\}$  iterations.
Table 7: (a) Object Detection (OD): mean Average Precision (mAP) on PASCAL VOC 2007 using model Yolov8n. (b) Language Modeling (LM): test perplexity (PPL) on Penn TreeBank (PTB) using the tensorized transformer. TempBalance (TB) consistently outperforms the CAL in different tasks.
CAL + Adam	TB + Adam	CAL + AdamW	TB + AdamW
59.59	60.03 (+0.44)	59.68	59.96 (+0.28)
(a) OD, VOC2007, mAP (↑)
CAL + Adam	TB + Adam
49.94	47.30 (-2.64)
(b) LM, PTB, PPL (↓)
We present additional results in Figure 22, showing the application of our method TempBalance to ResNet 101 on CIFAR-100, and we compare it with the baseline (CAL). We searched the initial learning rate among  $\{0.05, 0.1, 0.15\}$  for both the baseline and our method. The results report the mean and standard deviation across five seeds. We found that TempBalance offers improvements for the larger ResNet101 model comparable to those observed for ResNet18/34, demonstrating its potential for larger models.
Figure 22: (Applying the TempBalance (TB) to different sizes of ResNets). TempBalance consistently outperforms the baseline CAL method in the larger model ResNet101. The dataset is CIFAR100. Reporting mean/std over five random seeds.
H Analysis of computation overhead
We conducted a study on the computational overhead of TempBalance, demonstrating that our method is both applicable and scalable for large models. To do so, we conducted a scaling experiment to demonstrate that the computational cost remains low for different sizes of models. We recorded the duration of a single training epoch and the time taken to apply our method once. From this, we calculated the percentage increase in time when using TempBalance once per epoch, using this as an indicator of computational overhead. The experiment setup is based on ResNet-series on CIFAR100. We studied models of depth in {18, 34, 50, 101} and ResNet18 models of width in {512, 768, 1024, 2048}. We report the mean and the standard deviation of the results over 10 runs. The test platform was one Quadro RTX 6000 GPU with Intel Xeon Gold 6248 CPU. The results are presented in Figure 23. Our findings reveal that the computational overhead remains low (less than  $9\%$ ) even when applied to exceptionally wide or deep models (ResNet18 with width 2048 or ResNet101). The
(a) Scaling depth
(b) Scaling depth
(c) Scaling width
(d) Scaling width
Figure 23: (Computation overhead of TempBalance (TB) in scaling the model depth/width). (a)(c) Time duration (second) of one training epoch (blue) and using TempBalance once (red). (b)(d) Time increment of using TempBalance once per epoch. The dataset is CIFAR100, reporting mean/std over 10 epochs. The computational overhead of using TempBalance remains low (less than  $9\%$ ) even when applied to exceptionally wide or deep models.
CAL
Figure 24: (Varying the TempBalance (TB) update interval). Reducing the update interval from 390 iters (used in the paper) brings mild improvement in test accuracy. Both use ResNet18 on CIFAR100. Reporting mean/std over five random seeds.
computation overhead is not large because: 1) we select the efficient PL fitting method to obtain PL_Alpha_Hi11, which is demonstrated in Figure 8; and 2) the most computation-intensive part of our method is SVD decomposition, which we have optimized using GPU implementation and batch processing.
We conducted an experiment on reducing the update interval of the learning rate schedule to see if it affects the test accuracy of TempBalance. Figure 24 shows the experiments conducted with ResNet18 on CIFAR-100. We reduce the update interval from 390 iterations used in our paper (equivalent to one epoch) to 300, 200, 100, and 50. We observed that there indeed exists a trade-off between the computation time and test accuracy, but reducing the update interval only brings mild improvement.