SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Although deriving an exact equivalent between these two can be hard, we numerically show that the PL in one matrix informs the PL in the other. To visualize their relationship in the presence of PL, we train a simple MLP on MNIST [86] with one hidden layer and 2000 neurons for 50 epochs. We leverage the spectral regularization from Nassar et al. [34] to make the output covariance matrix exhibit a PL spectrum. Meanwhile, we calculate the top eigenvalues of the covariance and the Hessian [87], fit the PL exponent  $s$  for each matrix, and compare the PL exponents against each other. More specifically, we take trained NNs from epochs {1, 10, 20, 30, 40, 50} and plot the Hessian PL exponent  $s$  versus the output covariance PL exponent  $s$ . From the results shown in Figure 9, we can see that their PL exponent  $s$  shows a strong correlation, which supports our claim that the PL phenomena in one matrix can inform the other.
Connections to the NTK matrix. Interestingly, if we ignore the constant in (8) and switch the two matrices multiplied together, we obtain  $\nabla_{\theta}\tilde{f}_{\theta}(x)^{T}\nabla_{\theta}\tilde{f}_{\theta}(x)$ . This matrix is equal to Neural Tangent Kernel(NTK) [88], which is a kernel used to approximate the deep NN when NN's width is infinite. We thus conjecture that NTK should show PL when the NN is well trained [89]. Indeed, Karakida et al. [81] and Karakida et al. [82] study the eigenvalues of NTK, showing a PL trend. Some other work on stochastic gradient [90] claim that the so-called "stochastic gradient matrix" (which is similar to the NTK matrix) shows a PL spectrum as well, which matches our expectations. Also, Lewkowycz et al. [91], Dyer and Gur-Ari [92] show that the eigenvalues of NTK are similar to those in the Hessian, which again meets our expectation because the Hessian tends to be PL when NNs are well-trained [35].
In summary, this section investigates different "important matrices" and shows that they are tightly correlated to each other in terms of the PL trends: if one matrix shows a PL spectrum, there is a high chance that the other ones show something similar.
A.2 Connections between PL in ESD and PL in decaying eigenvalues
Next, we derive the connection between our PL_Alpha_Hill metric and the exponent of PL distribution on decaying eigenvalues. Take the covariance matrix (5) as an instance. According to Nassar et al. [34], the HT phenomenon in the output covariance matrix is similar to the layer-wise covariance matrices. Thus, without the loss of generality, we can consider the case when there is only one layer in the NN. We assume the weight matrix  $L$  is in  $\mathbf{R}^{N\times Q}$ . According to prior works, when  $L$  is well-trained, the ESD follows a PL distribution:
$$
p (\lambda) = \frac {1}{H} \lambda^ {- \alpha}, \quad \lambda_ {\min } <   \lambda <   \lambda_ {\max }. \tag {19}
$$
Here,  $H$  is a normalizing constant, and  $\alpha$  is the PL exponent.
Another way to characterize the PL phenomenon is to consider eigenvalues directly following a PL series. For example, Xie et al. [35] show that the decaying eigenvalues follows the following PL
Figure 10: We show the connection between PL_Alpha_Hill and the PL exponent of the decaying eigenvalues (denoted as  $s$ ) satisfy  $s = \frac{1}{\mathrm{PL\_Alpha\_Hill} - 1}$ . Results are shown for different matrix size  $Q$ . In particular, we see that PL_Alpha_Hill = 2 [19] is equivalent to  $s = 1$  [33] in the linear case.
series:
$$
\lambda_ {k} = \lambda_ {1} k ^ {- s}, k = 1, 2, \dots , Q, \tag {20}
$$
where  $\lambda_{1}$  is the same as  $\lambda_{\mathrm{max}}$  used in the main paper.
Now, we will analytically and empirically show that these two ways of characterizing PL are strongly related. Furthermore, the two PL coefficients satisfy  $s = \frac{1}{\alpha - 1}$ .
An analytical way to show that  $s = \frac{1}{\alpha - 1}$ . The derivation is actually quite simple. Consider the case that  $\lambda_k = \lambda_1 k^{-s}$  (i.e., (20) holds), and suppose  $\Lambda$  is a random variable distributed according to the empirical distribution from these eigenvalues  $\lambda_k = \lambda_1 k^{-s}$ . Now, from (20), we can see that the distribution function takes the following form:
$$
\mathbb {P} \left(\Lambda > \lambda_ {1} k ^ {- s}\right) = \frac {k}{Q}. \tag {21}
$$
By changing variables  $\lambda_1 k^{-s} = \lambda$ , we get the cumulative distribution function of  $\Lambda$ :
$$
\mathbb {P} (\Lambda > \lambda) \sim \lambda^ {- \frac {1}{s}}. \tag {22}
$$
After that, we take the derivative with respect to  $\lambda$ , and we get the ESD:
$$
p (\lambda) \sim \lambda^ {- \left(\frac {1}{s} + 1\right)}. \tag {23}
$$
In other words, we have  $\lambda^{-\left(\frac{1}{s} + 1\right)} = \lambda^{-\alpha}$ , which means  $s = \frac{1}{\alpha - 1}$ .
An empirical way to show that  $s = \frac{1}{\alpha - 1}$ . We consider matrices of size  $Q \times Q$ , where we choose  $Q$  in {16, 32, 64, 128, 256, 512, 768, 1024}, and we assign the parameters such that the decaying eigenvalues obey the formula  $\lambda_1 k^{-s}$ , for  $s$  in {0.2, 0.3, 0.4, ..., 3.2}. Then, we fit the ESD and get our estimate PL_Alpha_Hill. We plot the relationship between PL_Alpha_Hill and  $s$  in Figure10. From Figure 10, we find that the connection between PL_Alpha_Hill and  $s$  shows a good fit with the formula  $s = \frac{1}{\alpha - 1}$ . With increasing matrix size  $Q$ , the fitting becomes increasingly accurate.
When  $s = \frac{1}{\alpha - 1}$ ,  $s = 1$  corresponds to  $\alpha = 2$ . Some prior works Nassar et al. [34], Xie et al. [35], Bartlett et al. [93] measure the HT phenomena from the perspective of decaying eigenvalues with PL exponent  $s$ , and they show either theoretically or empirically that  $s = 1$  is the optimal exponent. Now that we have  $s = \frac{1}{\alpha - 1}$  in the linear case, and from the theory of NTK[88], the infinite wide NN is approximated as a linear model, we tend to believe that  $\alpha = 2$  satisfies a similar property. Indeed, one of the main contributions of Martin and Mahoney [19] is to establish different HT families of ESDs, and  $\alpha = 2$  is believed to be the boundary between "moderately HT" and "very HT," corresponding to the best models. Martin and Mahoney [19] further argue that the optimal exponent for PL_Alpha is in the range [2,4]. Combining the perspective from Nassar et al. [34], Xie et al. [35], Bartlett et al. [93] and those from Martin and Mahoney [19], it is reasonable to believe that the optimal exponent for PL_Alpha is around 2. When PL_Alpha is much higher or lower than 2,
the NN probably has some issue in training. Although we argued in the main paper that the absolute numerical value of PL_Alpha is unimportant in implementing our TempBalance algorithm, it is, however, helpful to have an "optimal" PL_Alpha value to test if our algorithm actually works in controlling the ESDs. We will show visualization results in Appendix B that TempBalance leads to a better distribution of our estimated PL_Alpha_Hill.
In summary, this section explores two distinct methods for determining PL fit. We demonstrate that, although these two methods yield numerically distinct PL exponents, they essentially capture the same underlying phenomenon. Moreover, it is noteworthy that the "optimal" values of the PL exponents reported in various papers are consistent with one another [19, 34, 35, 93].
B Visualization results: how does TempBalance control ESDs
We demonstrate that the proposed method, TempBalance, effectively controls the shape of ESDs, resulting in a more favorable distribution of PL_Alpha_Hill among the layers of NNs compared to the baseline method CAL. This observation elucidates the superior performance of TempBalance over CAL in our main experiment, as presented in Section 4.2.
We evaluate the models reported in the main paper. For each individual NN, we compute and aggregate PL_Alpha_Hill values across all layers, excluding the first and last layers that have an extremely small number of eigenvalues and thus cause inaccurate PL_Alpha_Hill estimation. We aggregate the PL_Alpha_Hill values from five models trained using different random seeds for each method. Figure 11 shows the distribution of PL_Alpha_Hill of TempBalance and the baseline CAL. Comparing TempBalance with CAL, we see that TempBalance consistently yields a more concentrated distribution. Furthermore, TempBalance causes the median and mean of the distribution to approach 2 (shown in each subplot respectively as the middle vertical line and the red star). The value 2 represents the theoretically optimal PL_Alpha_Hill value, as we have justified in Appendix A.
Next, in Figure 12, we group the models into different subgroups based on their architectures and/or datasets, aggregating the PL_Alpha_Hill values and comparing the distributions of the two methods TempBalance and CAL. Once again, we observe that TempBalance results in a more concentrated distribution, with a larger number of samples (layers) having PL_Alpha_Hill values closer to 2.
We provide visualization to demonstrate how the learning rates are distributed over layers during the training. In Figure 13, we report the learning rate and PL_Alpha_Hi11 every epoch throughout the 200-epoch training duration. The key observation includes the following.
1. How does the learning rate vary across layers? We observed a correlation between the layer-wise learning rate and the layer-wise PL_Alpha_Hill distribution: layers with larger PL_Alpha_Hill are allocated larger learning rates, whereas those with smaller PL_Alpha_Hill receive smaller learning rates.
2. How does the layer-wise learning rate evolve during training? The variations in layerwise learning rates closely reflect shifts in the layer-wise PL_Alpha_Hill distribution. Initially, the PL_Alpha_Hill distributes uniformly across layers but eventually converge to a layer-wise pattern where earlier layers have smaller PL_Alpha_Hill and later layers have larger ones.
We present visualizations of how PL_Alpha_Hill and learning rate evolve through training. In Figure 14, we show PL_Alpha_Hill and learning rate of two layers within the same ResNet18 during the training process. The two layers are layer1.0Conv2 (index=1) and layer4.0Conv2 (index=15). From Figure 14b and 14d, we can see that with the baseline CAL scheduler (blue curves), the earlier layer (index=1) achieves a smaller PL_Alpha_Hill value compared to the larger PL_Alpha_Hill value of the later layer (index=15). In contrast, TempBalance (orange curves) narrows this gap, indicating our approach balances the undertraining/overtraining levels (as signified by PL_Alpha_Hill) of different layers. This balancing effect is further corroborated by Figures 11 and 12, where our method consistently refines the layer-wise PL_Alpha_Hill distribution. Regarding the learning rate plots in Figure 14a and 14c, TempBalance allocates a lower learning rate for earlier layers and a higher one for later layers than the baseline does. This leads to a more balanced PL_Alpha_Hill distribution between layers as mentioned above. Additionally, we noted
(i) ResNet18, CIFAR10
(j)VGG16,CIFAR10
(k) ResNet18, SVHN
(1)VGG16,SVHN
Figure 11: Comparing the distribution of PL_Alpha_Hill of NNs trained by our method TempBalance (TB) and CAL. The mean of each distribution is indicated by a red star marker. Each distribution aggregates the PL_Alpha_Hill values from models trained using five different random seeds. Across all experiments, our method TempBalance consistently yields a more concentrated distribution, resulting in the mean and median approaching the theoretically optimal PL_Alpha_Hill value of 2, as supported in Appendix A.
(e) TIN
(f) CIFAR100
(g) CIFAR10
Figure 12: Comparing our method TempBalance (TB) to CAL in terms of the distribution of PL_Alpha_Hill of aggregating NNs into different architectures and datasets. Each distribution aggregates the PL_Alpha_Hill of models trained with five random seeds. Across all subgroups, our method TempBalance consistently exhibits a more concentrated distribution, accompanied by a higher number of layers approaching a PL_Alpha_Hill value close to 2. This value of 2 corresponds to the theoretically optimal PL_Alpha_Hill value, as justified in Appendix A.
(h) SVHN
(a) LR, ResNet18
(b) Alpha, ResNet18
(c) LR, ResNet34
(d) Alpha, ResNet34
Figure 13: (Visualization of layer-wise learning rate (LR) and PL_Alpha_Hill (Alpha) over training). (a-b) The layer-wise LR and PL_Alpha_Hill of ResNet18 over training. (c-d) The layer-wise LR and PL_Alpha_Hill of ResNet34 over training.
(a) LR, layer index=1
Figure 14: (Visualization of learning rate (LR) and PL_Alpha_Hill (Alpha) of two layers during training) (a-b) LR and PL_Alpha_Hill of one layer with index  $= 1$  in ResNet18. (c-d) LR and PL_Alpha_Hill of one layer with index  $= 15$  in ResNet18. The ResNet18 is trained on CIFAR100.
(b) Alpha, layer index=1
(c) LR, layer index=15
(d) Alpha, layer index=15
instability in the learning rate curves during early training phases, while smoother transitions emerge in later phases.
We provide additional ablation studies on the choices of learning rate assignment function, assignment hyperparameters.
Varying LR assignment function. For TempBalance, we selected the linear interpolation (Equation 2) for learning rate assignment function  $f_{t}$ , based on its superior performance in our ablation study.
We evaluated three alternative learning rate assignment functions: Square root (Sqrt), Log2, and Step:
$\mathrm{Sqrt}:f_t(i) = \eta_t\frac{\sqrt{\alpha_t^i}}{\frac{1}{L}\sum_{j = 1}^L\sqrt{\alpha_t^j}},$
$\bullet$  Log2:  $f_{t}(i) = \eta_{t}\frac{log(\alpha_{t}^{i})}{\frac{1}{L}\sum_{j = 1}^{L}log(\alpha_{t}^{j})},$
- Step: For layer  $i$  with  $k$ -th minimum PL_Alpha_Hill among all the layers,
$$
f _ {t} (i) = \eta_ {t} \left(s _ {1} + (k - 1) \frac {s _ {2} - s _ {1}}{L - 1}\right)
$$
Here,  $\eta_{t}$  denotes the base global learning rate at epoch  $t$ ,  $(s_1,s_2)$  represents the minimum and maximum learning rate scaling ratios relative to  $\eta_t$ ,  $\alpha_t^i$  is the PL_Alpha_Hil1 estimate of the layer  $i$  at epoch  $t$ , and  $L$  is the total number of model layers. All these notations are consistently used in the main paper.
As depicted in Figure 15, TempBalance (TB), with the current assignment function, surpasses the other designs when tested on VGG and ResNet architectures on CIFAR100. All hyperparameters are consistent with the main paper. Each experiment was conducted with five random seeds.
Figure 15: (Different designs for learning rate assignment function.) Results of using different learning rate assignment functions on different architectures and CIFAR-100. Our design in the main paper TempBalance (TB) outperforms others. Reporting mean/std over five random seeds.
Varying LR assignment function hyperparameters. We provide additional results of a hyperparameter study on  $(s_1, s_2)$ , in which we consider five different settings for  $(s_1, s_2)$ : [(0.5, 1.5), (0.6, 1.4), (0.7, 1.3), (0.8, 1.2), (0.9, 1.1)]. We run tasks on CIFAR100 with four VGG and ResNet architectures, each with five random seeds. Our results in Figure 16 show that a larger learning rate scaling range (0.5, 1.5) performs best. This hyperparameter setting is the default setting used in our paper. All hyperparameters are consistent with those described in the main paper.
Figure 16: (Hyperparameter study on  $(s_1, s_2)$ ). Search for hyperparameters  $(s_1, s_2)$  with different architectures on CIFAR100. The current hyperparameter choice (0.5, 1.5) used in the paper performs best among all the cases. Reporting mean/std over five random seeds.
D Hyperparameter settings for reproducing our results
We report all hyperparameters, random seeds and all numerical values of experimental results shown in the main paper (in Section 4).
First, we report the common hyperparameters shared by all the experiments: the default optimizer is SGD, trained with batch size 128, number of training epochs 200, weight decay 5e-4, and momentum 0.9. The default HT-SR metric used in TempBalance is PL_Alpha_Hill. For each experimental setting, we use five random seeds, which are always 43, 37, 13, 51, 71, and we report the mean and standard deviation of the test accuracy across these seeds.
First, Table 1 reports the details of experiments shown in Figure 3. We carefully tune the initial learning rate  $\eta_0$  and  $\lambda_{sr}$  for the two baseline methods CAL and SNR. Then, Table 2 reports the detailed hyperparameter settings of the experiments shown in Figure 4. We again carefully tune the hyperparameters of various baseline optimizers and schedulers, as specified in their papers. Finally, Table 3, Table 4, Table 5 and Tabel 6 respectively report the details of the experiments shown in Figure 5, Figure 6, Figure 7 and Figure 8.
Table 1: Parameter settings of the experiment reported in Section 4.2 Figure 3. The hyperparameter in bold is the best hyperparameter selection reported in the main paper. The five random seeds for each setting are {43, 37, 13, 51, 71}, and the means and standard deviations of the test accuracy among the five seeds are reported.
Index	Dataset	Model	Method	Initial learning rate η0	λsr	Test Acc (best hyperparam.)	scaling ratio (s1, s2)
0	ResNet18	CAL	0.05, 0.1, 0.15	-	78.31 ± 0.05	-
1	ResNet18	SNR	0.1	0.001, 0.005, 0.01, 0.015	78.65 ± 0.29	-
2	ResNet18	TB	0.1	-	78.97 ± 0.29	(0.5, 1.5)
3	ResNet18	TB + SNR	0.1	0.001	79.06 ± 0.32	(0.6, 1.4)
4	ResNet34	CAL	0.05, 0.1, 0.15	-	78.98 ± 0.14	-
5	ResNet34	SNR	0.1	0.001, 0.005, 0.01, 0.015	79.97 ± 0.21	-
6	ResNet34	TB	0.1	-	79.89 ± 0.15	(0.5, 1.5)
7	CIFAR100	ResNet34	TB + SNR	0.1	0.005	80.09 ± 0.35	(0.6, 1.4)
8	VGG16	CAL	0.025, 0.05, 0.1	-	74.59 ± 0.23	-
9	VGG16	SNR	0.05	0.001, 0.005, 0.01, 0.015	74.80 ± 0.28	-
10	VGG16	TB	0.05	-	74.96 ± 0.15	(0.5, 1.5)
11	VGG16	TB + SNR	0.05	0.005	75.52 ± 0.46	(0.6, 1.4)
12	VGG19	CAL	0.025, 0.05, 0.1	-	73.26 ± 0.37	-
13	VGG19	SNR	0.05	0.001, 0.005, 0.01, 0.015	74.37 ± 0.16	-
14	VGG19	TB	0.05	-	73.77 ± 0.43	(0.5, 1.5)
15	VGG19	TB + SNR	0.05	0.01	74.74 ± 0.10	(0.5, 1.5)
16	ResNet18	CAL	0.05, 0.1, 0.15	-	66.25 ± 0.17	-
17	ResNet18	SNR	0.1	0.001, 0.005, 0.01, 0.015	66.20 ± 0.22	-
18	ResNet18	TB	0.1	-	66.77 ± 0.25	(0.6, 1.4)
19	ResNet18	TB + SNR	0.1	0.001	66.86 ± 0.22	(0.6, 1.4)
20	ResNet34	CAL	0.05, 0.1, 0.15	-	68.19 ± 0.16	-
21	ResNet34	SNR	0.1	0.001, 0.005, 0.01, 0.015	68.69 ± 0.13	-
22	ResNet34	TB	0.1	-	69.12 ± 0.16	(0.6, 1.4)
23	ResNet34	TB + SNR	0.1	0.001	69.27 ± 0.21	(0.6, 1.4)
24	TinyImageNet	WRN16-8	CAL	0.05, 0.1, 0.15	-	63.67 ± 0.09	-
25	WRN16-8	SNR	0.1	0.00005, 0.0001, 0.001	63.98 ± 0.23	-
26	WRN16-8	TB	0.1	-	64.09 ± 0.17	(0.6, 1.4)
27	WRN16-8	TB + SNR	0.1	0.0001	64.08 ± 0.07	(0.6, 1.4)
28	WRN28-6	CAL	0.05, 0.1, 0.15	-	65.88 ± 0.20	-
29	WRN28-6	SNR	0.1	0.00005, 0.0001, 0.001	66.09 ± 0.25	-
30	WRN28-6	TB	0.1	-	66.58 ± 0.23	(0.6, 1.4)
31	WRN28-6	TB + SNR	0.1	0.0001	66.79 ± 0.25	(0.6, 1.4)
32	ResNet18	CAL	0.05, 0.1, 0.15	-	95.53 ± 0.12	-
33	ResNet18	SNR	0.1	0.001, 0.005, 0.01, 0.015	95.57 ± 0.06	-
34	ResNet18	TB	0.1	-	95.63 ± 0.08	(0.5, 1.5)
35	CIFAR10	ResNet18	TB + SNR	0.1	0.001	95.66 ± 0.09	(0.6, 1.4)
36	VGG16	CAL	0.025, 0.05, 0.1	-	93.98 ± 0.12	-
37	VGG16	SNR	0.05	0.001, 0.005, 0.01, 0.015	94.04 ± 0.07	-
38	VGG16	TB	0.05	-	94.14 ± 0.06	(0.5, 1.5)
39	VGG16	TB + SNR	0.05	0.005	94.26 ± 0.10	(0.6, 1.4)
40	ResNet18	CAL	0.05, 0.1, 0.15	-	96.59 ± 0.08	-
41	ResNet18	SNR	0.1	0.001, 0.005, 0.015, 0.01	96.65 ± 0.12	-
42	ResNet18	TB	0.1	-	96.63 ± 0.06	(0.5, 1.5)
43	ResNet18	TB + SNR	0.1	0.01	96.67 ± 0.09	(0.6, 1.4)
44	SVHN	VGG16	CAL	0.025, 0.05, 0.1	-	96.28 ± 0.04	-
45	VGG16	SNR	0.05	0.001, 0.005, 0.015, 0.01	96.32 ± 0.07	-
46	VGG16	TB	0.05	-	96.33 ± 0.06	(0.5, 1.5)
47	VGG16	TB + SNR	0.05	0.005	96.40 ± 0.08	(0.6, 1.4)
Table 2: Parameter settings of the experiment reported in Section 4.2 Figure 4. The hyperparameter in bold is the best hyperparameter selection reported in the main paper. The five random seeds for each setting are {43, 37, 13, 51, 71}, and the means and standard deviations of the test accuracy among the five seeds are reported.
Index	Dataset	Model	Method	Initial learning rate η0	SGDR (T0, Tmut)	Lookahead k	Lookahead α	Test Acc (best hyperparams.)	scaling ratio (s1, s2)
0	ResNet18	CAL	0.05, 0.1, 0.15	-	-	-	78.31 ± 0.05	-
1	ResNet18	SGDR	0.05, 0.1, 0.15	(100,1), (10, 2), (1, 2)	-	-	77.69 ± 0.20	-
2	ResNet18	LARS	26, 28, 30, 32, 34	-	-	-	78.44 ± 0.12	-
3	ResNet18	Lookahead	0.05, 0.1, 0.15	-	10, 5	0.8, 0.5	78.46 ± 0.18	-
4	ResNet18	SGDP	0.01, 0.05, 0.1, 0.15, 0.2	-	-	-	78.74 ± 0.11	-
5	ResNet18	TB	0.05, 0.1, 0.15	-	-	-	78.97 ± 0.29	(0.5, 1.5)
6	CIFAR100	ResNet18	TB + SGDP	0.05, 0.1, 0.15	-	-	-	79.13 ± 0.15	(0.5, 1.5)
7	ResNet34	CAL	0.05, 0.1, 0.15	-	-	-	78.98 ± 0.14	-
8	ResNet34	SGDR	0.05, 0.1, 0.15	(100,1), (10, 2), (1, 2)	-	-	78.61 ± 0.20	-
9	ResNet34	LARS	26, 28, 30, 32, 34	-	-	-	78.94 ± 0.19	-
10	ResNet34	Lookahead	0.05, 0.1, 0.15	-	10, 5	0.8, 0.5	79.19 ± 0.12	-
11	ResNet34	SGDP	0.01, 0.05, 0.1, 0.15, 0.2	-	-	-	79.34 ± 0.21	-
12	ResNet34	TB	0.05, 0.1, 0.15	-	-	-	79.89 ± 0.15	(0.5, 1.5)
13	ResNet34	TB + SGDP	0.05, 0.1, 0.15	-	-	-	79.94 ± 0.30	(0.5, 1.5)
Table 3: Parameter settings of the experiment reported in Section 4.3 Figure 5. The five random seeds for each setting are {43, 37, 13, 51, 71}, and the means and standard deviations of the test accuracy among the five seeds are reported.
Index	Dataset	Model	Method	Initial learning rate η0	Test Acc	scaling ratio (s1, s2)
0	ResNet18	CAL	0.05, 0.1, 0.15	78.08 ± 0.19, 78.31 ± 0.05, 77.72 ± 0.44	-
1	ResNet18	TB	0.05, 0.1, 0.15	78.48 ± 0.27, 78.97 ± 0.29, 78.69 ± 0.11	(0.5, 1.5)
2	ResNet34	CAL	0.05, 0.1, 0.15	78.98 ± 0.14, 78.89 ± 0.24, 78.51 ± 0.34	-
3	CIFAR100	ResNet34	TB	0.05, 0.1, 0.15	79.36 ± 0.18, 79.89 ± 0.15, 79.09 ± 0.64	(0.5, 1.5)
4	VGG16	CAL	0.025, 0.05, 0.1	73.96 ± 0.27, 74.59 ± 0.23, 74.46 ± 0.12	-
5	VGG16	TB	0.025, 0.05, 0.1	74.40 ± 0.31, 74.96 ± 0.15, 74.94 ± 0.16	(0.5, 1.5)
6	VGG19	CAL	0.025, 0.05, 0.1	72.57 ± 0.45, 73.26 ± 0.37, 72.98 ± 0.16	-
7	VGG19	TB	0.025, 0.05, 0.1	73.47 ± 0.16, 73.77 ± 0.43, 73.40 ± 0.38	(0.5, 1.5)
Table 4: Parameter settings of the experiment reported in Section 4.3 Figure 6. The five random seeds for each setting are {43, 37, 13, 51, 71}, and the means and standard deviations of the test accuracy among the five seeds are reported.
Index	Dataset	Model	Method	Initial learning rate η0	Width	Test Acc	scaling ratio (s1, s2)
0	ResNet18	CAL	0.1	256, 512, 768	75.05 ± 0.26, 78.31 ± 0.05, 79.44 ± 0.26	-
1	ResNet18	TB	0.1	256, 512, 768	75.63 ± 0.12, 78.97 ± 0.29, 80.47 ± 0.18	(0.5, 1.5)
2	ResNet34	CAL	0.1	256, 512, 768	76.79 ± 0.34, 78.89 ± 0.24, 79.94 ± 0.31	-
3	CIFAR100	ResNet34	TB	0.1	256, 512, 768	77.25 ± 0.14, 79.89 ± 0.15, 80.23 ± 0.53	(0.5, 1.5)
4	VGG16	CAL	0.05	256, 512, 768	71.04 ± 0.14, 74.59 ± 0.23, 75.53 ± 0.32	-
5	VGG16	TB	0.05	256, 512, 768	71.26 ± 0.26, 74.96 ± 0.15, 76.19 ± 0.14	(0.5, 1.5)
6	VGG19	CAL	0.05	256, 512, 768	69.58 ± 0.39, 73.26 ± 0.37, 74.39 ± 0.33	-
7	VGG19	TB	0.05	256, 512, 768	69.96 ± 0.25, 73.77 ± 0.43, 74.80 ± 0.35	(0.5, 1.5)