SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Comparison with Lin and Rosasco [37]. Lin and Rosasco [37] study stochastic gradient descent with arbitrary batchsize, which is reduced to full batch gradient descent when setting the batchsize to sample size  $n$ . Their results are different from ours, since they require the boundness assumption, and focus more on the optimal early stopping time rather than the largest compatibility region, in the same spirit of Yao et al. [72]. Specifically, Lin and Rosasco [37] derive a region like  $(0, n^{\frac{\zeta + 1}{2\zeta + \gamma}})$ , where  $\zeta$  and  $\gamma$  are problem dependent constants (See Theorem 1 in Lin and Rosasco [37] for details). The following examples demonstrate that this paper's results yield larger regions for a wide range of distribution classes.
Example B.2. (Inverse Polynomial). If the spectrum of  $\Sigma$  satisfies
for some  $\alpha > 1$ . For this distribution, we have  $\zeta = \frac{1}{2}$ ,  $\gamma = \frac{1}{\alpha}$ , and their region is  $(0, n^{\frac{3\alpha}{2\alpha + 1}})$ , which is smaller than  $(0, n^{\frac{3\alpha + 1}{2\alpha + 1}})$  given in Example 5.2.
Example B.3. (Inverse Log-Polynomial). If the spectrum of  $\Sigma$  satisfies
for some  $\beta > 1$ . For this distribution, we have  $\zeta = \frac{1}{2}$ ,  $\gamma = 1$ , and their region is  $(0, n^{\frac{3}{4}})$ , which is smaller than  $(0, n)$  given Corollary 5.1.
B.5 Calculations in Example 5.1
We calculate the quantities  $r(\Sigma), k_0, k_1, k_2$  for the example distributions in 5.1. The results validate that  $k_{1}$  is typically a much smaller quantity than  $k_{0}$ .
1. Calculations for  $\lambda_{k} = \frac{1}{k^{\alpha}},\alpha >1$
Define  $r_k(\Sigma) = \frac{\sum_{i > k} \lambda_i}{\lambda_{k + 1}}$  as in Bartlett et al. [8]. Since  $\sum_{i > k} \frac{1}{i^{\alpha}} = \Theta \left( \frac{1}{k^{\alpha - 1}} \right)$ , we have  $r_k(\Sigma) = \Theta \left( \frac{\frac{1}{k^{\alpha - 1}}}{\frac{1}{k^{\alpha}}} \right) = \Theta(k)$ . Hence,  $k_0 = \Theta(n)^3$ , and the conditions of theorem 5.1 is satisfied.
As  $\sum_{i > 0}\lambda_i < \infty$ , By its definition we know that  $k_{1}$  is the smallest  $l$  such that  $\lambda_{l + 1} = O(\frac{1}{n})$ . Therefore,  $k_{1} = \Theta (n^{\frac{1}{\alpha}})$ .
2. Calculations for  $\lambda_{k} = \frac{1}{k\log^{\beta}(k + 1)},\beta >1.$
$\sum_{i > k}\frac{1}{i\log^{\beta}(i + 1)} = \Theta (\int_{k}^{\infty}\frac{1}{x\log^{\beta}x} dx) = \Theta (\frac{1}{\log^{\beta - 1}k})$  , which implies  $r_k(\Sigma) = k\log k$
Solving  $k_0 \log k_0 \geq \Theta(n)$ , we have  $k_0 = \Theta\left(\frac{n}{\log n}\right)$ .
3The calculations for  $k_{0},k_{1}$  and  $k_{2}$  in this section only apply when  $n$  is sufficiently large.
By the definition of  $k_{1}$ , we know that  $k_{1}$  is the smallest  $l$  such that  $l\log^{\beta}(l + 1)\geq \Theta (n)$ . Therefore,  $k_{1} = \Theta \left(\frac{n}{\log^{\beta}n}\right)$ .
3. Calculations for  $\lambda_{i} = \frac{1}{n^{1 + \varepsilon}}, 1 \leq i \leq n^{1 + \varepsilon}, \varepsilon > 0$ .
Since  $r_0(\Sigma) = n^{1 + \varepsilon}$ , we have  $k_{0} = 0$ . By the definition of  $k_{1}$ , we also have  $k_{1} = 0$ .
4. Calculations for  $\lambda = \begin{cases} \frac{1}{s} & 1 \leq k \leq s \\ \frac{1}{d - s} & s + 1 \leq k \leq d \end{cases}$ ,  $s = n^r$ ,  $d = n^q$ ,  $0 < r \leq 1$ ,  $q \geq 1$ .
For  $0 \leq k < s$ ,  $r_k(\Sigma) = \Theta\left(\frac{1}{s}\right) = \Theta(n^r) = o(n)$ , while  $r_s(\Sigma) = \frac{1}{\frac{1}{d - s}} = \Theta(n^q) = \omega(n)$ .
Therefore,  $k_0 = s = n^r$ .
Similarly, noting that  $\lambda_{k} = \frac{1}{n^{r}} = \omega (n)$  for  $0\leq k < s$  and  $\lambda_{s} = \Theta (\frac{1}{n^{d}}) = o(n)$ , we know that  $k_{1} = s = n^{r}$ .
B.6 Calculations for  $\lambda_{k} = 1 / k^{\alpha}$ ,  $\alpha > 1$  in Section 5.3
Set  $c(t,n) = \frac{1}{n^{\beta}}$ , where  $\beta > 0$  will be chosen later. First we calculate  $k_{2}$  under this choice of  $c(t,n)$ . Note that  $\sum_{i > k} \frac{1}{i^{\alpha}} = \Theta \left( \frac{1}{k^{\alpha - 1}} \right)$ . Therefore,  $k_{2}$  is the smallest  $k$  such that  $\frac{1}{k^{\alpha - 1}} + \frac{n}{k^{\alpha}} = O \left( \frac{1}{n^{\beta}} \right)$ . For the bound on  $V(\theta_t)$  to be consistent, we need  $k_{2} = o(n)$ . Hence,  $\frac{1}{k^{\alpha - 1}} = O \left( \frac{n}{k^{\alpha}} \right)$ , which implies  $k_{2} = n^{\frac{\beta + 1}{\alpha}}$ .
Plugging the value of  $c(t,n)$  and  $k_{2}$  into our bound, we have
$$
V (\theta_ {t}) = O \left(n ^ {\left(\frac {1}{\alpha} - 1\right) + \left(\frac {1}{\alpha} + 1\right) \beta} + n ^ {2 \tau - \beta - 2}\right)
$$
which attains its minimum  $\Theta \left(n^{\frac{2\alpha\tau - 3\alpha + 2\tau - 1}{2\alpha + 1}}\right)$  at  $\beta = \Theta \left(\frac{2\alpha\tau - \alpha - 1}{2\alpha + 1}\right)$ .
For  $V(\theta) = O(1)$ , we need  $\tau \leq \frac{3\alpha + 1}{2\alpha + 2}$ . For  $\beta \geq 0$ , we need  $\tau \geq \frac{\alpha + 1}{2\alpha}$ . Putting them together gives the range of  $t$  in which the above calculation applies.
B.7 Discussions on  $\mathcal{D}_n$
In this paper, the distribution  $\mathcal{D}$  is regarded as a sequence of distribution  $\{\mathcal{D}_n\}$  which may depend on sample size  $n$ . The phenomenon comes from overparameterization and asymptotic requirements. In the definition of compatibility, we require  $n \to \infty$ . In this case, overparameterization requires that the dimension  $p$  (if finite) cannot be independent of  $n$  since  $n \to \infty$  would break overparameterization. Therefore, the covariance  $\Sigma$  also has  $n$ -dependency, since  $\Sigma$  is closely related to  $p$ .
Several points are worth mentioning: (1) Similar arguments generally appear in related works, for example, in Bartlett et al. [8] when discussing the definition of benign covariance. (2) One can avoid such issues by considering an infinite dimensional feature space. This is why we discuss the special case  $p = \infty$  in Theorem 4.1. (3) If  $p$  is a fixed finite constant that does not alter with  $n$ , the problem becomes underparameterized and thus trivial to get a consistent generalization bound via standard concentration inequalities.
C.1 Details for Linear Regression Experiments
In this section, we provide the experiment details for linear regression experiments and present additional empirical results.
The linear regression experiment in Figure 1 follows the setting described in section 6. Although the final iterate does not interpolate the training data, the results suffice to demonstrate the gap between the early-stopping and final-iterate excess risk. The training plot for different covariances are given in Figure 2.
Next, we provide the experiment results of sample sizes  $n = 50$ ,  $n = 200$  and feature dimensions  $p = 500$ ,  $p = 2000$ . The settings are the same as described above, except for the sample size. The optimal excess risk and min-norm excess risk are provided in Table 3, 4, 6 and 5. The tables indicate that the two observations stated above hold for different sample size  $n$ .
(a)  $\lambda_{i} = \frac{1}{i}$
(b)  $\lambda_{i} = \frac{1}{i^{2}}$
(c)  $\lambda_{i} = \frac{1}{i^{3}}$
(d)  $\lambda_{i} = \frac{1}{i\log(i + 1)}$
(e)  $\lambda_{i} = \frac{1}{i\log^{2}(i + 1)}$
(f)  $\lambda_{i} = \frac{1}{i\log^{3}(i + 1)}$
Figure 2: The training plot for overparameterized linear regression with different covariances using GD.
Table 3: The effective dimension  $k_{1}$ , the optimal early stopping excess risk and min-norm excess risk for different feature distributions, with sample size  $n = 50$ ,  $p = 1000$ . We calculate the 95% confidence interval for the excess risk.
DISTRIBUTIONS	k1	OPTIMAL EXCESS RISK	MIN-NORM EXCESS RISK
λi=1/i	Θ(n)	2.515 ± 0.0104	12.632 ± 0.1602
λi=1/ i2	Θ(n1/2)	0.269 ± 0.0053	50.494 ± 0.9378
λi=1/i3	Θ(n1/3)	0.083 ± 0.0011	13.208 ± 0.4556
λi=1/i log(i+1)	Θ(n/log n)	0.808 ± 0.0090	46.706 ± 0.6983
λi=1/i log2(i+1)	Θ(n/log2n)	0.381 ± 0.0076	74.423 ± 1.1472
λi=1/i log3(i+1)	Θ(n/log3n)	0.233 ± 0.0052	43.045 ± 0.8347
C.2 Details for MNIST Experiments
In this section, we provide the experiment details and additional results in MNIST dataset.
The MNIST experiment details are described below. We create a noisy version of MNIST with label noise rate  $20\%$ , i.e. randomly perturbing the label with probability  $20\%$  for each training data, to simulate the label noise which is common in real datasets, e.g ImageNet [60, 65, 73]. We do not inject noise into the test data.
We choose a standard four layer convolutional neural network as the classifier. We use a vanilla SGD optimizer without momentum or weight decay. The initial learning rate is set to 0.5. Learning rate is decayed by 0.98 every epoch. Each model is trained for 300 epochs. The training batch size is set to 1024, and the test batch size is set to 1000. We choose the standard cross entropy loss as the loss function.
We provide the plot for different levels of label noise in Figure 3. We present the corresponding test error of the best early stopping iterate and the final iterate in Table 7. Since the theoretical part of this paper focuses on GD, we also provide a corresponding plot of GD training in Figure 4 for completeness.
Table 4: The effective dimension  $k_{1}$ , the optimal early stopping excess risk and min-norm excess risk for different feature distributions, with sample size  $n = 200$ ,  $p = 1000$ . We calculate the 95% confidence interval for the excess risk.
DISTRIBUTIONS	k1	OPTIMAL EXCESS RISK	MIN-NORM EXCESS RISK
λi=1/i	Θ(n)	2.173 ± 0.0065	52.364 ± 0.4009
λi=1/ i2	Θ(n1/2)	0.161 ± 0.0039	36.855 ± 0.4833
λi=1/ i3	Θ(n1/3)	0.068 ± 0.0012	8.1990 ± 0.2313
λi=1/i log(i+1)	Θ(n/log n)	0.628 ± 0.0034	152.70 ± 1.1073
λi=1/ i log2(i+1)	Θ(n/log2 n)	0.241 ± 0.0036	83.550 ± 0.7596
λi=1/ i log3(i+1)	Θ(n/log3 n)	0.146 ± 0.0108	33.469 ± 0.4540
Table 5: The effective dimension  $k_{1}$ , the optimal early stopping excess risk and min-norm excess risk for different feature distributions, with sample size  $n = 100$ ,  $p = 500$ . We calculate the 95% confidence interval for the excess risk.
DISTRIBUTIONS	k1	OPTIMAL EXCESS RISK	MIN-NORM EXCESS RISK
λi = 1/i	Θ(n)	1.997 ± 0.0876	27.360 ± 0.3019
λi = 1/ i2	Θ(n1/2)	0.211 ± 0.0050	43.531 ± 0.7025
λi = 1/ i3	Θ(n1/3)	0.076 ± 0.0011	10.062 ± 0.3022
λi = 1/i log(i+1)	Θ(n/log n)	0.645 ± 0.0056	96.465 ± 1.0594
λi = 1/i log2(i+1)	Θ(n/log2n)	0.289 ± 0.0055	83.694 ± 0.9827
λi = 1/i log3(i+1)	Θ(n/log3n)	0.181 ± 0.0045	38.090 ± 0.6378
Table 6: The effective dimension  $k_{1}$ , the optimal early stopping excess risk and min-norm excess risk for different feature distributions, with sample size  $n = 100$ ,  $p = 2000$ . We calculate the 95% confidence interval for the excess risk.
DISTRIBUTIONS	k1	OPTIMAL EXCESS RISK	MIN-NORM EXCESS RISK
λi=1/i	Θ(n)	2.662 ± 0.0066	23.111 ± 0.2278
λi=1/ i2	Θ(n1/2)	0.219 ± 0.0050	43.130 ± 0.6421
λi=1/ i3	Θ(n1/3)	0.077 ± 0.0010	10.031 ± 0.2942
λi=1/i log(i+1)	Θ(n/log n)	0.749 ± 0.0055	88.744 ± 0.9414
λi=1/ i log2(i+1)	Θ(n/log2 n)	0.312 ± 0.0057	82.859 ± 0.9394
λi=1/ i log3(i+1)	Θ(n/log3 n)	0.190 ± 0.0047	37.782 ± 0.5945
Table 7: The test error of optimal stopping iterate and final iterate on MNIST dataset with different levels of label noise. The results demonstrate that stopping iterate can have significantly better generalization performance than interpolating solutions for real datasets.
NOISE LEVEL	OPTIMAL TEST ERROR	FINAL TEST ERROR
0%	1.07%	1.13%
10%	1.75%	10.16%
20%	2.88%	19.90%
30%	2.18%	27.94%
40%	2.57%	35.15%
50%	2.71%	42.95%
(a)  $0\%$  label noise
(b)  $10\%$  label noise
(c)  $20\%$  label noise
(d)  $30\%$  label noise
Figure 3: The training plot for corrupted MNIST with different levels of label noise using SGD.  
Figure (c) is copied from Figure 1.
(e)  $40\%$  label noise
(f)  $50\%$  label noise
Figure 4: The training plot for corrupted MNIST with  $20\%$  label noise using GD.