SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathfrak {2} = \operatorname {T r} \left[ \right. \boldsymbol {U} \mathrm {d i a g} \left\{\overbrace {0 , \cdots 0 ,} ^ {\substack {k _ {3} \text {t i m e s}}} \overbrace {\left[ 1 - \left(1 - \frac {\lambda}{n} \mu_ {k _ {3} + 1}\right) ^ {t} \right] ^ {2}, \cdots , \left[ 1 - \left(1 - \frac {\lambda}{n} \mu_ {n}\right) ^ {t} \right] ^ {2}}} ^ {n - k _ {3} \text {t i m e s}} \right\} \\ \left. \boldsymbol {U} \tilde {\Lambda} ^ {- 2} \boldsymbol {U} ^ {\top} \boldsymbol {U} \tilde {\Lambda} ^ {\frac {1}{2}} \boldsymbol {W} ^ {\top} \boldsymbol {\Sigma} \boldsymbol {W} \tilde {\Lambda} ^ {\frac {1}{2}} \boldsymbol {U} ^ {\top} \right] \\ = \operatorname {T r} \left[ \operatorname {d i a g} \left\{\overbrace {0 , \cdots 0} ^ {k _ {3} \text {t i m e s}}, \overbrace {\frac {1}{\mu_ {k _ {3} + 1}} \left[ 1 - \left(1 - \frac {\lambda}{n} \mu_ {k _ {3} + 1}\right) ^ {t} \right] ^ {2}}, \dots , \frac {1}{\mu_ {n}} \left[ 1 - \left(1 - \frac {\lambda}{n} \mu_ {n}\right) ^ {t} \right] ^ {2} \right\} \right. \\ \left. \boldsymbol {W} ^ {\top} \boldsymbol {\Sigma} \boldsymbol {W} \right]. \tag {51} \\ \end{array}
$$
From Bernoulli's inequality and the definition of  $k_{3}$ , for any  $k_{3} + 1 \leq j \leq n$ , we have
$$
\frac {1}{\mu_ {k}} \left[ 1 - \left(1 - \frac {\lambda}{n} \mu_ {k}\right) ^ {t} \right] ^ {2} \leq \frac {1}{\mu_ {k}} \left(\frac {\lambda}{n} \mu_ {k} t\right) ^ {2} = \left(\frac {\lambda t}{n}\right) ^ {2} \mu_ {k} \leq c _ {3} \left(\frac {\lambda t}{n}\right) ^ {2} c (t, n) \sum_ {i > 0} \lambda_ {i}, \tag {52}
$$
$$
\begin{array}{l} \text {2} \leq c _ {3} c (t, n) \left(\frac {\lambda t}{n}\right) ^ {2} \sum_ {i > 0} \lambda_ {i} \operatorname {T r} [ \boldsymbol {W} ^ {\top} \boldsymbol {\Sigma} \boldsymbol {W} ] \\ = c _ {3} c (t, n) \left(\frac {\lambda t}{n} \sum_ {i > 0} \lambda_ {i}\right) ^ {2}. \tag {53} \\ \end{array}
$$
From the bounds for 1 and 2 given above, we know that there exists a constant  $c$  such that with probability at least  $1 - e^{-\frac{n}{c}}$ , the trace of the variance matrix  $C$  has the following upper bound
$$
\operatorname {T r} [ C ] \leq c \left(\frac {k _ {1}}{n} + \frac {k _ {2}}{c (t , n) n} + c (t, n) \left(\frac {\lambda t}{n} \sum_ {i > 0} \lambda_ {i}\right) ^ {2}\right). \tag {54}
$$
Proof of theorem 5.1. Lemma A.4, A.7 and Theorem A.1 gives the complete proof. Note that the high probability events in the proof are independent of the epoch number  $t$ , and this implies that the theorem holds uniformly for all  $t \in \mathbb{N}$ .
A.6 Proof of Compatibility Results
Corollary A.2 (Corollary 5.1 restated). Let Assumption 1, 2 and 3 hold. Fix a constant  $c(t,n)$ . Suppose  $k_{0} = O(n)$ ,  $k_{1} = o(n)$ ,  $r(\Sigma) = o(n)$ ,  $\lambda = O\left(\frac{1}{\sum_{i > 0}\lambda_i}\right)$ . Then there exists a sequence of positive constants  $\{\delta_n\}_{n\geq 0}$  which converge to 0, such that with probability at least  $1 - \delta_{n}$ , the excess risk is consistent for  $t\in \left(\omega \left(\frac{1}{\lambda}\right),o\left(\frac{n}{\lambda}\right)\right)$ , i.e.
$$
R (\boldsymbol {\theta} _ {t}) = o (1).
$$
$$
\min  _ {t} R (\boldsymbol {\theta} _ {t}) \lesssim \frac {\max  \{\sqrt {r (\boldsymbol {\Sigma})} , 1 \}}{\sqrt {n}} + \frac {\max  \{k _ {1} , 1 \}}{n}.
$$
Proof. According to Lemma A.7, with probability at least  $1 - \frac{\delta_n}{2}$ , the following inequality holds for all  $t$ :
$$
B \left(\boldsymbol {\theta} _ {t}\right) \lesssim \left(\frac {1}{\lambda t} + \max  \left\{\sqrt {\frac {r (\boldsymbol {\Sigma})}{n}}, \frac {r (\boldsymbol {\Sigma})}{n}, \sqrt {\frac {\log \left(\frac {1}{\delta_ {n}}\right)}{n}}, \frac {\log \left(\frac {1}{\delta_ {n}}\right)}{n} \right\}\right). \tag {55}
$$
If  $\delta_{n}$  is chosen such that  $\log \frac{1}{\delta_n} = o(n)$ , we have that with probability at least  $1 - \frac{\delta_n}{2}$ , we have for all  $t = \omega\left(\frac{1}{\lambda}\right)$ :
$$
B \left(\boldsymbol {\theta} _ {t}\right) = o (1), \tag {56}
$$
in the sample size  $n$ .
When  $c(t, n)$  is a constant, we have  $k_2 \leq k_1$  with high probability as given in lemma A.8. Therefore, according to Lemma A.4 and Theorem A.1, we know that if  $\log \frac{1}{\delta_n} = O(n)$ , with probability at least  $1 - \frac{\delta_n}{2}$ , the following bound holds for all  $t$ :
$$
V \left(\boldsymbol {\theta} _ {t}\right) \lesssim \log \left(\frac {1}{\delta_ {n}}\right) \left(\frac {k _ {1}}{n} + \frac {\lambda^ {2} t ^ {2}}{n ^ {2}}\right). \tag {57}
$$
Since  $k_{1} = o(n)$ ,  $t = o\left(\frac{n}{\lambda}\right)$ , we have  $\frac{k_1}{n} + \frac{\lambda^2t^2}{n^2} = o(1)$ . Therefore, there exists a mildly decaying sequence of  $\delta_{n}$  with  $\log \left(\frac{1}{\delta_n}\right)\left(\frac{k_1}{n} + \frac{\lambda^2t^2}{n^2}\right) = o(1)$ , i.e.,
$$
V \left(\boldsymbol {\theta} _ {t}\right) = o (1). \tag {58}
$$
To conclude,  $\delta_{n}$  can be chosen such that
$$
\log \left(\frac {1}{\delta_ {n}}\right) = \omega (1), \log \left(\frac {1}{\delta_ {n}}\right) = O (n), \log \left(\frac {1}{\delta_ {n}}\right) = O \left(\frac {1}{\frac {k _ {1}}{n} + \frac {\lambda^ {2} t ^ {2}}{n ^ {2}}}\right), \tag {59}
$$
and then with probability at least  $1 - \delta_{n}$ , the excess risk is consistent for all  $t\in \left(\omega \left(\frac{1}{\lambda}\right),o\left(\frac{n}{\lambda}\right)\right)$ :
$$
R \left(\boldsymbol {\theta} _ {t}\right) = B \left(\boldsymbol {\theta} _ {t}\right) + V \left(\boldsymbol {\theta} _ {t}\right) = o (1). \tag {60}
$$
This completes the proof for the first claim. The second claim follows from Equation 55 and 57 by setting  $t = \Theta \left(\frac{\sqrt{n}}{\lambda}\right)$ .
Lemma A.9 (Lemma 5.1 restated). For any fixed (i.e. independent of sample size  $n$ ) feature covariance  $\Sigma$  satisfying assumption 1, we have  $k_{1}(n) = o(n)$ .
Proof. Suppose there exists constant  $c$ , such that  $k_{1}(n) \geq cn$ . By definition of  $k_{1}$ , we know that  $\lambda_l \geq \frac{c_1 \sum_{i > 0} \lambda_i}{n}$  holds for  $1 \leq l \leq k_{1}(n)$ . Hence we have
$$
\sum_ {l = \lfloor c n 2 ^ {i + 1} \rfloor + 1} ^ {\lfloor c n 2 ^ {i + 1} \rfloor} \lambda_ {l} \gtrsim \frac {c _ {1} \sum_ {i > 0} \lambda_ {i}}{n 2 ^ {i + 1}} c n 2 ^ {i} \gtrsim \sum_ {i > 0} \lambda_ {i}. \tag {61}
$$
summing up all  $l$  leads to a contradiction since  $\sum_{i > 0} \lambda_i < \infty$ , which finishes the proof.
Theorem A.2 (Theorem 4.1 restated). Consider the overparameterized linear regression setting defined in section 4.1. Let Assumption 1,2 and 3 hold. Assume the learning rate satisfies  $\lambda = O\left(\frac{1}{\mathrm{Tr}(\Sigma)}\right)$ .
- If the covariance satisfies  $k_{0} = o(n), R_{k_{0}}(\pmb{\Sigma}) = \omega(n)$ ,  $r(\pmb{\Sigma}) = o(n)$ , it is compatible with the region  $T_{n} = \left( \omega \left( \frac{1}{\lambda} \right), \infty \right)$ .
- If the covariance satisfies  $k_{0} = O(n), k_{1} = o(n), r(\pmb{\Sigma}) = o(n)$ , it is compatible with the region  $T_{n} = \left(\omega\left(\frac{1}{\lambda}\right), o\left(\frac{n}{\lambda}\right)\right)$ .
- If the covariance does not change with  $n$ , and satisfies  $k_0 = O(n)$  and  $p = \infty$ , it is compatible with the region  $T_n = \left(\omega\left(\frac{1}{\lambda}\right), o\left(\frac{n}{\lambda}\right)\right)$ .
Proof. For the first argument, notice that (a) the bias term can still be bounded when  $t = \omega(1)$  and  $r(\Sigma) = o(n)$ , according to Lemma A.4; (b) the variance term can be bounded with  $t \to \infty$  (that is, the variance loss would increase with time  $t$ ). Therefore, the first argument directly follows Theorem 4 in Bartlett et al. [8].
The second argument follows Corollary 5.1, and the third argument follows Corollary 5.1 and Lemma 5.1. Specifically, for any  $\varepsilon >0$ , there exists  $\{\delta_n\}_{n > 0}$  and  $N$  such that for any sample size  $n > N$ , we have
$$
\Pr \left[ \left| \sup  _ {t \in T _ {n}} R (\boldsymbol {\theta} _ {t}) \right| > \varepsilon \right] \leq \delta_ {n}. \tag {62}
$$
Let  $n\to \infty$  shows that  $\sup_{t\in T_n}R(\pmb {\theta}_t)$  converges to 0 in probability, which completes the proof for the second and the third claim.
B Comparisons and Discussions
In this section, we provide additional discussions and calculations for the main results, and compare it with previous works, including benign overfitting (Section B.1), stability-based bounds (Section B.2), uniform convergence (Section B.3), and early-stopping bounds (Section B.4).
B.1 Comparisons with Benign Overfitting
We summarize the results in Bartlett et al. [8], Zou et al. [76] and our results in Table 1, and provide a detailed comparison with them below.
Comparison to Bartlett et al. [8]. In this seminal work, the authors study the excess risk of the min-norm interpolator. As discussed before, gradient descent converges to the min-norm interpolator in the overparameterized linear regression setting. One of the main results in Bartlett et al. [8] is to provide a tight bound for the variance part in excess risk as
$$
V (\hat {\boldsymbol {\theta}}) = O \left(\frac {k _ {0}}{n} + \frac {n}{R _ {k _ {0}} (\boldsymbol {\Sigma})}\right), \tag {63}
$$
where  $\hat{\pmb{\theta}} = \pmb{X}^{\top}(\pmb{X}\pmb{X}^{\top})^{-1}\pmb{Y}$  denotes the min-norm interpolator, and  $R_{k}(\Sigma) = (\sum_{i > k}\lambda_{i})^{2}/(\sum_{i > k}\lambda_{i}^{2})$  denote another type of effective rank.
By introducing the time factor, Theorem 5.1 improves over Equation (63) in at least two aspects. Firstly, Theorem 5.1 guarantees the consistency of the gradient descent dynamics for a broad range of step number  $t$ , while Bartlett et al. [8] study the limiting behavior of the dynamics of  $t \to \infty$ . Secondly, Theorem 5.1 implies that the excess risk of early stopping gradient descent solution can be much better than the min-norm interpolator. Compared to the bound in Equation (63), the bound in Corollary 5.1 (a.) replaces  $k_{0}$  with a much smaller quantity  $k_{1}$ ; and (b.) drops the second term involving  $R_{k_0}(\pmb{\Sigma})$ . Therefore, we can derive a consistent bound for an early stopping solution, even though the excess risk of limiting point (min-norm interpolator) can be  $\Omega(1)$ .
Comparison to Zou et al. [76]. Zou et al. [76] study a different setting, which focuses on the one-pass stochastic gradient descent solution of linear regression. The authors prove a bound for the excess risk as
$$
R \left(\tilde {\boldsymbol {\theta}} _ {t}\right) = O \left(\frac {k _ {1}}{n} + \frac {n \sum_ {i > k _ {1}} \lambda_ {i} ^ {2}}{\left(\sum_ {i > 0} \lambda_ {i}\right) ^ {2}}\right), \tag {64}
$$
where  $\tilde{\theta}_t$  denotes the parameter obtained using stochastic gradient descent (SGD) with constant step size at epoch  $t$ . Similar to our bound, Equation 64 also uses the effective dimension  $k_{1}$  to characterize the variance term. However, we emphasize that Zou et al. [76] derive the bound in a pretty different scenario from ours, which is one-pass SGD scenario. During the one-pass SGD training, one uses a fresh data point to perform stochastic gradient descent in each epoch, and therefore they set  $t = \Theta(n)$  by default. As a comparison, we apply the standard full-batch gradient descent, and thus the time can be more flexible. Besides, our results in Corollary 5.1 improve the bound in Equation (64) by dropping the second term. We refer to the third and fourth example in Example 5.1 for a numerical comparison of the bounds<sup>2</sup>.
B.2 Comparisons with Stability-Based Bounds
In this section, we show that Theorem 5.1 gives provably better upper bounds than the stability-based method. We cite a result from Teng et al. [67], which uses stability arguments to tackle overparameterized linear regression under similar assumptions.
Theorem B.1 (modified from Theorem 1 in Teng et al. [67]). Under the overparameterized linear regression settings, assume that  $\| \pmb {x}\| \leq 1$ ,  $|\varepsilon |\leq V$ ,  $w = \frac{\theta^{*,\top}x}{\sqrt{\theta^{*,\top}\Sigma\theta^{*}}}$  is  $\sigma_w^2$  subgaussian. Let  $B_{t} = \sup_{\tau \in [t]}\| \pmb {\theta}_{t}\|$ . the following inequality holds with probability at least  $1 - \delta$ :
$$
R \left(\boldsymbol {\theta} _ {t}\right) = \tilde {O} \left(\max  \left\{1, \boldsymbol {\theta} ^ {*}, ^ {\top} \boldsymbol {\Sigma} \boldsymbol {\theta} ^ {*} \sigma_ {w} ^ {2}, (V + B _ {t}) ^ {2} \right\} \sqrt {\frac {\log (4 / \delta)}{n}} + \frac {\| \boldsymbol {\theta} ^ {*} \| ^ {2}}{\lambda t} + \frac {\lambda t (V + B _ {t}) ^ {2}}{n}\right). \tag {65}
$$
Theorem B.1 applies the general stability-based results [19, 24] in the overparameterized linear regression setting, by replacing the bounded Lipschitz condition with the bounded domain condition. A fine-grained analysis [31] may remove the bounded Lipschitz condition, but it additionally requires
Due to the bias term in Theorem 5.1, the overall excess risk bound cannot surpass the order  $O(1 / \sqrt{n})$ , which leads to the cases that Zou et al. [76] outperforms our bound. However, we note that such differences come from the intrinsic property of GD and SGD, which may be unable to avoid in the GD regimes.
zero noise or decaying learning rate, which is different from our setting. We omit the excess risk decomposition technique adopted in Teng et al. [67] for presentation clarity.
Theorem B.1 can not directly yield the stability argument in Theorem 4.1, since obtaining a high probability bound of  $B_{t}$  requires a delicate trajectory analysis and is a non-trivial task. Therefore, data-irrelevant methods such as stability-based bounds can not be directly applied to our setting. Even if one can replace  $B_{t}$  in Equation 65 with its expectation that is easier to handle (this modification will require adding concentration-related terms, and make the bound in Equation 65 looser), we can still demonstrate that Theorem 5.1 is tighter than the corresponding stability-based analysis by providing a lower bound on  $\mathbb{E}[B_t^2]$ , which will imply a lower bound on the righthand side of Equation 65.
Theorem B.2. Let Assumption 1, 2, 3 holds. Suppose  $\lambda = O\left(\frac{1}{\sum_{i > 0}\lambda_i}\right)$ . Suppose the conditional variance of the noise  $\varepsilon |x$  is lower bounded by  $\sigma_{\varepsilon}^{2}$ . There exists constant  $c$ , such that with probability at least  $1 - ne^{-\frac{n}{c}}$ , we have for  $t = o(n)$
$$
\mathbb {E} \| \boldsymbol {\theta} _ {t} \| ^ {2} = \Omega \left(\frac {\lambda^ {2} t ^ {2}}{n} \left(\sum_ {i > k _ {0}} \lambda_ {i}\right)\right) \tag {66}
$$
First we prove the following lemma, bounding the number of large  $\mu_t$ .
Lemma B.1. Suppose  $t = o(n)$ . Let  $l$  denote the number of  $\mu_i$ , such that  $\mu_i = \Omega\left(\frac{n}{t}\right)$ . Then with probability at least  $1 - n e^{-\frac{n}{c}}$ , we have  $l = O(t)$ .
Proof. According to Lemma A.1, we know that with probability at least  $1 - n e^{-\frac{n}{c}}$ , Equation 12 holds for all  $0 \leq k \leq n - 1$ . Conditioned on this, we have
$$
\frac {n}{t} l \lesssim \sum_ {k = 1} ^ {l} \mu_ {i} \lesssim \sum_ {k = 1} ^ {l} \left(\sum_ {i \geq k} \lambda_ {i} + \lambda_ {k} n\right) \lesssim (l + n) \sum_ {i > 0} \lambda_ {i} \lesssim l + n. \tag {67}
$$
Since  $t = o(n)$ , we have  $l = O(t)$  as claimed.
We also need the result from Bartlett et al. [8], which gives a lowerbound of  $\mu_{n}$ .
Lemma B.2. (Lemma 10 in Bartlett et al. [8]) For any  $\sigma_x$ , there exists a constant  $c$ , such that with probability at least  $1 - e^{-\frac{n}{c}}$  we have,
$$
\mu_ {n} \geq c \left(\sum_ {i > k _ {0}} \lambda_ {i}\right). \tag {68}
$$
Proof. We begin with the calculation of  $\| \pmb{\theta}_t\|^2$ . By Lemma A.2, the conditional unbiasedness of noise in Assumption 2 and the noise variance lower bound, we have
$$
\begin{array}{l} \mathbb {E} \| \boldsymbol {\theta} _ {t} \| ^ {2} = \left\| \left(\boldsymbol {I} - \frac {\lambda}{n} \boldsymbol {X} ^ {\top} \boldsymbol {X}\right) ^ {t} (\boldsymbol {\theta} _ {0} - \boldsymbol {X} ^ {\dagger} \boldsymbol {Y}) + \boldsymbol {X} ^ {\dagger} \boldsymbol {Y} \right\| ^ {2} \\ = \mathbb {E} \left\| \left(\boldsymbol {I} - \left(\boldsymbol {I} - \frac {\lambda}{n} \boldsymbol {X} ^ {\top} \boldsymbol {X}\right) ^ {t}\right) \boldsymbol {X} ^ {\dagger} \left(\boldsymbol {X} \boldsymbol {\theta} ^ {*} + \varepsilon\right) \right\| ^ {2} \\ = \mathbb {E} \left\| \left(\boldsymbol {I} - \left(\boldsymbol {I} - \frac {\lambda}{n} \boldsymbol {X} ^ {\top} \boldsymbol {X}\right) ^ {t}\right) \boldsymbol {X} ^ {\dagger} \boldsymbol {X} \boldsymbol {\theta} ^ {*} \right\| ^ {2} + \mathbb {E} \left\| \left(\boldsymbol {I} - \left(\boldsymbol {I} - \frac {\lambda}{n} \boldsymbol {X} ^ {\top} \boldsymbol {X}\right) ^ {t}\right) \boldsymbol {X} ^ {\dagger} \boldsymbol {\varepsilon} \right\| ^ {2} \\ \geq \mathbb {E} \left\| \left(\boldsymbol {I} - \left(\boldsymbol {I} - \frac {\lambda}{n} \boldsymbol {X} ^ {\top} \boldsymbol {X}\right) ^ {t}\right) \boldsymbol {X} ^ {\dagger} \varepsilon \right\| ^ {2} \\ = \mathbb {E} \operatorname {T r} \left[ \left(\boldsymbol {I} - \left(\boldsymbol {I} - \frac {\lambda}{n} \boldsymbol {X} ^ {\top} \boldsymbol {X}\right) ^ {t}\right) \boldsymbol {X} ^ {\dagger} \varepsilon \varepsilon^ {\top} \boldsymbol {X} ^ {\dagger , \top} \left(\boldsymbol {I} - \left(\boldsymbol {I} - \frac {\lambda}{n} \boldsymbol {X} ^ {\top} \boldsymbol {X}\right) ^ {t}\right) \right] \\ \geq \sigma_ {\varepsilon} ^ {2} \mathbb {E} \operatorname {T r} \left[ \left(\boldsymbol {I} - \left(\boldsymbol {I} - \frac {\lambda}{n} \boldsymbol {X} ^ {\top} \boldsymbol {X}\right) ^ {t}\right) \boldsymbol {X} ^ {\dagger} \boldsymbol {X} ^ {\dagger , \top} \left(\boldsymbol {I} - \left(\boldsymbol {I} - \frac {\lambda}{n} \boldsymbol {X} ^ {\top} \boldsymbol {X}\right) ^ {t}\right) \right] \\ = \sigma_ {\varepsilon} ^ {2} \sum_ {i = 1} ^ {n} \frac {\left[ 1 - \left(1 - \frac {\lambda}{n} \mu_ {i}\right) ^ {t} \right] ^ {2}}{\mu_ {i}}. \tag {69} \\ \end{array}
$$
When  $\mu_{i} = o\left(\frac{n}{t}\right)$ , we have
$$
1 - \left(1 - \frac {\lambda}{n} \mu_ {i}\right) ^ {t} = 1 - 1 + \frac {\lambda}{n} \mu_ {i} t + O \left(\left(\frac {\lambda}{n} \mu_ {i} t\right) ^ {2}\right) = \Theta \left(\frac {\lambda}{n} \mu_ {i} t\right). \tag {70}
$$
Plugging it into Equation 69 and then use Lemma B.1, B.2, we know that under the high probability event in Lemma B.1 and B.2,
$$
\mathbb {E} \| \boldsymbol {\theta} _ {t} \| ^ {2} = \Omega \left((n - l) \frac {\lambda^ {2}}{n ^ {2}} \mu_ {n} t ^ {2}\right) = \Omega \left(\frac {\lambda^ {2}}{n} \mu_ {n} t ^ {2}\right) = \Omega \left(\frac {\lambda^ {2} t ^ {2}}{n} \left(\sum_ {i > k _ {0}} \lambda_ {i}\right)\right) \tag {71}
$$
Therefore, the stability-based bound, i.e., the right hand side of Equation 65, can be lower bounded in expectation as  $\Omega\left(\frac{\lambda^3t^3}{n^2}\sum_{i > k_0}\lambda_i\right)$ . This implies that the stability-based bound is vacuous when  $t = \Omega\left(\frac{n^{\frac{2}{3}}\left(\sum_{i > k_0}\lambda_i\right)^{-\frac{1}{3}}}{\lambda}\right)$ . Thus, stability-based methods will provably yield smaller compatibility region than  $\left(\omega\left(\frac{1}{\lambda}\right),o\left(\frac{n}{\lambda}\right)\right)$  in Theorem 4.1 when  $\sum_{i > k_0}\lambda_i$  is not very small, as demonstrated in the examples below.
Example B.1. Let Assumption 1, 2, 3 holds. Assume without loss of generality that  $\lambda = \Theta(1)$ . We have the following examples:
1. (Inverse Polynomial). If the spectrum of  $\Sigma$  satisfies
$$
\lambda_ {k} = \frac {1}{k ^ {\alpha}},
$$
for some  $\alpha > 1$ , we derive that  $k_{0} = \Theta(n)$ ,  $\sum_{i > k_{0}} \lambda_{i} = \Theta\left(\frac{1}{n^{\alpha - 1}}\right)$ . Therefore, the stability bound in Theorem B.1 is vacuous when
$$
t = \Omega \left(n ^ {\frac {\alpha + 1}{3}}\right),
$$
which is outperformed by the region in Theorem 5.1 when  $\alpha < 2$ .
2. (Inverse Log-Polynomial). If the spectrum of  $\Sigma$  satisfies
$$
\lambda_ {k} = \frac {1}{k \log^ {\beta} (k + 1)},
$$
for some  $\beta > 1$ , we derive that  $k_{0} = \Theta \left(\frac{n}{\log n}\right)$ ,  $\sum_{i > k_0} \lambda_i = \tilde{\Theta}(1)$ . Therefore, the stability bound in Theorem B.1 is vacuous when
$$
t = \tilde {\Omega} \left(n ^ {\frac {2}{3}}\right),
$$
which is outperformed by the region in Theorem 5.1.
3. (Constant). If the spectrum of  $\Sigma$  satisfies
$$
\lambda_ {k} = \frac {1}{n ^ {1 + \varepsilon}}, 1 \leq k \leq n ^ {1 + \varepsilon},
$$
for some  $\varepsilon > 0$ , we derive that  $k_{0} = 0$ ,  $\sum_{i > k_{0}} \lambda_{i} = 1$ . Therefore, the stability bound in Theorem B.1 is vacuous when
$$
t = \Omega \left(n ^ {\frac {2}{3}}\right),
$$
4. (Piecewise Constant). If the spectrum of  $\pmb{\Sigma}$  satisfies
$$
\lambda_ {k} = \left\{ \begin{array}{l l} \frac {1}{s} & 1 \leq k \leq s, \\ \frac {1}{d - s} & s + 1 \leq k \leq d, \end{array} \right.
$$
where  $s = n^r, d = n^q, 0 < r \leq 1, q \geq 1$ , we derive that  $k_0 = n^r$ ,  $\sum_{i > k_0} \lambda_i = 1$ . Therefore, the stability bound in Theorem B.1 is vacuous when
B.3 Comparisons with Uniform Convergence Bounds
We first state a standard bound on the Rademacher complexity of linear models.
Theorem B.3 (Theorem in Mohri et al. [44]). Let  $S \subseteq \{x : \| x \|_2 \leq r\}$  be a sample of size  $n$  and let  $\mathcal{H} = \{x \mapsto \langle w, x \rangle : \| w \|_2 \leq \Lambda\}$ . Then, the empirical Rademacher complexity of  $\mathcal{H}$  can be bounded as follows:
$$
\hat {\mathcal {R}} _ {S} (\mathcal {H}) \leq \sqrt {\frac {r ^ {2} \Lambda^ {2}}{n}}. \tag {72}
$$
Furthermore, Talagrand's Lemma (See Lemma 5.7 in Mohri et al. [44]) indicates that
$$
\hat {\mathcal {R}} _ {S} (l \circ \mathcal {H}) \leq L \hat {\mathcal {R}} _ {S} (\mathcal {H}) = \frac {\Theta \left(\Lambda^ {2}\right)}{\sqrt {n}}, \tag {73}
$$
where  $L = \Theta(\Lambda)$  is the Lipschitz coefficient of the square loss function  $l$  in our setting. Therefore, the Rademacher generalization bound is vacuous when  $\Lambda = \Omega(n^{\frac{1}{4}})$ . By Theorem B.2, we know that  $\mathbb{E}\|\pmb{\theta}_t\|^2 = \Omega(n^{\frac{1}{2}})$  when  $t = \Omega\left(\frac{n^{\frac{3}{4}}}{\lambda\left(\sum_{i > k_0}\lambda_i\right)^{\frac{1}{2}}}\right)$ . A similar comparison as in Example B.1 can demonstrate that uniform stability arguments will provably yield smaller compatibility regions than that in Theorem 5.1 for example distributions.
B.4 Comparison with Previous Works on Early Stopping
A line of works focuses on deriving the excess risk guarantee of linear regression or kernel regression with early stopping (stochastic) gradient descent. We refer to Section 2 for details. Here we compare our results with some most relevant works, including [37, 51, 72].
Comparison with Yao et al. [72]. Yao et al. [72] study kernel regression with early stopping gradient descent. Their approaches are different from ours in the following aspects.
Firstly, the assumptions used in the two approaches are different, due to different goals and techniques. Yao et al. [72] assume that the input feature and data noise have bounded norm (see Section 2.1 in Yao et al. [72]), while we require that the input feature is subgaussian with independent entries.
Furthermore, although Yao et al. [72] obtain a minimax bound in terms of the convergence rate, it is suboptimal in terms of compatibility region. Specifically, The results in our paper show a region like  $(0,n)$  while the techniques Yao et al. [72] can only lead to a region like  $(0,\sqrt{n})$ . See Proof of the Main Theorem in section 2 in Yao et al. [72] for details. Such differences come from different goals of the two approaches, where Yao et al. [72] focus on providing the optimal early-stopping time while we focus on providing a larger time region in which the loss is consistent.