SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In this subsection, we focus on bounding the operator norm  $\delta$  of the fluctuation matrix  $\pmb{\Delta}$ .
First, we establish some concentration results.
Lemma C.15 (Theorem 3.59 in [37]). Let  $\mathbf{A}$  be an  $n\times N$  matrix with independent isotropic sub-Gaussian columns in  $\mathbb{R}^n$  which sub-gaussian norm is bounded by a positive constant  $G > 0$ . Then for all  $t\geq 0$ , with probability at least  $1 - 2\exp (-\frac{1}{3} t^{2})$ , we have
$$
\left\| \frac {1}{N} \mathbf {A} \mathbf {A} ^ {\top} - \mathbf {I} _ {n} \right\| _ {o p} \leq \max  (a, a ^ {2}), \tag {34}
$$
where  $a \stackrel{\text{def.}}{=} C \sqrt{\frac{n}{N}} + \frac{t}{\sqrt{N}}$ , for all constant  $C \geq 12G^2$ .
Proof. Let  $a \stackrel{\text{def.}}{=} C \sqrt{\frac{n}{N}} + \frac{t}{\sqrt{N}}$  with  $C > 0$  to be determined, and  $\epsilon \stackrel{\text{def.}}{=} \max \{a, a^2\}$ . The first step to show that:
$$
\max  _ {x \in \mathcal {N}} \left| \frac {1}{N} \| \mathbf {A} ^ {\top} x \| _ {2} ^ {2} - 1 \right| \leq \epsilon
$$
for some  $\frac{1}{4}$ -net  $\mathcal{N}$  on the sphere  $\mathbb{S}^{n-1} \subset \mathbb{R}^n$ . Choose such a net  $\mathcal{N}$  with  $|\mathcal{N}| < \left(1 + \frac{2}{1/4}\right)^n = 9^n$ . Let  $\mathbf{A}_i$  be the  $i$ th column of the matrix  $\mathbf{A}$  and let  $Z_i \stackrel{\mathrm{def.}}{=} \mathbf{A}_i^\top x$  be a random variable. By definition of  $\mathbf{A}$ ,  $Z_i$  is centered with unit variance with sub-Gaussian norm upper bounded by  $G$ . Note that  $G \geq \frac{1}{\sqrt{2}} \mathbb{E}[Z_i^2]^{1/2} = \frac{1}{\sqrt{2}}$ , and the random variable  $Z_i^2 - 1$  is centered and has sub-exponential norm upper bounded by  $4G^2$ . Hence by an exponential deviation inequality, we have, for any  $x \in \mathbb{S}^{n-1}$ :
$$
\begin{array}{l} \mathbb {P} \left\{\left| \frac {1}{N} \left\| \mathbf {A} ^ {\top} x \right\| _ {2} ^ {2} - 1 \right| \geq \frac {\epsilon}{2} \right\} = \mathbb {P} \left\{\left| \frac {1}{N} \sum_ {i = 1} ^ {N} Z _ {i} ^ {2} - 1 \right| \geq \frac {\epsilon}{2} \right\} \\ \leq 2 \exp \left(- \frac {1}{2} e ^ {- 1} G ^ {- 4} \min  \{\epsilon , \epsilon^ {2} \}\right) \\ = \leq 2 \exp \left(- \frac {1}{2} e ^ {- 1} G ^ {- 4} a ^ {2}\right) \\ \leq 2 \exp \left(- \frac {1}{2} e ^ {- 1} G ^ {- 4} \left(C ^ {2} n + t ^ {2}\right)\right). \\ \end{array}
$$
Then by union bound, we have
$$
\begin{array}{l} \mathbb {P} \left\{\max  _ {x \in \mathcal {N}} \left| \frac {1}{N} \left\| \mathbf {A} ^ {\top} x \right\| _ {2} ^ {2} - 1 \right| \geq \frac {\epsilon}{2} \right\} \leq 9 ^ {n} \cdot 2 \exp \left(- \frac {1}{2} e ^ {- 1} G ^ {- 4} \left(C ^ {2} n + t ^ {2}\right)\right) \\ \leq 2 \exp \left(- \frac {1}{2} e ^ {- 1} G ^ {- 4} t ^ {2}\right), \\ \end{array}
$$
This inequality is Corollary 5.17 from [37].
for  $C \geq \sqrt{2e\log 9} G^2$ . Since  $12 > \sqrt{2e\log 9}$ , for simplicity, we assume  $C > 12G^2$ . Moreover, since  $G \geq \frac{1}{\sqrt{2}}$ , we have  $\frac{1}{2} e^{-1} G^{-4} \leq \frac{1}{3}$ , we have
$$
\mathbb {P} \left\{\max  _ {x \in \mathcal {N}} \left| \frac {1}{N} \| \mathbf {A} ^ {\top} x \| _ {2} ^ {2} - 1 \right| \geq \frac {\epsilon}{2} \right\} \leq 2 \exp \left(- \frac {1}{3} t ^ {2}\right).
$$
Then by the  $\frac{1}{4}$ -net argument, with probability at least  $1 - 2\exp \left(-\frac{1}{3} t^2\right)$ , we have
$$
\begin{array}{l} \left\| \frac {1}{N} \mathbf {A} \mathbf {A} ^ {\top} - \mathbf {I} _ {n} \right\| _ {\text {o p}} \leq \frac {4}{2} \max  _ {x \in \mathcal {N}} \left| \frac {1}{N} \| \mathbf {A} ^ {\top} x \| _ {2} ^ {2} - 1 \right| \\ \leq \epsilon = \max  \{a, a ^ {2} \}. \\ \end{array}
$$
Lemma C.16. Assume Assumption 4.1 holds and that  $N > \exp (4(12G^2)^2 (M + 1))$ . Then with a probability of at least  $1 - 2 / N$ , we have
$$
\max  \left\{\delta , \| \boldsymbol {E} _ {M} \| _ {2} \right\} \leq \sqrt {\frac {\log N}{N}}.
$$
Proof. Set  $n = M + 1$ ,  $\mathbf{A} = \left( \begin{array}{c}\Psi_{\leq M}\\ \psi_{>M}(\mathbf{X})^\top \end{array} \right)\in \mathbb{R}^{(M + 1)\times N}$ . Then
$$
\frac {1}{N} \mathbf {A} \mathbf {A} ^ {\top} - \mathbf {I} _ {n} = \left( \begin{array}{c c} \frac {1}{N} \boldsymbol {\Psi} _ {\leq M} \boldsymbol {\Psi} _ {\leq M} ^ {\top} & \boldsymbol {E} _ {M} \\ \boldsymbol {E} _ {M} ^ {\top} & \eta_ {> M} + 1 \end{array} \right) - \mathbf {I} _ {n} = \left( \begin{array}{c c} \boldsymbol {\Delta} _ {M} & \boldsymbol {E} _ {M} \\ \boldsymbol {E} _ {M} ^ {\top} & \eta_ {> M} \end{array} \right).
$$
where  $\eta_{>M} \stackrel{\mathrm{def.}}{=} \frac{1}{N} \sum_{i=1}^{N} \psi_{>M}(x_i)^2 - 1$ . On one hand, the operator norm of the above matrix bounds  $\delta$  and  $\| \pmb{E}_M \|_2$  from above:
$$
\begin{array}{l} \left\| \left( \begin{array}{c c} \boldsymbol {\Delta} _ {M} & \boldsymbol {E} _ {M} \\ \boldsymbol {E} _ {M} ^ {\top} & \eta_ {> M} \end{array} \right) \right\| _ {\mathrm {o p}} = \max  _ {\| \mathbf {u} \| _ {2} ^ {2} + v ^ {2} = 1} \left\| \left( \begin{array}{c c} \boldsymbol {\Delta} _ {M} & \boldsymbol {E} _ {M} \\ \boldsymbol {E} _ {M} ^ {\top} & \eta_ {> M} \end{array} \right) \left( \begin{array}{c} \mathbf {u} \\ v \end{array} \right) \right\| _ {2} \\ = \max  _ {\| \mathbf {u} \| _ {2} ^ {2} + v ^ {2} = 1} \left\| \left( \begin{array}{c} \boldsymbol {\Delta} _ {M} \mathbf {u} + v \boldsymbol {E} _ {M} \\ \boldsymbol {E} _ {M} ^ {\top} \mathbf {u} + \eta_ {> M} v \end{array} \right) \right\| _ {2} \\ \geq \max  _ {\| \mathbf {u} \| _ {2} ^ {2} + v ^ {2} = 1} \| \boldsymbol {\Delta} _ {M} \mathbf {u} + v \boldsymbol {E} _ {M} \| _ {2} \\ \geq \max  _ {\| \mathbf {u} \| _ {2} ^ {2} = 1, v = 0} \| \boldsymbol {\Delta} _ {M} \mathbf {u} + v \boldsymbol {E} _ {M} \| _ {2} \\ \geq \max  _ {\| \mathbf {u} \| _ {2} ^ {2} = 1} \| \boldsymbol {\Delta} _ {M} \mathbf {u} \| _ {2} = \delta , \\ \end{array}
$$
$$
\left\| \left( \begin{array}{cc}\boldsymbol{\Delta}_{M} & \boldsymbol{E}_{M}\\ \boldsymbol{E}_{M}^{\top} & \eta_{>M} \end{array} \right)\right\|_{\mathrm{op}}\geq \max_{\|\mathbf{u}\|_{2}^{2} + v^{2} = 1}\| \boldsymbol{\Delta}_{M}\mathbf{u} + v\boldsymbol{E}_{M}\|_{2}\geq \max_{\|\mathbf{u}\|_{2}^{2} = 0,|v| = 1}\| \boldsymbol{\Delta}_{M}\mathbf{u} + v\boldsymbol{E}_{M}\|_{2} = \| \boldsymbol{E}_{M}\|_{2}.
$$
On the other hand, set  $t = \frac{1}{2}\sqrt{\log N}$ ,  $C = 12G^2$ , since  $N > \exp(4C^2(M + 1))$ , we have
$$
a = C \sqrt {\frac {n}{N}} + \frac {t}{\sqrt {N}} = 1 2 G ^ {2} \sqrt {\frac {M + 1}{N}} + \frac {1}{2} \sqrt {\frac {\log N}{N}} \leq \sqrt {\frac {\log N}{N}} <   1.
$$
By Lemma C.16, then with probability of at least  $1 - 2\exp \left(-\frac{1}{3} t^2\right) = 1 - 2\exp \left(-\frac{1}{12}\right) / N > 1 - 2 / N$ , we have
$$
\left\| \left( \begin{array}{c c} \boldsymbol {\Delta} _ {M} & \boldsymbol {E} _ {M} \\ \boldsymbol {E} _ {M} ^ {\top} & \eta_ {> M} \end{array} \right) \right\| _ {\mathrm {o p}} \leq \max  \{a, a ^ {2} \} = a \leq \sqrt {\frac {\log N}{N}}.
$$
Combine the both results and we conclude the upper bounds.
In particular, as  $N \to \infty$ ,  $\delta$  vanishes almost surely. In empirical calculation, if the requirement  $N > \exp(4(12G^2)^2(M + 1))$  exponential in  $M$  is too demanding for a large integer  $M$ , we can take  $t = N^s$  for any positive number  $s \in (0, \frac{1}{2})$  instead of  $t = \frac{1}{2} \log N$ . In this way, we decrease the requirement to  $N$  polynomial in  $M$  in sacrificing the decay from  $\mathcal{O}\left(\sqrt{\frac{\log N}{N}}\right)$  to  $\mathcal{O}\left(N^{s - 1/2}\right)$ . For simplicity purpose, we do not list out the result with this decay in this paper.
C.4 Refined Test Error Analysis
We can apply the above concentration results to refine the following bounds on the finite-rank KRR test error. First of all, we realize the decay of target function coefficient comparable to the spectral decay:
Definition C.17 (Comparable Decay). Denote  $\underline{r} \stackrel{\mathrm{def.}}{=} \min_k \{|\tilde{\gamma}_k / \lambda_k|\}$  and  $\overline{r} \stackrel{\mathrm{def.}}{=} \max_k \{|\tilde{\gamma}_k / \lambda_k|\}$ .
C.4.1 Refined Bounds on Bias
Recall that Proposition C.13 bounding the bias in terms of  $\delta$  and  $\eta_{k}$ . For the former one, we can choose: for  $N > \max\{\exp(4(12G^{2})^{2}(M + 1)), 9\}$ , by Lemma C.16, with probability of at least  $1 - 2/N$ , we have  $\delta \leq \sqrt{\frac{\log N}{N}} < \sqrt{\frac{\log 9}{9}} < \frac{1}{2}$ . For the latter one, we have to control the vector  $w$ :
Lemma C.18. Let  $w = \lambda \Lambda^{-1}\tilde{\gamma} - \tilde{\gamma}_{>M}E$ . We have
$$
\left\| w \right\| _ {2} ^ {2} \leq \left(\lambda \bar {r} \sqrt {M} + | \tilde {\gamma} _ {> M} | \left\| \boldsymbol {E} \right\| _ {2}\right) ^ {2};
$$
$$
\frac {\lambda^ {2} \lambda_ {M}}{(\lambda_ {M} + \lambda) ^ {2}} \| \tilde {f} _ {\leq M} \| _ {\mathcal {H}} ^ {2} - \frac {1}{2} | \tilde {\gamma} _ {> M} | \| \tilde {f} _ {\leq M} \| _ {L _ {\rho} ^ {2}} \| \boldsymbol {E} \| _ {2} \leq \left\| \bar {\boldsymbol {\mathrm {P}}} w \right\| _ {2} ^ {2} \leq \lambda \| \tilde {f} _ {\leq M} \| _ {\mathcal {H}} ^ {2} + \frac {1}{2} | \tilde {\gamma} _ {> M} | \| \tilde {f} _ {\leq M} \| _ {L _ {\rho} ^ {2}} \| \boldsymbol {E} \| _ {2} + \tilde {\gamma} _ {> M} ^ {2} \| \boldsymbol {E} \| _ {2} ^ {2}.
$$
Proof. Since  $\lambda^2\underline{r}^2 M\leq \left\| \lambda \pmb{\Lambda}^{-1}\tilde{\pmb{\gamma}}\right\| _2^2\leq \lambda^2\overline{r}^2 M$  and  $\| \tilde{\gamma}_{>M}\pmb {E}\| _2^2 = \tilde{\gamma}_{>M}^2\| \pmb {E}\| _2^2$ , we have
$$
\left\| w \right\| _ {2} ^ {2} \leq \left(\lambda \bar {r} \sqrt {M} + | \tilde {\gamma} _ {> M} | \left\| E \right\| _ {2}\right) ^ {2}.
$$
Similarly, we can bound  $\left\| \bar{\mathbf{P}} w\right\| _2$  . Observe that:
$$
\left\| \bar {\mathbf {P}} w \right\| _ {2} ^ {2} = \underbrace {\lambda^ {2} \sum_ {k = 1} ^ {M} \frac {\tilde {\gamma} _ {k} ^ {2}}{(\lambda_ {k} + \lambda) ^ {2}}} _ {I} \underbrace {- 2 \lambda \tilde {\gamma} _ {> M} \sum_ {k = 1} ^ {M} \frac {\tilde {\gamma} _ {k} \lambda_ {k} \eta_ {k}}{(\lambda_ {k} + \lambda) ^ {2}}} _ {I I} + \underbrace {\tilde {\gamma} _ {> M} ^ {2} \sum_ {k = 1} ^ {M} \frac {\lambda_ {k} ^ {2} \eta_ {k} ^ {2}}{(\lambda_ {k} + \lambda) ^ {2}}} _ {I I I}.
$$
Since  $1 \geq \frac{\lambda}{\lambda_k + \lambda} \geq \frac{\lambda}{\lambda_M + \lambda}$ , we have the upper bound:
$$
I = \lambda^ {2} \sum_ {k = 1} ^ {M} \frac {\tilde {\gamma} _ {k} ^ {2}}{(\lambda_ {k} + \lambda) ^ {2}} \leq \lambda \sum_ {k = 1} ^ {M} \frac {\lambda}{\lambda_ {k} + \lambda} \frac {\tilde {\gamma} _ {k} ^ {2}}{\lambda_ {k} + \lambda} \leq \lambda \sum_ {k = 1} ^ {M} \frac {\tilde {\gamma} ^ {2}}{\lambda_ {k}} = \lambda \| \tilde {f} _ {\leq M} \| _ {\mathcal {H}} ^ {2}. \tag {35}
$$
where  $\tilde{f}_{\leq M} \stackrel{\mathrm{def.}}{=} \sum_{k=1}^{M} \tilde{\gamma}_k \psi_k = \tilde{f} - \tilde{\gamma}_{>M} \psi_{>M}$ . For the lower bound, we have:
$$
I = \lambda^ {2} \sum_ {k = 1} ^ {M} \frac {\tilde {\gamma} _ {k} ^ {2}}{(\lambda_ {k} + \lambda) ^ {2}} \geq \lambda^ {2} \sum_ {k = 1} ^ {M} \frac {\lambda_ {k}}{(\lambda_ {k} + \lambda) ^ {2}} \frac {\tilde {\gamma} _ {k} ^ {2}}{\lambda_ {k}} \geq \lambda^ {2} \frac {\lambda_ {M}}{(\lambda_ {M} + \lambda) ^ {2}} \| \tilde {f} _ {\leq M} \| _ {\mathcal {H}} ^ {2} \tag {36}
$$
Similarly, since  $4\lambda \lambda_{k}\leq (\lambda_{k} + \lambda)^{2}$
$$
| I I | = 2 \lambda | \tilde {\gamma} _ {> M} | \sum_ {k = 1} ^ {M} \frac {| \tilde {\gamma} _ {k} | \lambda_ {k} | \eta_ {k} |}{(\lambda_ {k} + \lambda) ^ {2}} \leq \frac {1}{2} | \tilde {\gamma} _ {> M} | \sum_ {k + 1} ^ {M} | \tilde {\gamma} _ {k} \eta_ {k} | \leq \frac {1}{2} | \tilde {\gamma} _ {> M} | \sqrt {\sum_ {k + 1} ^ {M} \tilde {\gamma} _ {k} ^ {2} \sum_ {k = 1} ^ {M} \eta_ {k} ^ {2}} \leq \frac {1}{2} | \tilde {\gamma} _ {> M} | \| \tilde {f} _ {\leq M} \| _ {L _ {\rho} ^ {2}} \| \boldsymbol {E} \| _ {2}.
$$
$$
I I I = \tilde {\gamma} _ {> M} ^ {2} \sum_ {k = 1} ^ {M} \frac {\lambda_ {k} ^ {2} \eta_ {k} ^ {2}}{(\lambda_ {k} + \lambda) ^ {2}} \leq \tilde {\gamma} _ {> M} ^ {2} \sum_ {k = 1} ^ {M} \eta_ {k} ^ {2} = \tilde {\gamma} _ {> M} ^ {2} \| \boldsymbol {E} \| _ {2} ^ {2}.
$$
Combining the above result, we state the following theorem:
Theorem C.19. For  $N > \max \left\{\exp (4(12G^2)^2 (M + 1)),9\right\}$  and for any constant  $C_1 > 8\left(\lambda \overline{r}\sqrt{M} +\frac{1}{2} |\tilde{\gamma}_{>M}|\right)^2 +\frac{5}{2}\| \tilde{f}\|_{L_\rho^2}^2$  (independent to  $N$ ), with a probability of at least  $1 - 2 / N$ , we have the upper and lower bounds of bias:
$$
b i a s \leq \tilde {\gamma} _ {> M} ^ {2} + \lambda \| \tilde {f} _ {\leq M} \| _ {\mathcal {H}} ^ {2} + \left(\frac {1}{4} \| \tilde {f} \| _ {L _ {\rho} ^ {2}} ^ {2} + 2 \lambda \| \tilde {f} _ {\leq M} \| _ {\mathcal {H}} ^ {2}\right) \sqrt {\frac {\log N}{N}} + C _ {1} \frac {\log N}{N};
$$
$$
\text {b i a s} \geq \tilde {\gamma} _ {> M} ^ {2} + \frac {\lambda^ {2} \lambda_ {M}}{(\lambda_ {M} + \lambda) ^ {2}} \| \tilde {f} _ {\leq M} \| _ {\mathcal {H}} ^ {2} - \left(\frac {1}{4} \| \tilde {f} \| _ {L _ {\rho} ^ {2}} ^ {2} + \frac {2 \lambda^ {2}}{\lambda_ {1} + \lambda} \| \tilde {f} _ {\leq M} \| _ {\mathcal {H}} ^ {2}\right) \sqrt {\frac {\log N}{N}} - C _ {1} \frac {\log N}{N}.
$$
For  $\lambda \to 0$ , we have a simpler bound: with a probability of at least  $1 - 2 / N$ , we have
$$
\lim  _ {\lambda \rightarrow 0} \text {b i a s} \leq \left(1 + \frac {\log N}{N}\right) \tilde {\gamma} _ {> M} ^ {2} + 6 \tilde {\gamma} _ {> M} ^ {2} \left(\frac {\log N}{N}\right) _ {3} ^ {\frac {3}{2}}; \tag {37}
$$
$$
\lim  _ {\lambda \to 0} b i a s \geq \left(1 - \frac {\log N}{N}\right) \tilde {\gamma} _ {> M} ^ {2} - 6 \tilde {\gamma} _ {> M} ^ {2} \left(\frac {\log N}{N}\right) ^ {\frac {3}{2}}.
$$
For  $\tilde{\gamma}_{>M}^2 = 0$ , that is  $\tilde{f} \in \mathcal{H}$ , we have a simpler upper bound on bias: with a probability of at least  $1 - 2/N$ , we have
$$
\left. \text {b i a s} \leq \lambda \| \tilde {f} \| _ {\mathcal {H}} ^ {2} \left(1 + 2 \sqrt {\frac {\log N}{N}}\right) + C _ {1} \frac {\log N}{N}; \right. \tag {38}
$$
$$
b i a s \geq \frac {\lambda^ {2} \lambda_ {M}}{(\lambda_ {M} + \lambda) ^ {2}} \| \tilde {f} \| _ {\mathcal {H}} ^ {2} \left(1 - 2 \sqrt {\frac {\log N}{N}}\right) - C _ {1} \frac {\log N}{N}.
$$
Proof. By Proposition C.13 and Lemma C.18,
$$
\text {f i t t i n g} \leq (1 + 2 \delta) \| \bar {\mathbf {P}} w \| _ {2} ^ {2} + \| w \| _ {2} ^ {2} \delta^ {2} p (\delta) \tag {39}
$$
$$
\leq (1 + 2 \delta) (\lambda \| \tilde {f} _ {\leq M} \| _ {\mathcal {H}} ^ {2} + \frac {1}{2} | \tilde {\gamma} _ {> M} | \| \tilde {f} _ {\leq M} \| _ {L _ {\rho} ^ {2}} \| \boldsymbol {E} \| _ {2} + \tilde {\gamma} _ {> M} ^ {2} \| \boldsymbol {E} \| _ {2} ^ {2}) + \| w \| _ {2} ^ {2} \delta^ {2} p (\delta) \tag {40}
$$
$$
\leq (1 + 2 \delta) (\lambda \| \tilde {f} _ {\leq M} \| _ {\mathcal {H}} ^ {2} + \frac {1}{4} \| \tilde {f} \| _ {L _ {\rho} ^ {2}} ^ {2} \| \boldsymbol {E} \| _ {2} + \tilde {\gamma} _ {> M} ^ {2} \| \boldsymbol {E} \| _ {2} ^ {2}) + \| w \| _ {2} ^ {2} \delta^ {2} p (\delta). \tag {41}
$$
where in line (39), we use Proposition C.13; in line (40), we use Lemma C.18; in line (40), we use the fact that  $2ab \leq a^2 + b^2$  where  $a = |\tilde{\gamma}_{>M}|$ ,  $b = \| \tilde{f}_{\leq M}\|_{L_\rho^2}$ .
Now we apply the concentration result in Lemma C.16: with a probability of at least  $1 - 2 / N$ :
$$
\begin{array}{l} \text {f i t t i n g e r r o r} \leq \left(1 + 2 \sqrt {\frac {\log N}{N}}\right) \lambda \| \tilde {f} _ {\leq M} \| _ {\mathcal {H}} ^ {2} + \frac {1}{4} \| \tilde {f} \| _ {L _ {\rho} ^ {2}} ^ {2} \sqrt {\frac {\log N}{N}} \\ + \frac {\log N}{N} \left(\| w \| _ {2} ^ {2} p (\delta) + (1 + 2 \delta) \tilde {\gamma} _ {> M} ^ {2} + \frac {1}{2} \| \tilde {f} \| _ {L _ {\rho} ^ {2}} ^ {2}\right) \\ \leq \lambda \| \tilde {f} _ {\leq M} \| _ {\mathcal {H}} ^ {2} + \left(\frac {1}{4} \| \tilde {f} \| _ {L _ {\rho} ^ {2}} ^ {2} + 2 \lambda \| \tilde {f} _ {\leq M} \| _ {\mathcal {H}} ^ {2}\right) \sqrt {\frac {\log N}{N}} + C _ {1} \frac {\log N}{N}, \\ \end{array}
$$
where we choose  $C_1 > 0$  to be such that:
$$
\begin{array}{l} \| w \| _ {2} ^ {2} p (\delta) + (1 + 2 \delta) \tilde {\gamma} _ {> M} ^ {2} + \frac {1}{2} \| \tilde {f} \| _ {L _ {\rho} ^ {2}} ^ {2} \leq \| w \| _ {2} ^ {2} p (\delta) + \left(1 + 2 \delta + \frac {1}{2}\right) \| \tilde {f} \| _ {L _ {\rho} ^ {2}} ^ {2} \\ \leq \| w \| _ {2} ^ {2} p \left(\frac {1}{2}\right) + \left(1 + 2 \cdot \frac {1}{2} + \frac {1}{2}\right) \| \tilde {f} \| _ {L _ {\rho} ^ {2}} ^ {2} \\ \leq 8 \left(\lambda \bar {r} \sqrt {M} + | \tilde {\gamma} _ {> M} | \| E \| _ {2}\right) ^ {2} + \frac {5}{2} \| \tilde {f} \| _ {L _ {\rho} ^ {2}} ^ {2} \\ \leq 8 \left(\lambda \bar {r} \sqrt {M} + \frac {1}{2} | \tilde {\gamma} _ {> M} |\right) ^ {2} + \frac {5}{2} \| \tilde {f} \| _ {L _ {\rho} ^ {2}} ^ {2} <   C _ {1}. \\ \end{array}
$$
Hence we have an upper bound for the bias. We argue similarly for the lower bound:
$$
\begin{array}{l} \text {f i t t i n g e r r o r} \geq \left(1 - 2 \sqrt {\frac {\log N}{N}}\right) \frac {\lambda^ {2} \lambda_ {M}}{(\lambda_ {M} + \lambda) ^ {2}} \| \tilde {f} _ {\leq M} \| _ {\mathcal {H}} ^ {2} - \frac {1}{4} \| \tilde {f} \| _ {L _ {\rho} ^ {2}} ^ {2} \sqrt {\frac {\log N}{N}} \\ - \frac {\log N}{N} \left(\| w \| _ {2} ^ {2} p (\delta) + (1 + 2 \delta) \tilde {\gamma} _ {> M} ^ {2} + | \tilde {\gamma} _ {> M} | \| \tilde {f} _ {\leq M} \| _ {L _ {\rho} ^ {2}}\right) \\ \geq \frac {\lambda^ {2} \lambda_ {M}}{(\lambda_ {M} + \lambda) ^ {2}} \| \tilde {f} _ {\leq M} \| _ {\mathcal {H}} ^ {2} - \left(\frac {1}{4} \| \tilde {f} \| _ {L _ {\rho} ^ {2}} ^ {2} + \frac {2 \lambda^ {2}}{\lambda_ {1} + \lambda} \| \tilde {f} _ {\leq M} \| _ {\mathcal {H}} ^ {2}\right) \sqrt {\frac {\log N}{N}} - C _ {1} \frac {\log N}{N}. \\ \end{array}
$$
For  $\lambda \to 0$ , note that  $w \to -\tilde{\gamma}_{>M}E$ . This yields
$$
\begin{array}{l} \lim  _ {\lambda \rightarrow 0} \text {f i t t i n g e r r o r} \leq \lim  _ {\lambda \rightarrow 0} \left\{\left(1 + 2 \delta\right)\left\| \bar {\mathbf {P}} w \right\| _ {2} ^ {2} + \| w \| _ {2} ^ {2} \delta^ {2} p (\delta) \right\} \\ = (1 + 2 \delta) \| - \tilde {\gamma} _ {> M} \boldsymbol {E} \| _ {2} ^ {2} + \| - \tilde {\gamma} _ {> M} \boldsymbol {E} \| _ {2} ^ {2} \delta^ {2} p (\delta) \\ = \tilde {\gamma} _ {> M} ^ {2} \| \boldsymbol {E} \| _ {2} ^ {2} (1 + \delta (2 + \delta p (\delta))). \\ \end{array}
$$
Hence, by plugging in  $\delta < \frac{1}{2}$ , with probability of at least  $1 - 2 / N$ ,
$$
\begin{array}{l} \lim  _ {\lambda \rightarrow 0} \text {f i t t i n g e r r o r} \leq \tilde {\gamma} _ {> M} ^ {2} \frac {\log N}{N} \left(1 + 6 \sqrt {\frac {\log N}{N}}\right) \\ \lim  _ {\lambda \rightarrow 0} \operatorname {b i a s} \leq \left(1 + \frac {\log N}{N}\right) \tilde {\gamma} _ {> M} ^ {2} + 6 \tilde {\gamma} _ {> M} ^ {2} \left(\frac {\log N}{N}\right) ^ {\frac {3}{2}}. \\ \end{array}
$$
For lower bound, it follows similarly:
$$
\lim  _ {\lambda \rightarrow 0} \operatorname {b i a s} \geq \left(1 - \frac {\log N}{N}\right) \tilde {\gamma} _ {> M} ^ {2} - 6 \tilde {\gamma} _ {> M} ^ {2} \left(\frac {\log N}{N}\right) ^ {\frac {3}{2}},
$$
and we obtain line (37). For the case where  $\tilde{\gamma}_{>M} = 0$ , recalculate and simplify line (40) to obtain line (38).
C.4.2 Refined Bounds on Variance
Similarly, we can refine Theorem C.14 to get a bound on the variance:
Theorem C.20. For  $N > \max \left\{\left(12G\right)^{4}(M + 1)^{2},9\right\}$ , and set  $C_2 = 12$  (independent to  $N$ ), with a probability of at least  $1 - 2 / N$ , we have the upper and lower bounds of variance:
$$
\text {v a r i a n c e} \leq \sigma^ {2} \frac {M}{N} \left(1 + \sqrt {\frac {\log N}{N}} + C _ {2} \frac {\log N}{N}\right);
$$
$$
\text {v a r i a n c e} \geq \frac {\lambda_ {M} ^ {2}}{(\lambda_ {M} + \lambda) ^ {2}} \sigma^ {2} \frac {M}{N} \left(1 - \sqrt {\frac {\log N}{N}}\right) - C _ {2} \sigma^ {2} \frac {M}{N} \frac {\log N}{N}.
$$
Proof. We argue analogously as in Theorem C.19: by Proposition C.14 and Lemma C.16, we have
$$
\begin{array}{l} \text {v a r i a n c e} \leq (1 + \delta) \frac {\sigma^ {2}}{N} \sum_ {k = 1} ^ {M} \frac {\lambda_ {k} ^ {2}}{(\lambda_ {k} + \lambda) ^ {2}} + M \frac {\sigma^ {2}}{N} (1 + \delta) \delta^ {2} p (\delta) \\ \leq (1 + \delta) \sigma^ {2} \frac {M}{N} + \sigma^ {2} \frac {M}{N} (1 + \delta) \delta^ {2} p (\delta) \\ \leq \left(1 + \sqrt {\frac {\log N}{N}}\right) \sigma^ {2} \frac {M}{N} + \sigma^ {2} \frac {M}{N} \frac {\log N}{N} \left(1 + \frac {1}{2}\right) p \left(\frac {1}{2}\right) \\ \leq \left(1 + \sqrt {\frac {\log N}{N}}\right) \sigma^ {2} \frac {M}{N} + 1 2 \sigma^ {2} \frac {M}{N} \frac {\log N}{N} \\ \end{array}
$$
Hence we can choose  $C_2 = 12$ . For the lower bound, since  $\frac{\lambda_k^2}{(\lambda_k + \lambda)^2} > \frac{\lambda_M^2}{(\lambda_M + \lambda)^2}$ , we have
$$
\begin{array}{l} \text {v a r i a n c e} \geq (1 - \delta) \frac {\sigma^ {2}}{N} \sum_ {k = 1} ^ {M} \frac {\lambda_ {k} ^ {2}}{(\lambda_ {k} + \lambda) ^ {2}} - M \frac {\sigma^ {2}}{N} (1 + \delta) \delta^ {2} p (\delta) \\ \geq \frac {\lambda_ {M} ^ {2}}{(\lambda_ {M} + \lambda) ^ {2}} \sigma^ {2} \frac {M}{N} \left(1 - \sqrt {\frac {\log N}{N}}\right) - 1 2 \sigma^ {2} \frac {M}{N} \frac {\log N}{N}. \\ \end{array}
$$
Note that in both Theorems C.19 and C.20, the constants  $C_1, C_2 > 0$  is not optimized.
D Numerical Validation
In this section, we illustrate our result for KRR with two different finite rank kernels.
D.1 Truncated NTK
First, we need to define a finite-rank kernel  $K: \mathcal{X} \times \mathcal{X} \to \mathbb{R}$ . We set  $\mathcal{X} = \mathbb{S}^1 \subset \mathbb{R}^2$ . By reparametrization, we write  $\mathbb{S}^1 \cong [0, 2\pi] / 0 \sim 2\pi$ . We assume the data are drawn uniformly on the circle, that is  $\rho_{\mathcal{X}} = \mathrm{unif}[\mathbb{S}^1]$ . We can use the Fourier functions  $\cos(k\cdot), \sin(k\cdot)$  as the orthogonal eigenfunctions of the kernel. We define the NTK
$$
K ^ {(\infty)} (\theta , \theta^ {\prime}) \stackrel {{\text {d e f .}}} {{=}} \frac {\cos (\theta - \theta^ {\prime}) (\pi - | \theta - \theta^ {\prime} |)}{2 \pi}
$$
for all  $\theta, \theta' \in [0, 2\pi]$ . 3) We choose a rank- $M$  truncation  $K(\theta, \theta') = \sum_{k=1}^{M} \lambda_k \psi_k(\theta) \psi_k(\theta')$  for all  $\theta, \theta' \in [0, 2\pi]$ . For the first few eigenvalues of the kernel, please see Table 3 for example. Before
k	1	2	3	4	5	6	7	∞
λk	1/π2	1/8	1/8	5/9π2	5/9π2	17/225π2	17/225π2	-
ψk(θ)	1	√2 cos(θ)	√2 sin(θ)	√2 cos(2θ)	√2 sin(2θ)	√2 cos(4θ)	√2 sin(4θ)	-
∑k'=0k' λk'	0.1013	0.2263	0.3513	0.4076	0.4639	0.4716	0.4792	0.5
Table 3: The first few eigenvalues of the NTK
proceeding to test error computation, we present a training example, Figure 4, to give readers an intuition on the truncated NTK (tNTK).
Figure 4: (left): NTK training; (right): tNTK training where  $N = 50$ ,  $M = 7$ .  $\sigma^2 = 0.05$ ,  $\lambda = \sigma^2 / N$ .
D.2 Test Error Computations
In the following tNTK training, we set the hyperparameters as follows:
Target function We choose a simple target function  $\tilde{f}(x) = \cos x = \frac{1}{\sqrt{2}} \psi_2(x)$ . Throughout the experiment, we set the noise variance  $\sigma^2 = 0.05$ .
Ridge We choose  $\lambda = \frac{\sigma^2}{N}$ . In Figure 5 (left), we set  $N = 50$ ,  $\lambda = 0.05 / 50$  for tNTK training; (right) we set set  $\lambda = 0.05 / 50$  for varying  $N$  from 10 to 200.
Error bars In Figure 7 (right), for each value of  $N$ , we run over 10 iterations of random samples and compute the test error. The error bars are shown as the difference between the upper and the lower quartiles.
Figure 5: (left): tNTK training; (right): the decay of test error as  $N$  varies.
Lower bound See the subsection below.
D.3 Bound Comparison
We continue with the experiment on the tNTK this time with varying  $N$  and compare our upper bound with [6].
Upper bounds In Figure 6, the expression of Bach's and our upper bounds are directly computed:
$$
\text {B a c h ' s u p p e r b o u n d} = 4 \lambda \| \tilde {f} \| _ {\mathcal {H}} ^ {2} + \frac {8 \sigma^ {2} R ^ {2}}{\lambda N} (1 + 2 \log N)
$$