SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\text {O u r} = \lambda \| \tilde {f} \| _ {\mathcal {H}} ^ {2} \left(1 + 2 \sqrt {\frac {\log N}{N}}\right),
$$
where the constants  $\| \tilde{f}\|_{\mathcal{H}}^2$  and  $R^2$  can be computed directed from the choice of kernel and target function. For simplicity reason, we drop the residue term  $C_1\frac{\log N}{N}$  since it is overshadowed by the other terms and the constant  $C_1$  is not optimized.
D.4 Legendre Kernel
To illustrate the bounds with another finite-rank, we choose a simple legendre kernel (LK):
$$
K (x, z) = \sum_ {k = 0} ^ {M} \lambda_ {k} P _ {k} (x) P _ {k} (z)
$$
where  $P_{k}$  is the Legendre polynomial of degree  $k$ , and  $\lambda_{k} > 0$  are the eigenvalues.
Figure 6: Test error bound improvement on tNTK. Same as Figure 2.
Eigenvalues To better compare the Legendre kernel  $K$  with the NTK, we choose  $\lambda_{k} = C \cdot (k + 1)^{-2}$  of quadratic decay such that the spectral sums are the same:  $\sum_{k=0}^{\infty} \lambda_{k} = 0.5$ . Hence we choose  $C = 0.5 / \sum_{k=1}^{\infty} k^{-2} = \frac{3}{\pi^{2}}$ .
Target function We choose a simple target function  $\tilde{f}(x) = x^2 = \frac{1}{3} P_0(x) + \frac{2}{3} P_2(x)$ . Throughout the experiment, we set the noise variance  $\sigma^2 = 0.05$ .
D.5 Test Error Computation
Ridge As before, our bound suggests that, to balance the bias and the variance with a fixed  $N$ , we can choose  $\lambda = \frac{\sigma^2}{N}$ . In Figure 7 (left), we set  $N = 50$ ,  $\lambda = 0.05 / 50$  for KRR training; (right) we set set  $\lambda = 0.05 / 50$  for varying  $N$  from 10 to 200.
Error bars In Figure 7 (right), for each value of  $N$ , we run over 10 iterations of random samples and compute the test error. The error bars are shown as the different between the upper and the lower quartiles. The median is taken as average.
Figure 7: (left): LK training; (right): the decay of test error as  $N$  varies. Same as Figure 1.
Upper bounds In Figure 8, the expression of Bach's and our upper bounds are directly computed:
where the constants  $\| \tilde{f}\|_{\mathcal{H}}^2$  and  $R^2$  can be computed directed from the choice of kernel and target function.
Figure 8: Test error bound improvement on LK. Same as Figure 2.
Lower Bound Last but not least, we need to show our lower bound is valid. To see this clearly, we need to write the bound in exact sums instead of in HKRS norm square  $\| \tilde{f}\|_{\mathcal{H}}^2$ : namely, we compute  $I$
$$
\frac {\lambda^ {2} \lambda_ {M}}{(\lambda_ {M} + \lambda) ^ {2}} \| \tilde {f} \| _ {\mathcal {H}} ^ {2} \leq I = \lambda^ {2} \sum_ {k = 1} ^ {M} \frac {\tilde {\gamma} _ {k} ^ {2}}{(\lambda_ {k} + \lambda) ^ {2}} \leq \lambda \| \tilde {f} \| _ {\mathcal {H}} ^ {2}, \tag {42}
$$
instead of using the inequality (42) in Lemma C.18; and
$$
M \frac {\lambda_ {M} ^ {2}}{(\lambda_ {M} + \lambda) ^ {2}} \leq \sum_ {k = 1} ^ {M} \frac {\lambda_ {k} ^ {2}}{(\lambda_ {k} + \lambda) ^ {2}} \leq M, \tag {43}
$$
instead of using the inequality (42) in Theorem C.20. Then we can compute our bounds as:
Our upper bound without residue  $= \lambda^2 I\left(1 + 2\sqrt{\frac{\log N}{N}}\right) + \frac{\sigma^2}{N}\sum_{k = 1}^{M}\frac{\lambda_k^2}{(\lambda_k + \lambda)^2}\left(1 + \sqrt{\frac{\log N}{N}}\right),$
Our lower bound without residue  $= \lambda^2 I\left(1 - 2\sqrt{\frac{\log N}{N}}\right) + \frac{\sigma^2}{N}\sum_{k = 1}^{M}\frac{\lambda_k^2}{(\lambda_k + \lambda)^2}\left(1 - \sqrt{\frac{\log N}{N}}\right),$
and we drop the residue terms  $C_1 \frac{\log N}{N}$  and  $C_2 \frac{\sigma^2}{N} M \frac{\log N}{N}$  by the same reason as before. From Figure 3, we can see that our bounds precisely describe the decay of the test error. Our bounds are not 'bounding' the test errors in smaller instances due to the absence of the residue terms, which increases the interval of confidence of our approximation. But for larger instances, say  $N > 100$ , all upper and lower bounds, and the averaged test error converge to the same limit.
Figure 9: Our bounds comparing to the averaged test error with varying  $N$ , over 10 iterations. Same as Figure 3.