SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \operatorname {C o v} \left(\partial L _ {\text {o l d}}\right) - \operatorname {C o v} \left(\partial L _ {\text {n e w}}\right) = N \operatorname {C o v} \left(\mathbf {K} _ {\boldsymbol {x x}} \varepsilon\right) - 2 N \operatorname {C o v} \left(\mathbf {K} _ {\boldsymbol {x x}} \mathbf {K} _ {\boldsymbol {x x}} \boldsymbol {\alpha}, \mathbf {K} _ {\boldsymbol {x x}} \varepsilon\right) (54) \\ = N \mathbf {K} _ {\boldsymbol {x x}} \boldsymbol {\Sigma} \mathbf {K} _ {\boldsymbol {x x}} - 2 N \mathbf {K} _ {\boldsymbol {x x}} ^ {2} \left(\mathbf {K} _ {\boldsymbol {x x}} + \boldsymbol {\Sigma}\right) ^ {- 1} \boldsymbol {\Sigma} \mathbf {K} _ {\boldsymbol {x x}}. (55) \\ \end{array}
$$
Letting  $\boldsymbol{\Sigma} = \sigma^2\mathbf{I}$  and  $\mathbf{U}\boldsymbol{\Lambda}\mathbf{U}^T = \mathbf{K}_{xx}$  be the eigendecomposition of the kernel matrix, we have
$$
\begin{array}{l} N \mathbf {K} _ {\boldsymbol {x x}} \boldsymbol {\Sigma} \mathbf {K} _ {\boldsymbol {x x}} - 2 N \mathbf {K} _ {\boldsymbol {x x}} ^ {2} \left(\mathbf {K} _ {\boldsymbol {x x}} + \boldsymbol {\Sigma}\right) ^ {- 1} \boldsymbol {\Sigma} \mathbf {K} _ {\boldsymbol {x x}} (56) \\ = N \mathbf {U} \boldsymbol {\Lambda} ^ {2} \left(\sigma^ {2} \mathbf {I}\right) \left(\mathbf {I} - 2 \boldsymbol {\Lambda} \left(\boldsymbol {\Lambda} + \sigma^ {2} \mathbf {I}\right) ^ {- 1}\right) \mathbf {U} ^ {T}. (57) \\ \end{array}
$$
Thus, at convergence, whether our proposed objective reduces variance rests on whether or not the matrix  $\mathbf{I} - 2\boldsymbol{\Lambda}(\boldsymbol{\Lambda} + \sigma^2\mathbf{I})^{-1}$  is positive definite.
Following Section 3.4, in practice, the representative weights only converge along the top few principal directions. For these, the corresponding eigenvalues will likely be much larger than the noise and thus  $1 - 2\frac{\lambda}{\lambda + \sigma^2} < 0$ . However, in the vast majority of directions, the opposite is true: this yields lower variance to the minibatch estimator of our proposed objective, which is shown in Figure 2.
E The Implicit Bias of SGD GPs
This section provides our main theoretical results.
E.1 Convergence of Stochastic Gradient Descent
Before studying what happens in Gaussian processes, we first prove a fairly standard result on the convergence of SGD for a quadratic objective appearing in kernel ridge regression, which represents the posterior mean. For simplicity, we do not analyze Nesterov momentum, nor aspects such as gradient clipping. We will take the Gaussian observation noise covariance to be a constant multiple of the identity in this subsection. Recall that a random variable  $z$  is called  $G$ -sub-Gaussian if  $\mathbb{E}(\exp (\lambda (z - \mathbb{E}(z)))) \leq \exp (\frac{1}{2} G\lambda^2)$  holds for all  $\lambda$ . Then, a random vector is called  $G$ -sub-Gaussian if its dot product with any unit vector is  $G$ -sub-Gaussian. One can show this condition is equivalent to having tails that are no heavier than those of a Gaussian random vector, formulated appropriately. Let  $\mathbf{U}\pmb{\Lambda}\mathbf{U}^{T} = \mathbf{K}_{\pmb{x}\pmb{x}}$  be the eigenvalue decomposition of the kernel matrix. The eigenvectors  $\pmb{u}_i$  and eigenvalues  $\lambda_{i}$  are given in descending order:  $\lambda_1 \geq .\geq \lambda_N > 0$ .
Lemma 2. Let  $\delta >0$  and  $\pmb {\Sigma} = \sigma^2\mathbf{I}$  for  $\sigma^2 >0$  .Let  $\eta$  be a sufficiently small learning rate of  $0 <   \eta <  \frac{\sigma^2}{\lambda_1(\lambda_1 + \sigma^2)}$  . Let  $\alpha^{*}\in \mathbb{R}^{N}$  be the solution of the respective linear system, and let  $\overline{\alpha}_t$  be the Polyak-averaged SGD iterate after t steps, starting from an initial condition of  $\alpha_0 = 0$  . Assume that the stochastic estimate of the gradient is unbiased and  $G$  -sub-Gaussian. Then, with probability  $1 - \delta$  we have for any  $\mathcal{L}\subseteq \{1,\dots ,N\}$  and all  $i\in \mathcal{L}$  that
$$
\left| \boldsymbol {u} _ {i} ^ {T} \left(\boldsymbol {\alpha} ^ {*} - \overline {{\boldsymbol {\alpha}}} _ {t}\right) \right| \leq \frac {1}{\lambda_ {i}} \left(\frac {\| \boldsymbol {y} \| _ {2}}{\eta \sigma^ {2} t} + G \sqrt {\frac {2}{t} \log \frac {| \mathcal {I} |}{\delta}}\right). \tag {58}
$$
Proof. Consider the objective
$$
L (\boldsymbol {\alpha}) = \frac {1}{2 \sigma^ {2}} \sum_ {i = 1} ^ {N} \left(y _ {i} - \mathbf {K} _ {x _ {i}, \boldsymbol {x}} \boldsymbol {\alpha}\right) ^ {2} + \frac {1}{2} \boldsymbol {\alpha} ^ {T} \mathbf {K} _ {\boldsymbol {x x}} \boldsymbol {\alpha} \tag {59}
$$
and respective gradient
$$
\frac {\partial L}{\partial \boldsymbol {\alpha}} (\boldsymbol {\alpha}) = \frac {1}{\sigma^ {2}} \left(\mathbf {K} _ {x x} ^ {2} \boldsymbol {\alpha} - \mathbf {K} _ {x x} \boldsymbol {y}\right) + \mathbf {K} _ {x x} \boldsymbol {\alpha} = \frac {1}{\sigma^ {2}} \left(\mathbf {K} _ {x x} \left(\mathbf {K} _ {x x} + \sigma^ {2} \mathbf {I}\right) \boldsymbol {\alpha} - \mathbf {K} _ {x x} \boldsymbol {y}\right). \tag {60}
$$
Let us first look at non-stochastic gradient optimization of  $L$ , without Polyak averaging. The iterate  $\alpha_{t}$  is given by
$$
\boldsymbol {\alpha} _ {t} = \boldsymbol {\alpha} _ {t - 1} - \eta \frac {\partial L}{\partial \boldsymbol {\alpha}} \left(\boldsymbol {\alpha} _ {t - 1}\right) = \left(\mathbf {I} - \frac {\eta}{\sigma^ {2}} \mathbf {K} _ {\boldsymbol {x x}} \left(\mathbf {K} _ {\boldsymbol {x x}} + \sigma^ {2} \mathbf {I}\right)\right) \boldsymbol {\alpha} _ {t - 1} + \frac {\eta}{\sigma^ {2}} \mathbf {K} _ {\boldsymbol {x x}} \boldsymbol {y}. \tag {61}
$$
Writing  $\mathbf{M} = \left(\mathbf{I} - \frac{\eta}{\sigma^2}\mathbf{K}_{xx}(\mathbf{K}_{xx} + \sigma^2\mathbf{I})\right)$ , and recalling that  $\alpha_0 = 0$ , we thus have that
$$
\boldsymbol {\alpha} _ {t} = \frac {\eta}{\sigma^ {2}} \sum_ {j = 0} ^ {t - 1} \mathbf {M} ^ {j} \mathbf {K} _ {x x} \boldsymbol {y} = \frac {\eta}{\sigma^ {2}} (\mathbf {I} - \mathbf {M}) ^ {- 1} (\mathbf {I} - \mathbf {M} ^ {t}) \mathbf {K} _ {x x} \boldsymbol {y} = \boldsymbol {\alpha} ^ {*} - \left(\mathbf {K} _ {x x} + \sigma^ {2} \mathbf {I}\right) ^ {- 1} \mathbf {M} ^ {t} \boldsymbol {y} \tag {62}
$$
where, for our choice of learning rate, (i) by direct calculation using simultaneous diagonalizability of  $\mathbf{M}$  and  $\mathbf{K}_{xx}$ , the matrix  $\mathbf{M}$  has eigenvalues whose absolute values are strictly less than 1, (ii) thus, the geometric series converges, (iii) and  $\mathbf{I} - \mathbf{M}$  is positive definite. Examining the error in direction  $\pmb{u}_i$ , we use simultaneous diagonalizability to see that
$$
\begin{array}{l} \left| \boldsymbol {u} _ {i} ^ {T} \left(\boldsymbol {\alpha} ^ {*} - \boldsymbol {\alpha} _ {t}\right) \right| = \left| \boldsymbol {u} _ {i} ^ {T} \left(\mathbf {K} _ {\boldsymbol {x x}} + \sigma^ {2} \mathbf {I}\right) ^ {- 1} \mathbf {M} ^ {t} \boldsymbol {y} \right| = \frac {\left(1 - \frac {\eta \lambda_ {i}}{\sigma^ {2}} \left(\lambda_ {i} + \sigma^ {2}\right)\right) ^ {t}}{\lambda_ {i} + \sigma^ {2}} \left| \boldsymbol {u} _ {i} ^ {T} \boldsymbol {y} \right| (63) \\ \leq \frac {\left(1 - \frac {\eta \lambda_ {i}}{\sigma^ {2}} \left(\lambda_ {i} + \sigma^ {2}\right)\right) ^ {t}}{\lambda_ {i} + \sigma^ {2}} \| \boldsymbol {y} \| _ {2} (64) \\ \end{array}
$$
which applies to ordinary gradient descent without stochastic gradients or Polyak averaging. Next, we consider stochastic gradient optimization. We will first consider the case where the gradient is independently perturbed by  $\zeta_t \sim \mathrm{N}(\mathbf{0},\mathbf{I})$  for each step  $t > 0$ , and then relax this to sub-Gaussian noise in the sequel. Specifically, we consider
$$
\boldsymbol {\alpha} _ {t} ^ {\prime} = \boldsymbol {\alpha} _ {t - 1} ^ {\prime} - \eta \left(\frac {\partial L}{\partial \boldsymbol {\alpha}} \left(\boldsymbol {\alpha} _ {t - 1} ^ {\prime}\right) + \boldsymbol {\zeta} _ {t}\right) = \boldsymbol {\alpha} _ {t} - \eta \sum_ {j = 0} ^ {t - 1} \mathbf {M} ^ {j} \boldsymbol {\zeta} _ {t - j} \tag {65}
$$
with  $\alpha_0' = \alpha_0 = 0$ . For these iterates, consider the respective Polyak-averaged iterates denoted by  $\overline{\alpha}_t = \frac{1}{t}\sum_{j=1}^t \alpha_j'$ . We have
$$
\left| \boldsymbol {u} _ {i} ^ {T} \left(\boldsymbol {\alpha} ^ {*} - \overline {{\boldsymbol {\alpha}}} _ {t}\right) \right| = \left| \frac {1}{t} \sum_ {j = 1} ^ {t} \boldsymbol {u} _ {i} ^ {T} \left(\boldsymbol {\alpha} ^ {*} - \boldsymbol {\alpha} _ {j} ^ {\prime}\right) \right| \leq \underbrace {\left| \frac {1}{t} \sum_ {j = 1} ^ {t} \boldsymbol {u} _ {i} ^ {T} \left(\boldsymbol {\alpha} ^ {*} - \boldsymbol {\alpha} _ {j}\right) \right|} _ {A _ {i, t}} + \underbrace {\left| \frac {1}{t} \sum_ {j = 1} ^ {t} \boldsymbol {u} _ {i} ^ {T} \left(\boldsymbol {\alpha} _ {j} - \boldsymbol {\alpha} _ {j} ^ {\prime}\right) \right|} _ {B _ {i, t}} \tag {66}
$$
by expanding the Polyak averages and applying the triangle inequality, and where we have introduced the notation  $A_{i,t}$  and  $B_{i,t}$ . Using (63), the triangle inequality and another geometric series, we can bound the first sum as
$$
\left| A _ {i, t} \right| = \frac {\left\| \boldsymbol {y} \right\| _ {2}}{\left(\lambda_ {i} + \sigma^ {2}\right) t} \sum_ {j = 1} ^ {t} \left(1 - \frac {\eta \lambda_ {i}}{\sigma^ {2}} \left(\lambda_ {i} + \sigma^ {2}\right)\right) ^ {j} \leq \frac {\sigma^ {2} \left\| \boldsymbol {y} \right\| _ {2}}{\eta \lambda_ {i} \left(\lambda_ {i} + \sigma^ {2}\right) ^ {2} t} \tag {67}
$$
For the second sum, note that we can re-index the order of summation to to count each  $\zeta_{j}$  only once, rather than once for each Polyak average. This gives
$$
\begin{array}{l} B _ {i, t} = \frac {\eta}{t} \sum_ {j = 1} ^ {t} \sum_ {q = 0} ^ {j - 1} \boldsymbol {u} _ {i} ^ {T} \mathbf {M} ^ {q} \boldsymbol {\zeta} _ {j - q} = \frac {\eta}{t} \sum_ {j = 1} ^ {t} \left(\sum_ {q = 0} ^ {t - j} \boldsymbol {u} _ {i} ^ {T} \mathbf {M} ^ {q}\right) \boldsymbol {\zeta} _ {j} (68) \\ = \frac {\eta}{t} \sum_ {j = 1} ^ {t} \sum_ {q = 0} ^ {t - j} \left(1 - \frac {\eta \lambda_ {i}}{\sigma^ {2}} \left(\lambda_ {i} + \sigma^ {2}\right)\right) ^ {q} \boldsymbol {u} _ {i} ^ {T} \boldsymbol {\zeta} _ {j} (69) \\ \end{array}
$$
where  $\pmb{u}_i^T\pmb{\zeta}_j \sim \mathrm{N}(0,1)$  are IID. The variance of  $B_i$  is bounded using another geometric series
$$
\begin{array}{l} \operatorname {V a r} \left(B _ {i, t}\right) = \operatorname {V a r} \left(\frac {\eta}{t} \sum_ {j = 1} ^ {t} \sum_ {q = 0} ^ {t - j} \left(1 - \frac {\eta \lambda_ {i}}{\sigma^ {2}} \left(\lambda_ {i} + \sigma^ {2}\right)\right) ^ {q} \boldsymbol {u} _ {i} ^ {T} \boldsymbol {\zeta} _ {j}\right) (70) \\ = \frac {\eta^ {2}}{t ^ {2}} \sum_ {j = 1} ^ {t} \operatorname {V a r} \left(\sum_ {q = 0} ^ {t - j} \left(1 - \frac {\eta \lambda_ {i}}{\sigma^ {2}} \left(\lambda_ {i} + \sigma^ {2}\right)\right) ^ {q} \boldsymbol {u} _ {i} ^ {T} \boldsymbol {\zeta} _ {j}\right) (71) \\ = \frac {\eta^ {2}}{t ^ {2}} \sum_ {j = 1} ^ {t} \left[ \left(\sum_ {q = 0} ^ {t - j} \left(1 - \frac {\eta \lambda_ {i}}{\sigma^ {2}} \left(\lambda_ {i} + \sigma^ {2}\right)\right) ^ {q}\right) ^ {2} \underbrace {\operatorname {V a r} \left(\boldsymbol {u} _ {i} ^ {T} \boldsymbol {\zeta} _ {j}\right)} _ {= 1} \right] (72) \\ \leq \frac {\eta^ {2}}{t ^ {2}} \sum_ {j = 1} ^ {t} \left(\frac {\sigma^ {2}}{\eta \lambda_ {i} (\lambda_ {i} + \sigma^ {2})}\right) ^ {2} = \frac {\sigma^ {4}}{\lambda_ {i} ^ {2} (\lambda_ {i} + \sigma^ {2}) ^ {2} t}. (73) \\ \end{array}
$$
Thus, from standard tail inequalities for Gaussian random variables, for any fixed  $\delta' > 0$  we have
$$
\mathbb {P} \left(| B _ {i, t} | \leq \sqrt {\frac {2 \sigma^ {4}}{\lambda_ {i} ^ {2} (\lambda_ {i} + \sigma^ {2}) ^ {2} t} \log \frac {1}{\delta^ {\prime}}}\right) \geq 1 - \delta^ {\prime}. \tag {74}
$$
We then take  $\delta' = \delta / |\mathcal{I}|$ , and apply the union bound for all indices in  $\mathcal{I}$ . Finally, by comparing moment generating functions, we can relax the Gaussian assumption to  $G$ -sub-Gaussian random variables. To complete the claim, we combine the bounds for the two sums by writing
$$
\begin{array}{l} \left| \boldsymbol {u} _ {i} ^ {T} \left(\boldsymbol {\alpha} ^ {*} - \overline {{\boldsymbol {\alpha}}} _ {t}\right) \right| \leq \left| A _ {i, t} \right| + \left| B _ {i, t} \right| \leq \frac {\sigma^ {2} \| \boldsymbol {y} \| _ {2}}{\eta \lambda_ {i} \left(\lambda_ {i} + \sigma^ {2}\right) ^ {2} t} + G \sqrt {\frac {2 \sigma^ {4}}{\lambda_ {i} ^ {2} \left(\lambda_ {i} + \sigma^ {2}\right) ^ {2} t} \log \frac {| \mathcal {I} |}{\delta}} (75) \\ = \frac {\sigma^ {2}}{\lambda_ {i} \left(\lambda_ {i} + \sigma^ {2}\right)} \left(\frac {\| \boldsymbol {y} \| _ {2}}{\eta \left(\lambda_ {i} + \sigma^ {2}\right) t} + G \sqrt {\frac {2}{t} \log \frac {| \mathcal {I} |}{\delta}}\right) (76) \\ \leq \frac {1}{\lambda_ {i}} \left(\frac {\| \boldsymbol {y} \| _ {2}}{\eta \sigma^ {2} t} + G \sqrt {\frac {2}{t} \log \frac {| \mathcal {I} |}{\delta}}\right) (77) \\ \end{array}
$$
which gives the claim.
E.2 Convergence of SGD in RKHS Subspaces
The idea is to use the preceding result to show that SGD converges fast with respect to a certain seminorm. Let  $k$  be the kernel and let  $H_{k}$  be its associated reproducing kernel Hilbert space. We now define the spectral basis functions from Section 3.4.
Definition 3. Let  $\mathbf{K}_{xx} = \mathbf{U}\boldsymbol{\Lambda}\mathbf{U}^T$  be the respective eigendecomposition. Define the SPECTRAL BASIS FUNCTIONS
$$
u ^ {(i)} (\cdot) = \sum_ {j = 1} ^ {N} \frac {U _ {j i}}{\sqrt {\lambda_ {i}}} k (x _ {j}, \cdot). \tag {78}
$$
These are precisely the basis functions which arise in kernel principal component analysis. We also introduce two subspaces of the RKHS: the span of the representative weights, and the span of a subset of spectral basis functions. These are defined as follows.
Definition 4. Define the REPRESENTER WEIGHT SPACE
$$
R _ {k, \boldsymbol {x}} = \operatorname {s p a n} \left\{k \left(x _ {i}, \cdot\right): i = 1,.., N \right\} \subseteq H _ {k} \tag {79}
$$
equipped with the subspace inner product.
Definition 5. Let  $\mathcal{I} \subseteq \{1,..,N\}$  be an arbitrary set of indices. Define the INTERPOLATION SUBSPACE
$$
R _ {k, \boldsymbol {x}} ^ {\mathcal {I}} = \operatorname {s p a n} \left\{u ^ {(i)}: i \in \mathcal {I} \right\} \subseteq R _ {k, \boldsymbol {x}}. \tag {80}
$$
We can use these subspaces to define a seminorm on the RKHS, which we will use to measure convergence rates of S $\overline{\mathsf{G}}$ D.
Definition 6. Define the INTERPOLATION SEMINORM
$$
\left| f \right| _ {R _ {k, x} ^ {\mathcal {T}}} = \left\| \operatorname {p r o j} _ {R _ {k, x} ^ {\mathcal {T}}} f \right\| _ {H _ {k}}. \tag {81}
$$
Here,  $\mathrm{proj}_{\mathcal{S}}f$  denotes orthogonal projection of a function  $f$  onto the subspace  $\mathcal{S}$  of a Hilbert space: to ease notation, in cases where we project onto the span of a single vector, we write the vector in place of  $\mathcal{S}$ . The first order of business is to understand the relationship between the spectral basis functions and representer weight space.
Lemma 7. The functions  $u^{(i)}$ , for  $i = 1,..,N$ , form an orthonormal basis of  $R_{k,\pmb{x}}$
Proof. Let  $i \neq j$ . Using the reproducing property, definition of an eigenvector, the assumption that  $\mathbf{u}$  is an eigenvector of  $\mathbf{K}_{xx}$ , and orthonormality of eigenvectors, we have
$$
\langle u ^ {(i)}, u ^ {(j)} \rangle_ {H _ {k}} = \frac {\boldsymbol {u} _ {i} ^ {T} \mathbf {K} _ {\boldsymbol {x x}} \boldsymbol {u} _ {j}}{\sqrt {\lambda_ {i} \lambda_ {j}}} = \frac {\lambda_ {i}}{\sqrt {\lambda_ {i} \lambda_ {j}}} \boldsymbol {u} _ {i} ^ {T} \boldsymbol {u} _ {j} = 0. \tag {82}
$$
The fact that  $u^{(i)}$  form a basis follows from the fact that they are linearly independent, that there are  $N$  of them in total, and that by definition this equals the dimension of  $R_{k,\pmb{x}}$ . Finally, we calculate the norm:
$$
\left\| u ^ {(i)} \right\| _ {H _ {k}} ^ {2} = \frac {\boldsymbol {u} _ {i} \mathbf {K} _ {\boldsymbol {x x}} \boldsymbol {u} _ {i} ^ {T}}{\lambda_ {i}} = 1. \tag {83}
$$
Next, we compute a change-of-basis formula between the canonical basis functions and spectral basis functions.
Lemma 8. For some vector  $\pmb{\theta} \in \mathbb{R}^N$ , let
$$
\theta (\cdot) = \sum_ {i = 1} ^ {N} \theta_ {i} k \left(x _ {i}, \cdot\right) = \sum_ {j = 1} ^ {N} w _ {j} u ^ {(j)} (\cdot). \tag {84}
$$
$$
\boldsymbol {w} = \boldsymbol {\Lambda} ^ {1 / 2} \mathbf {U} ^ {T} \boldsymbol {\theta} \quad \| \theta \| _ {H _ {k}} ^ {2} = \boldsymbol {w} ^ {T} \boldsymbol {w}. \tag {85}
$$
Proof. By expanding  $u^{(j)}$ , we have
$$
\theta (\cdot) = \sum_ {j = 1} ^ {N} w _ {j} u ^ {(j)} (\cdot) = \sum_ {i = 1} ^ {N} \sum_ {j = 1} ^ {N} w _ {j} \frac {U _ {i j}}{\sqrt {\lambda_ {j}}} k (x _ {i}, \cdot) \tag {86}
$$
which since  $k(x_{i},\cdot)$  is a basis means  $\theta_{i} = \sum_{j = 1}^{N}w_{j}\frac{U_{ij}}{\sqrt{\lambda_{j}}}$ , or in matrix-vector notation  $\pmb{\theta} = \mathbf{U}\pmb{\Lambda}^{-1 / 2}\pmb{w}$ . The final claim about norms follows from orthonormality of  $u^{(j)}$  by writing
$$
\left\| \theta \right\| _ {H _ {k}} ^ {2} = \langle \theta , \theta \rangle_ {H _ {k}} = \sum_ {i = 1} ^ {N} \sum_ {j = 1} ^ {N} w _ {i} w _ {j} \langle u ^ {(i)}, u ^ {(j)} \rangle_ {H _ {k}} = \sum_ {i = 1} ^ {N} w _ {i} ^ {2} = \boldsymbol {w} ^ {T} \boldsymbol {w}. \tag {87}
$$
This allows us to get an explicit expression for the previously-introduced seminorm.
Lemma 9. For  $\theta \in R_{k,\mathbf{x}}$ , letting  $\Lambda_{\mathcal{I}} = \mathrm{diag}(\lambda_1\mathbb{1}_{1\in \mathcal{I}},\dots ,\lambda_i\mathbb{1}_{i\in \mathcal{I}},\dots ,\lambda_N\mathbb{1}_{N\in \mathcal{I}})$ , we have
$$
\left| \theta \right| _ {R _ {k, x} ^ {\mathcal {I}}} ^ {2} = \boldsymbol {\theta} ^ {T} \mathbf {U} \boldsymbol {\Lambda} _ {\mathcal {I}} \mathbf {U} ^ {T} \boldsymbol {\theta} \tag {88}
$$
Proof. Decompose  $\theta$  in the above orthonormal basis, giving
$$
\theta (\cdot) = \sum_ {j = 1} ^ {N} w _ {j} u ^ {(j)} (\cdot). \tag {89}
$$
By orthonormality, the definition of  $R_{k,\pmb{x}}^{\mathcal{T}}$ , and properties of projections, we have
$$
\left(\operatorname {p r o j} _ {R _ {k, \boldsymbol {\pi}} ^ {\mathcal {I}}} \theta\right) (\cdot) = \sum_ {j \in \mathcal {I}} w _ {j} u ^ {(j)} (\cdot). \tag {90}
$$
Therefore, again using orthonormality, we obtain
$$
\left| \theta \right| _ {R _ {k, \mathbf {x}} ^ {\mathcal {I}}} ^ {2} = \sum_ {j \in \mathcal {I}} w _ {j} ^ {2} = \boldsymbol {\theta} ^ {T} \mathbf {U} \boldsymbol {\Lambda} _ {\mathcal {I}} \mathbf {U} ^ {T} \boldsymbol {\theta} \tag {91}
$$
With this at hand, we claim that our variant of SGD converges quickly with respect to this seminorm.
Proposition 10. Under the assumptions of Lemma 2, for any  $\mathcal{I}$  we have
$$
\left| \mu_ {f | \boldsymbol {y}} - \mu_ {\mathrm {S G D}} \right| _ {R _ {k, \boldsymbol {x}} ^ {\mathcal {I}}} \leq \left(\frac {\| \boldsymbol {y} \| _ {2}}{\eta \sigma^ {2} t} + G \sqrt {\frac {2}{t} \log \frac {| \mathcal {I} |}{\delta}}\right) \sqrt {\sum_ {i \in \mathcal {I}} \frac {1}{\lambda_ {i}}}. \tag {92}
$$
Proof. Using Lemma 9 and linearity, we have
$$
\begin{array}{l} \left| \mu_ {f | \boldsymbol {y}} - \mu_ {\mathrm {S G D}} \right| _ {R _ {k, \boldsymbol {x}} ^ {\mathcal {T}}} = \left| \sum_ {i = 1} ^ {N} \left(v _ {i} - v _ {i} ^ {*}\right) k \left(x _ {i}, \cdot\right) \right| _ {R _ {k, \boldsymbol {x}} ^ {\mathcal {T}}} = \sqrt {\left(\boldsymbol {v} - \boldsymbol {v} ^ {*}\right) ^ {T} \mathbf {U} \boldsymbol {\Lambda} _ {\mathcal {I}} \mathbf {U} ^ {T} \left(\boldsymbol {v} - \boldsymbol {v} ^ {*}\right)} (93) \\ = \sqrt {\sum_ {i \in \mathcal {I}} \left| \boldsymbol {u} _ {i} ^ {T} \left(\boldsymbol {v} - \boldsymbol {v} ^ {*}\right) \right| ^ {2} \lambda_ {i}} \leq \sqrt {\sum_ {i \in \mathcal {I}} \left| \frac {1}{\lambda_ {i}} \left(\frac {\| \boldsymbol {y} \| _ {2}}{\eta \sigma^ {2} t} + G \sqrt {\frac {2}{t} \log \frac {| \mathcal {I} |}{\delta}}\right) \right| ^ {2} \lambda_ {i}} (94) \\ = \left(\frac {\| \boldsymbol {y} \| _ {2}}{\eta \sigma^ {2} t} + G \sqrt {\frac {2}{t} \log \frac {| \mathcal {I} |}{\delta}}\right) \sqrt {\sum_ {i \in \mathcal {I}} \frac {1}{\lambda_ {i}}} (95) \\ \end{array}
$$
where the final inequality comes from substituting in the result of Lemma 2, yielding the claim.
Our main claim follows directly from the established framework.
Proposition 1. Let  $\delta >0$ . Let  $\pmb{\Sigma} = \sigma^{2}\mathbf{I}$  for  $\sigma^2 >0$ . Let  $\mu_{\mathrm{SGD}}$  be the predictive mean obtained by Polyak-averaged SGD after  $t$  steps, starting from an initial set of representative weights equal to zero, and using a sufficiently small learning rate of  $0 < \eta < \frac{\sigma^2}{\lambda_1(\lambda_1 + \sigma^2)}$ . Assume the stochastic estimate of the gradient is  $G$ -sub-Gaussian. Then, with probability  $1 - \delta$ , we have for  $i = 1,\dots,N$  that
Proof. Choose  $\mathcal{I}$  to be a singleton, and apply Proposition 10, where by replacing  $|\mathcal{I}|$  with  $N$  in the final bound the claim can be made to hold with probability  $1 - \delta$  for all  $i = 1, \dots, N$  simultaneously.
This concludes the first part of the argument for why SGD is a good idea: it converges fast with respect to this seminorm. In particular, taking  $\mathcal{I}$  to be the full index set, a position-dependent pointwise convergence bound follows.
Corollary 11. Under the assumptions of Lemma 2, we have
$$
\left| \mu_ {\mathrm {S G D}} (\cdot) - \mu_ {f | \boldsymbol {y}} (\cdot) \right| \leq \left(\frac {\| \boldsymbol {y} \| _ {2}}{\eta \sigma^ {2} t} + G \sqrt {\frac {2}{t} \log \frac {| \mathcal {I} |}{\delta}}\right) \sum_ {i = 1} ^ {N} \frac {1}{\sqrt {\lambda_ {i}}} \left| u ^ {(i)} (\cdot) \right|. \tag {96}
$$
Proof. Using Lemma 8, write
$$
\mu_ {\mathrm {S G D}} (\cdot) - \mu_ {f | \boldsymbol {y}} (\cdot) = \sum_ {j = 1} ^ {N} \left(v _ {j} - v _ {j} ^ {*}\right) k \left(x _ {i}, \cdot\right) = \sum_ {i = 1} ^ {N} \left(w _ {i} - w _ {i} ^ {*}\right) u ^ {(i)} (\cdot) \tag {97}
$$
where  $\pmb{w} - \pmb{w}^{*} = \Lambda^{1/2}\mathbf{U}^{T}(\pmb{v} - \pmb{v}^{*})$ . Applying Lemma 2 with  $\mathcal{I} = \{1,..,N\}$ , we conclude
$$
\begin{array}{l} \left| \mu_ {\mathrm {S G D}} (\cdot) - \mu_ {f | \boldsymbol {y}} (\cdot) \right| \leq \sum_ {i = 1} ^ {N} \left| w _ {i} - w _ {i} ^ {*} \right| \left| u ^ {(i)} (\cdot) \right| = \sum_ {i = 1} ^ {N} \sqrt {\lambda_ {i}} \left| \boldsymbol {u} _ {i} ^ {T} \left(\boldsymbol {v} - \boldsymbol {v} ^ {*}\right) \right| \left| u ^ {(i)} (\cdot) \right| (98) \\ \leq \sum_ {i = 1} ^ {N} \sqrt {\lambda_ {i}} \left| \frac {1}{\lambda_ {i}} \left(\frac {\| \boldsymbol {y} \| _ {2}}{\eta \sigma^ {2} t} + G \sqrt {\frac {2}{t} \log \frac {| \mathcal {I} |}{\delta}}\right) \right| | u ^ {(i)} (\cdot) | (99) \\ = \left(\frac {\| \boldsymbol {y} \| _ {2}}{\eta \sigma^ {2} t} + G \sqrt {\frac {2}{t} \log \frac {| \mathcal {I} |}{\delta}}\right) \sum_ {i = 1} ^ {N} \frac {1}{\sqrt {\lambda_ {i}}} | u ^ {(i)} (\cdot) |. (100) \\ \end{array}
$$
Examining the expression, the approximation error will be high at locations  $x \in X$  where  $u^{(i)}(x)$  corresponding to tail eigenfunctions is large. We proceed to analyze when this occurs.
E.3 Courant-Fischer in the Reproducing Kernel Hilbert Space
Since the preceding seminorm is induced by a projection within the RKHS, the next step is to understand what kind of functions the corresponding subspace contains. To get a qualitative view, the basic idea is to lift the Courant-Fischer characterization of eigenvalues and eigenvectors to the RKHS.
We will need the following variant of the Min-Max Theorem, which is useful for describing multiple eigenvectors at once—as solutions to a sequence of Raleigh-quotient-type optimization problems, where the next optimization problem is performed on a subspace which ensures its solution is orthogonal to all previous solutions. Mirroring the rest of this work, we consider eigenvalues in decreasing order, namely  $\lambda_1 \geq \ldots \geq \lambda_N \geq 0$ .
Result 12. Let  $\mathbf{A}$  be a positive semi-definite matrix. Then