SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proof of Lemma 3.1. Using the definitions (2) and (3) of the posterior and SGPR means, we have  $\nu_{m}(x) = \nu_{n}(x) - \langle \mathbf{r}_{m}(x),\mathbf{y}\rangle$ , from which (7) follows on taking the  $E_{f_0}$ -expectation. Next, recall that  $\nu_{n}(x) = \mathbf{k}_{n}(x)^{T}\left[\sum_{j = 1}^{n}\eta_{j}\mathbf{v}_{j}\mathbf{v}_{j}^{T}\right]\mathbf{y}$ , and thus
$$
\begin{array}{l} t _ {n} ^ {2} (x) = \operatorname {V a r} (\nu_ {n} (x)) = \left\| \left[ \sum_ {j = 1} ^ {n} \eta_ {j} \mathbf {v} _ {j} \mathbf {v} _ {j} ^ {T} \right] \mathbf {k} _ {n} (x) \right\| ^ {2} = \left\| \left[ \sum_ {j = 1} ^ {m} \eta_ {j} \mathbf {v} _ {j} \mathbf {v} _ {j} ^ {T} \right] \mathbf {k} _ {n} (x) + \mathbf {r} _ {m} (x) \right\| ^ {2} \\ = \left\| \left[ \sum_ {j = 1} ^ {m} \eta_ {j} \mathbf {v} _ {j} \mathbf {v} _ {j} ^ {T} \right] \mathbf {k} _ {n} (x) \right\| ^ {2} + \| \mathbf {r} _ {m} (x) \| ^ {2} \\ = t _ {m} ^ {2} (x) + \left\| \mathbf {r} _ {m} (x) \right\| ^ {2}, \\ \end{array}
$$
from which (8) follows. Finally,
$$
\begin{array}{l} \sigma_ {n} ^ {2} (x) = k _ {n} (x, x) = k (x, x) - \mathbf {k} _ {n} (x) ^ {T} \left[ \sum_ {j = 1} ^ {n} \eta_ {j} \mathbf {v} _ {j} \mathbf {v} _ {j} ^ {T} \right] \mathbf {k} _ {n} (x) \\ = k (x, x) - \mathbf {k} _ {n} (x) ^ {T} \left[ \sum_ {j = 1} ^ {m} \eta_ {j} \mathbf {v} _ {j} \mathbf {v} _ {j} ^ {T} \right] \mathbf {k} _ {n} (x) - \mathbf {k} _ {n} (x) ^ {T} \left[ \sum_ {j = m + 1} ^ {n} \eta_ {j} \mathbf {v} _ {j} \mathbf {v} _ {j} ^ {T} \right] \mathbf {k} _ {n} (x), \\ \end{array}
$$
which gives (9).
Proof of Proposition 3.2. We will prove this for the full posterior, with the expressions for the SGPR following the same. Denoting by  $E_{\Pi |Y}$  the expectation with respect to the posterior, Markov's inequality yields,
$$
\Pi \left(| f (x) - f _ {0} (x) | > M _ {n} \varepsilon_ {n} | Y\right) \leq \frac {E _ {\Pi | Y} \left(f (x) - f _ {0} (x)\right) ^ {2}}{M _ {n} ^ {2} \varepsilon_ {n} ^ {2}}.
$$
$$
E _ {\Pi | Y} \left(f (x) - f _ {0} (x)\right) ^ {2} = \left(\nu_ {n} (x) - f _ {0} (x)\right) ^ {2} + \operatorname {V a r} _ {\Pi | Y} \left(f (x)\right) = \left(\nu_ {n} (x) - f _ {0} (x)\right) ^ {2} + \sigma_ {n} ^ {2} (x).
$$
$$
E _ {f _ {0}} \left[ E _ {\Pi | Y} \left(f (x) - f _ {0} (x)\right) ^ {2} \right] = E _ {f _ {0}} \left[ (\nu_ {n} (x) - f _ {0} (x)) ^ {2} \right] + \sigma_ {n} ^ {2} (x) = b _ {n} ^ {2} (x) + t _ {n} ^ {2} (x) + \sigma_ {n} ^ {2} (x).
$$
Thus for  $\varepsilon_n^2 (x)\asymp b_n^2 (x) + t_n^2 (x) + \sigma_n^2 (x)$  and any  $M_{n}\to \infty$ $E_{f_0}\Pi (|f(x) - f_0(x)| > M_n\varepsilon_n|Y)\rightarrow$  0.
Proof of Proposition 3.4. We have,
$$
P _ {f _ {0}} (f _ {0} (x) \in C _ {m} ^ {\delta}) = P _ {f _ {0}} (| \nu_ {m} (x) - f _ {0} (x) | \leq z _ {1 - \delta} \sigma_ {m} (x)) = P _ {f _ {0}} (| V _ {m} | \leq z _ {1 - \delta}),
$$
for  $V_{m}\sim \mathcal{N}\left(\frac{b_{m}(x)}{\sigma_{m}(x)},\frac{t_{m}^{2}(x)}{\sigma_{m}^{2}(x)}\right)$ . Now, if  $\langle \mathbf{r}_m(x),\mathbf{f}_0\rangle = o(b_n(x))$ , we have  $b_{m}(x) = b_{n}(x)(1 + o(1))$  and further, by expressions (8) and (9) we have  $\frac{t_m^2(x)}{\sigma_m^2(x)}\leq \frac{t_n^2(x)}{\sigma_n^2(x)}$ , and thus,
$$
\liminf_{n\to \infty}P_{f_{0}}\left(|V_{m}|\leq z_{1 - \delta}\right)\geq \liminf_{n\to \infty}P_{f_{0}}\left(|\mathbf{V}_{n}|\leq z_{1 - \delta}\right) = \liminf_{n\to \infty}P_{f_{0}}\left(f_{0}(x)\in C_{n}^{\delta}\right).
$$
If in addition, we have  $\| \mathbf{r}_m(x) \|^2 = o(t_n^2(x))$  and  $\langle \mathbf{r}_m(x), \mathbf{k}_n(x) \rangle = o(\sigma_n^2(x))$  then by expressions (8) and (9) we have  $t_m^2(x) = t_n^2(x)(1 + o(1))$  and  $\sigma_m^2(x) = \sigma_n^2(x)(1 + o(1))$  and thus,
$$
\lim  _ {n \to \infty} P _ {f _ {0}} \left(\left| V _ {m} \right| \leq z _ {1 - \delta}\right) = \lim  _ {n \to \infty} P _ {f _ {0}} \left(\left| \mathbf {V} _ {n} \right| \leq z _ {1 - \delta}\right) = \lim  _ {n \to \infty} P _ {f _ {0}} \left(f _ {0} (x) \in C _ {n} ^ {\delta}\right).
$$
A.2 Rescaled Brownian motion prior
Recall that we take regularly spaced design points  $x_{i} = \frac{i}{n + 1 / 2}, i = 1, \dots, n$ , and that we consider the rescaled Brownian motion prior, which is a mean-zero Gaussian process with covariance kernel  $k(x, x') = c_{n}(x \wedge y) = c_{n} \min(x, y)$  with  $c_{n} = (n + 1 / 2)^{\frac{1 - 2\gamma}{1 + 2\gamma}}$  for  $\gamma > 0$ . For notational convenience, we write  $N = (n + 1 / 2) / c_{n} = (n + 1 / 2)^{1 - \frac{1 - 2\gamma}{1 + 2\gamma}}$ . The kernel matrix  $\mathbf{K}_{nn}$  evaluated at the sample points then equals
$$
\mathbf {K} _ {n n} = c _ {n} \left( \begin{array}{c c c c} 1 / (n + 1 / 2) & 1 / (n + 1 / 2) & \ldots & 1 / (n + 1 / 2) \\ 1 / (n + 1 / 2) & 2 / (n + 1 / 2) & \ldots & 2 / (n + 1 / 2) \\ \vdots & \vdots & \ddots & \vdots \\ 1 / (n + 1 / 2) & 2 / (n + 1 / 2) & \dots & n / (n + 1 / 2) \end{array} \right) = \frac {1}{N} \left( \begin{array}{c c c c} 1 & 1 & \ldots & 1 \\ 1 & 2 & \ldots & 2 \\ \vdots & \vdots & \ddots & \vdots \\ 1 & 2 & \dots & n \end{array} \right).
$$
For this specific choice of prior and design, we can compute fairly explicit expressions for the eigenvalues and eigenvectors of  $\mathbf{K}_{nn}$ , which will in turn allow us to obtain the precise pointwise asymptotics of the VB method needed for our results.
Lemma A.1 (Eigenvalues and eigenvectors). The eigenvalues  $(\mu_j)_{j=1}^n$  of the covariance matrix  $\mathbf{K}_{nn}$  are given by
$$
\mu_ {j} = \frac {1}{2 N (1 - \cos (\psi_ {j}))} \quad w i t h \quad \psi_ {j} = \frac {j - 1 / 2}{n + 1 / 2} \pi
$$
for  $j = 1, \dots, n$ . Moreover, their size satisfies
$$
\mu_ {j} \asymp \frac {1}{N \psi_ {j} ^ {2}} \asymp (n + 1 / 2) c _ {n} \frac {1}{(j - 1 / 2) ^ {2}}.
$$
The corresponding orthonormal eigenvectors  $\mathbf{v}_1, \ldots, \mathbf{v}_n$ , where  $\mathbf{v}_j = (v_j^1, \ldots, v_j^n)^T$ , are given by
$$
v _ {j} ^ {l} = \frac {2 \sin (l \psi_ {j})}{\sqrt {2 n + 1}}.
$$
Proof. Let  $v$  be an eigenvector of  $\mathbf{K}_{nn}$  with corresponding eigenvalue  $\mu$ , so that  $\mathbf{K}_{nn}v = \mu v$ . Applying elementary row operations to this equation, we arrive at the following equivalent equations:
$$
\frac {1}{N} \left[ \begin{array}{c} v ^ {1} \\ v ^ {2} \\ \vdots \\ v ^ {n - 1} \\ v ^ {n} \end{array} \right] = \mu \left[ \begin{array}{c} 2 v ^ {1} - v ^ {2} \\ 2 v ^ {2} - v ^ {1} - v ^ {3} \\ \vdots \\ 2 v ^ {n - 1} - v ^ {n - 2} - v ^ {n} \\ v ^ {n} - v ^ {n - 1} \end{array} \right].
$$
From this, one can see that to obtain a non-trivial eigenvector  $v \neq 0$ , we require  $v^1 \neq 0$ . We thus consider the unnormalized eigenvector  $w = (w^{1},\ldots ,w^{n})^{T} \in \mathbb{R}^{n}$ , for which we set  $w^{1} = 1$  for simplicity. Further define  $w^{0} = 0$  and  $w^{-1} = -1$ , so that the first  $(n - 1)$  equations in the last display are equivalent (after rearranging) to the recurrence relation
$$
w ^ {j} = \left(2 - \frac {1}{\mu N}\right) w ^ {j - 1} - w ^ {j - 2}, \quad j = 1, \dots , n, \tag {11}
$$
while the last equation gives the boundary condition
$$
w ^ {n} = \frac {\mu N}{\mu N - 1} w ^ {n - 1}. \tag {12}
$$
The linear recurrence relation (11) has solution
$$
w ^ {j} = \frac {1}{\phi_ {+} - \phi_ {-}} \left(\phi_ {+} ^ {j} - \phi_ {-} ^ {j}\right), \tag {13}
$$
$$
\phi_ {\pm} = 1 - \frac {1}{2 \mu N} \pm \frac {1}{2} \sqrt {\frac {1}{\mu^ {2} N ^ {2}} - \frac {4}{\mu N}} =: x \pm i y,
$$
with  $x = 1 - \frac{1}{2\mu N}$  and  $y = \frac{1}{2}\sqrt{\frac{4}{\mu N} - \frac{1}{\mu^2N^2}}$  (computation of the eigenvalues below reveals that  $\frac{1}{\mu^2N^2} -\frac{4}{\mu N} < 0$  for all eigenvalues  $\mu$ .) Note that (13) provides the solution to (11) for any arbitrary value of  $\mu$ , but that (12) will later be needed to ensure that  $\mu$  is an eigenvalue of  $\mathbf{K}_{nn}$ . Since  $x^{2} + y^{2} = 1$ , we can conveniently rewrite this in complex exponential form as
$$
\phi_ {\pm} = e ^ {\pm i \psi}, \quad \text {w h e r e} \quad \cos \psi = \operatorname {R e} \left(\phi_ {\pm}\right) = x = 1 - \frac {1}{2 \mu N}, \tag {14}
$$
and  $\psi$  is an alternative parametrization of  $\mu$ . This form makes it easy to compute expressions for  $\phi_{\pm}^{j}$ , and hence evaluate (13). Indeed, we have
$$
w ^ {j} = \frac {1}{\phi_ {+} - \phi_ {-}} \left(\phi_ {+} ^ {j} - \phi_ {-} ^ {j}\right) = \frac {\sin (j \psi)}{\sin \psi}, \tag {15}
$$
giving the form of the unnormalized eigenvector  $w = (1, w^2, \ldots, w^n)^T$  corresponding to an eigenvalue  $\mu$  via (14). To normalize the eigenvector, using that  $\cos(2j\psi) = 1 - 2\sin^2(j\psi)$ ,
$$
\begin{array}{l} \| w \| _ {2} ^ {2} = \sum_ {j = 1} ^ {n} (w ^ {j}) ^ {2} = \sum_ {j = 1} ^ {n} \frac {\sin^ {2} (j \psi)}{\sin^ {2} \psi} = \frac {1}{\sin^ {2} \psi} \sum_ {j = 1} ^ {n} \frac {1}{2} (1 - \cos (2 j \psi)) \\ = \frac {1}{4 \sin^ {2} \psi} \left(2 n + 1 - \frac {\sin ((2 n + 1) \psi)}{\sin \psi}\right), \\ \end{array}
$$
where in the last line we have used the Lagrange trigonometric identity that  $\sum_{j=0}^{n} \cos(2j\psi) = \frac{1}{2} + \frac{\sin(2n+1)\psi}{2\sin\psi}$  for any  $\psi \neq 0 (\mathrm{mod} \pi)$  (else it equals  $n+1$ ). On computing the values  $\psi$  corresponding to the eigenvalues in (17) below, one has  $\sin((2n+1)\psi) = 2\sin((n+1/2)\psi)\cos((n+1/2)\psi) = 0$  and hence  $\|w\|_2^2 = \frac{2n+1}{4\sin^2\psi}$ . Thus the normalized eigenvector with  $v = w / \|w\|_2$  has the required coordinates,
$$
v ^ {j} = \frac {2 \sin (j \psi)}{\sqrt {2 n + 1}}. \tag {16}
$$
Turning to the computation of the eigenvalues, the boundary condition (12) defines whether  $\mu$  is an eigenvalue. Substituting the solution (15) into (12), any  $\mu$  that satisfies
$$
\frac {\sin (n \psi)}{\sin \psi} = \frac {\mu N}{\mu N - 1} \frac {\sin ((n - 1) \psi)}{\sin \psi},
$$
$$
(\mu N - 1) \sin (n \psi) = \mu N \sin ((n - 1) \psi) \qquad \text {a n d} \qquad \sin \psi \neq 0,
$$
is an eigenvalue of  $\mathbf{K}_{nn}$ . Indeed, if one approaches the problem in the traditional way, by computing the determinant of  $(\mathbf{K}_{nn} - \mu \mathbf{I}_n)$  (see Lemma A.2 below), then one has
$$
\begin{array}{l} \left| \mathbf {K} _ {n n} - \mu \mathbf {I} _ {n} \right| = \frac {(- 1) ^ {n}}{\sin \psi} \left(\left[ \mu^ {n} - \frac {\mu^ {n - 1}}{N} \right] \sin (n \psi) - \mu^ {n} \sin ((n - 1) \psi)\right) = 0 \\ \Leftrightarrow \left[ \mu - \frac {1}{N} \right] \sin (n \psi) - \mu N \sin ((n - 1) \psi) = 0 \quad \text {a n d} \quad \sin \psi \neq 0 \\ \Leftrightarrow (\mu N - 1) \sin (n \psi) - \mu N \sin ((n - 1) \psi) = 0 \quad \text {a n d} \quad \sin \psi \neq 0, \\ \end{array}
$$
exactly as above. Recalling that  $\psi$  and  $\mu$  are related by  $\cos \psi = 1 - \frac{1}{2\mu N}$ , this last expression is equivalent to
$$
\begin{array}{l} 0 = \left(1 - \frac {1}{\mu N}\right) \sin (n \psi) - \sin ((n - 1) \psi) \\ = (2 \cos \psi - 1) \sin (n \psi) - \sin ((n - 1) \psi) \\ = \sin ((n + 1) \psi) - \sin (n \psi) \\ = 2 \sin (\psi / 2) \cos ((n + 1 / 2) \psi), \\ \end{array}
$$
where we have used the identity  $2\cos A\sin B = \sin (A + B) - \sin (A - B)$  twice. By (14), the eigenvalues  $\mu$  are only defined by  $\cos \psi$ , and hence we may restrict  $\psi$  to  $[- \pi, \pi)$  and look for solutions within this range. For  $\psi$  in this range, the condition  $\sin \psi \neq 0$  then implies  $\sin (\psi / 2) \neq 0$ , and hence we need only look for solutions to the equation  $\cos ((n + 1 / 2)\psi) = 0$ . For  $\psi \in [-\pi, \pi)$ , these are given by
$$
\psi_ {j} = \frac {j - 1 / 2}{n + 1 / 2} \pi , \quad j = - n, - n + 1, \dots , n. \tag {17}
$$
By symmetry,  $\cos \psi_{j} = \cos \psi_{1 - j}$  for  $j = 1,\ldots ,n$  and hence we may discard the roots  $j = -n + 1,\dots ,0$ . Moreover,  $\sin \psi_{-n} = 0$  violating the restriction  $\sin \psi \neq 0$ . This leaves us with the  $n$  eigenvalues
$$
\mu_ {j} = \frac {1}{2 N (1 - \cos \psi_ {j})}, \quad j = 1, \ldots , n
$$
It remains only to quantify the size of the eigenvalues  $\mu_{j}$ . Expanding  $\cos x = 1 - x^{2} / 2! + x^{4} / 4!$  and  $o(x^4)$ , we have  $x^{2} - x^{4} / 12 \leq 2(1 - \cos x) \leq x^{2}$  for all  $x$ , and thus
$$
\frac {1}{N x ^ {2}} \leq \frac {1}{2 N (1 - \cos x)} \leq \frac {1}{N x ^ {2} (1 - x ^ {2} / 1 2)}.
$$
Since  $\psi_{j} = \frac{j - 1 / 2}{n + 1 / 2}\pi$ , we have  $\psi_j^2 /12 < 5 / 6$  (on bounding  $\pi^2$  by 10 and the fraction by 1) and so  $\frac{1}{N\psi_j^2(1 - \psi_j^2 / 12)} < \frac{6}{N\psi_j^2}$ . Hence we have  $\frac{1}{N\psi_j^2}\leq \mu_j\leq \frac{6}{N\psi_j^2}$  and  $\mu_j\succ \frac{1}{N\psi_j^2}$ .
Let  $|A|$  denote the determinant of a matrix  $A$ .
Lemma A.2. The solutions to the characteristic equation of  $\mathbf{K}_{nn}$ ,  $|\mathbf{K}_{nn} - \mu \mathbf{I}_n| = 0$ , are given by the solutions to
$$
\frac {(- 1) ^ {n}}{\sin \psi} \left(\left(\mu^ {n} - \frac {1}{N} \mu^ {n - 1}\right) \sin (n \psi) - \mu^ {n} \sin ((n - 1) \psi)\right) = 0,
$$
where  $\cos \psi = 1 - \frac{1}{2N\mu}$
Proof. Using row operations, the desired determinant equals
$$
| \mathbf {K} _ {n n} - \mu \mathbf {I} _ {n} | = \left| \begin{array}{c c c c} \frac {1}{N} - \mu & \frac {1}{N} & \dots & \frac {1}{N} \\ \frac {1}{N} & \frac {2}{N} - \mu & \dots & \frac {2}{N} \\ \vdots & \vdots & \ddots & \vdots \\ \frac {1}{N} & \frac {2}{N} & \dots & \frac {n}{N} - \mu \end{array} \right| = \left| \begin{array}{c c c c c} \frac {1}{N} - \mu & \frac {1}{N} & \dots & \dots & \frac {1}{N} \\ \mu & \frac {1}{N} - \mu & \frac {1}{N} & \dots & \frac {1}{N} \\ 0 & \mu & \ddots & \ddots & \vdots \\ \vdots & \ddots & \ddots & \frac {1}{N} - \mu & \frac {1}{N} \\ 0 & \dots & 0 & \mu & \frac {1}{N} - \mu \end{array} \right|,
$$
which can be further be reduced to
$$
| \mathbf {K} _ {n n} - \mu \mathbf {I} _ {n} | = \left| \begin{array}{c c c c c} \frac {2}{N} - \mu & \mu & 0 & \dots & 0 \\ \mu & \frac {2}{N} - \mu & \mu & \ddots & \vdots \\ 0 & \mu & \ddots & \ddots & 0 \\ \vdots & \ddots & \ddots & \frac {2}{N} - \mu & \mu \\ 0 & \dots & 0 & \mu & \frac {1}{N} - \mu \end{array} \right| =: d _ {n},
$$
the determinant of a symmetric tridiagonal  $n \times n$  matrix. Using the Laplace expansion of the determinant in terms of the determinants of its  $(n - 1) \times (n - 1)$  submatrices, we obtain the recurrence relation  $d_{n} = \left(\frac{2}{N} - \mu\right)d_{n - 1} - \mu^{2}d_{n - 2}$ . The solution to this recurrence is given by
$$
d _ {n} = \frac {1}{\phi_ {+} - \phi_ {-}} \left(\phi_ {+} ^ {n + 1} - \phi_ {-} ^ {n + 1}\right),
$$
where  $\phi_{\pm}$  satisfy  $\phi^2 -\left(\frac{2}{N} -\mu\right)\phi +\mu^2 = 0$  . That is,
$$
\phi_ {\pm} = \frac {(\frac {2}{N} - \mu) \pm \sqrt {\frac {4}{N ^ {2}} - \frac {4 \mu}{N}}}{2}.
$$
Similar to when we solved the recurrence in the proof of Lemma A.1, we end up with  $\phi_{\pm} = \mu e^{\pm i\psi'}$  where  $\cos (\psi') = \frac{1}{2N\mu} - 1$ . We use the notation  $\psi'$  to emphasise that these angles are different to the  $\psi'$ s found in the proof of Lemma A.1; indeed, it is easy to see that  $\psi' = \psi + \pi$ , so that  $\sin (n\psi') = (-1)^n \sin (n\psi)$ . This yields
$$
d _ {n} = (- 1) ^ {n} \frac {\mu^ {n}}{\sin (\psi)} \sin ((n + 1) \psi),
$$
and the result is shown by seeing
$$
\begin{array}{l} | \mathbf {K} _ {n n} - \mu I | = \left(\frac {1}{N} - \mu\right) d _ {n - 1} - \mu^ {2} d _ {n - 2} \\ = (- 1) ^ {n - 1} \left(\frac {1}{N} - \mu\right) \frac {\mu^ {n - 1}}{\sin (\psi)} \sin (n \psi) - (- 1) ^ {n - 2} \mu^ {2} \frac {\mu^ {n - 2}}{\sin (\psi)} \sin ((n - 1) \psi) \\ = \frac {(- 1) ^ {n}}{\sin \psi} \left(\left(\mu^ {n} - \frac {1}{N} \mu^ {n - 1}\right) \sin (n \psi) - \mu^ {n} \sin ((n - 1) \psi)\right) \\ \end{array}
$$
Given the explicit expressions derived for the eigenvalues and eigenvectors of the covariance matrix  $\mathbf{K}_{nn}$ , we derive bounds for the size of the remainder terms presented in Lemma 3.1.
Lemma A.3. Suppose that  $f_0 \in C^\alpha[0,1]$ ,  $\alpha \in (0,1]$ , and consider the rescaled Brownian motion prior of regularity  $\gamma \in (0,1]$ . The corresponding SGPR  $Q^*$  based on the first  $m = m_n \to \infty$  eigenvector inducing variables satisfies, as  $n \to \infty$ ,
$$
b _ {m} (x) = b _ {n} (x) + O \left(n c _ {n} m ^ {- (1 + \alpha)}\right) \tag {18}
$$
$$
t _ {m} ^ {2} (x) = t _ {n} ^ {2} (x) + O \left(n c _ {n} ^ {2} m ^ {- 3}\right) \tag {19}
$$
$$
\sigma_ {m} ^ {2} (x) = \sigma_ {n} ^ {2} (x) + O \left(n c _ {n} ^ {2} m ^ {- 3}\right). \tag {20}
$$
Proof. First, suppose that  $x = x_{i}$  is a design point. Starting with the bias, we have,
$$
b _ {m} (x) = b _ {n} (x) + \mathbf {k} _ {n} (x) ^ {T} \left(\sum_ {j = m + 1} ^ {n} \eta_ {j} \mathbf {v} _ {j} \mathbf {v} _ {j} ^ {T}\right) \mathbf {f} _ {0} = b _ {n} (x) + \sum_ {j = m + 1} ^ {n} \eta_ {j} (\mathbf {k} _ {n} (x) ^ {T} \mathbf {v} _ {j}) (\mathbf {v} _ {j} ^ {T} \mathbf {f} _ {0}).
$$
In this case  $\mathbf{k}_n(x_i)^T\mathbf{v}_j = \mu_j v_j^i$  and the next step is to bound  $\mathbf{v}_j^T\mathbf{f}_0$  for each  $j$ . Using the explicit form for  $\mathbf{v}_j$  given in Lemma A.1,
$$
\begin{array}{l} \mathbf {v} _ {j} ^ {T} \mathbf {f _ {0}} = \sum_ {l = 1} ^ {n} \frac {2}{\sqrt {2 n + 1}} \sin (l \psi_ {j}) f _ {0} (x _ {l}) = \frac {2}{\sqrt {2 n + 1}} \sum_ {l = 1} ^ {n} \sin \left(l \frac {(j - 1 / 2) \pi}{n + 1 / 2}\right) f _ {0} (x _ {l}) \\ = \frac {2}{\sqrt {2 n + 1}} \sum_ {l = 1} ^ {n} \sin \left(x _ {l} (j - 1 / 2) \pi\right) f _ {0} \left(x _ {l}\right). \\ \end{array}
$$
Now, writing  $\pi_j \coloneqq (j - 1/2)\pi$ , we have  $\sin(x_l\pi_j) = \frac{1}{2i} \left( e^{ix_l\pi_j} - e^{-ix_l\pi_j} \right)$  and thus
$$
\begin{array}{l} \left| \sum_ {l = 1} ^ {n} \sin (x _ {l} (j - 1 / 2) \pi) f _ {0} (x _ {l}) \right| \lesssim \left| n \int_ {0} ^ {1} \sin (t (j - 1 / 2) \pi) f _ {0} (t) d t \right| + n ^ {1 - \alpha} \\ \leq \left| \frac {n}{2 i} \left[ \int_ {0} ^ {1} e ^ {i x \pi_ {j}} f _ {0} (x) d x - \int_ {0} ^ {1} e ^ {- i x \pi_ {j}} f _ {0} (x) d x \right] \right| + n ^ {1 - \alpha} \\ \lesssim n \pi_ {j} ^ {- \alpha} + n ^ {1 - \alpha} \asymp n \pi_ {j} ^ {- \alpha}, \\ \end{array}
$$
for  $f_0 \in C^\alpha([0,1])$ , where the first inequality follows from Lemma A.4 below. We have thus shown that  $|\mathbf{v}_j^T\mathbf{f}_0| \lesssim \sqrt{n} (j - 1/2)^{-\alpha}$ . Using this, that  $\mu_j \asymp (n + 1/2)c_n\frac{1}{(j - 1/2)^2}$  by Lemma A.1 and that  $\eta_j \lesssim 1$ ,
$$
\left| b _ {m} (x) - b _ {n} (x) \right| \lesssim \sum_ {j = m + 1} ^ {n} \eta_ {j} \mu_ {j} v _ {j} ^ {i} \sqrt {n} (j - 1 / 2) ^ {- \alpha} \asymp n c _ {n} \sum_ {j = m + 1} ^ {n} j ^ {- 2 - \alpha} \asymp n c _ {n} m ^ {- (1 + \alpha)}
$$
as required. Next, we have,
$$
t _ {m} ^ {2} (x) = t _ {n} ^ {2} (x) - \left\| \mathbf {r} _ {m} (x) \right\| ^ {2} = t _ {n} ^ {2} (x) - \mathbf {k} _ {n} (x) ^ {T} \sum_ {j = m + 1} ^ {n} \eta_ {j} ^ {2} \mathbf {v} _ {j} \mathbf {v} _ {j} ^ {T} \mathbf {k} _ {n} (x).
$$
Using again that  $x = x_{i}$  and so  $\mathbf{k}_n(x_i)^T\mathbf{v}_j = \mu_j v_j^i$ , we have,
$$
\mathbf {k} _ {n} (x) ^ {T} \left(\sum_ {j = m + 1} ^ {n} \eta_ {j} ^ {2} \mathbf {v} _ {j} \mathbf {v} _ {j} ^ {T}\right) \mathbf {k} _ {n} (x) = \sum_ {j = m + 1} ^ {n} \eta_ {j} ^ {2} \mu_ {j} ^ {2} (v _ {j} ^ {i}) ^ {2} \lesssim \frac {1}{n} \sum_ {j = m + 1} ^ {n} \frac {n ^ {2} c _ {n} ^ {2}}{(j - 1 / 2) ^ {4}} \lesssim n c _ {n} ^ {2} m ^ {- 3},
$$
as required. Finally,
$$
\sigma_ {m} ^ {2} (x) = \sigma_ {n} ^ {2} (x) + \mathbf {k} _ {n} (x) ^ {T} \left(\sum_ {j = m + 1} ^ {n} \eta_ {j} \mathbf {v} _ {j} \mathbf {v} _ {j} ^ {T}\right) \mathbf {k} _ {n} (x), \tag {21}
$$
and again at a design point  $x = x_{i}$
$$
\mathbf {k} _ {n} (x) ^ {T} \left(\sum_ {j = m + 1} ^ {n} \eta_ {j} \mathbf {v} _ {j} \mathbf {v} _ {j} ^ {T}\right) \mathbf {k} _ {n} (x) = \sum_ {j = m + 1} ^ {n} \eta_ {j} \mu_ {j} ^ {2} (v _ {j} ^ {i}) ^ {2} \lesssim n c _ {n} ^ {2} m ^ {- 3}.
$$
This establishes the three relations at the design points  $x = x_{i}$
Turning to the general case  $x \in [0,1]$ , let  $i$  be such that  $x_{i} < x < x_{i + 1}$ . Recalling that the covariance function equals  $k(x,x^{\prime}) = c_{n}(x\wedge x^{\prime})$ , we obtain
$$
\mathbf {k} _ {n} (x) = c _ {n} \left[ \begin{array}{c} x _ {1} \\ \vdots \\ x _ {i} \\ x \\ \vdots \\ x \end{array} \right] = \mathbf {k} _ {n} (x _ {i}) + c _ {n} \left[ \begin{array}{c} 0 \\ \vdots \\ 0 \\ x - x _ {i} \\ \vdots \\ x - x _ {i} \end{array} \right],
$$
where  $0 < x - x_{i} < \frac{1}{n + 1 / 2}$ . Consider first the case of the SGPR variance  $\sigma_m^2 (x)$ . The last display implies that  $(\mathbf{k}_n(x) - \mathbf{k}_n(x_i))^T\mathbf{v}_j = c_n(x - x_i)\sum_{l = i + 1}^{n}v_j^l$ , and hence the quadratic term in (21) can be rewritten as
$$
\begin{array}{l} \mathbf {k} _ {n} (x) ^ {T} \left(\sum_ {j = m + 1} ^ {n} \eta_ {j} \mathbf {v} _ {j} \mathbf {v} _ {j} ^ {T}\right) \mathbf {k} _ {n} (x) = \mathbf {k} _ {n} (x _ {i}) ^ {T} \left(\sum_ {j = m + 1} ^ {n} \eta_ {j} \mathbf {v} _ {j} \mathbf {v} _ {j} ^ {T}\right) \mathbf {k} _ {n} (x _ {i}) \\ + 2 c _ {n} \sum_ {j = m + 1} ^ {n} \eta_ {j} \left[ \left(x - x _ {i}\right) \sum_ {l = i + 1} ^ {n} v _ {j} ^ {l} \right] \mathbf {v} _ {j} ^ {T} \mathbf {k} _ {n} \left(x _ {i}\right) \tag {22} \\ + c _ {n} ^ {2} \sum_ {j = m + 1} ^ {n} \eta_ {j} \left[ (x - x _ {i}) \sum_ {l = i + 1} ^ {n} v _ {j} ^ {l} \right] ^ {2}. \\ \end{array}
$$
The first term is exactly the quadratic term evaluated at the design point  $x_{i}$ , which we showed above has size  $O(nc_n^2 m^{-3})$ . Using Lemmas A.1 and A.4,
$$
\begin{array}{l} \sum_ {l = i + 1} ^ {n} v _ {j} ^ {l} = \frac {2}{\sqrt {2 n + 1}} \sum_ {l = i + 1} ^ {n} \sin (x _ {l} (j - 1 / 2) \pi) \\ = \frac {2 n}{\sqrt {2 n + 1}} \left(\int_ {x _ {i}} ^ {1} \sin (t (j - 1 / 2) \pi) d t + O (n ^ {- 1})\right) \\ = \frac {2 n}{\sqrt {2 n + 1}} \frac {\cos (x _ {i} (j - 1 / 2) \pi)}{(j - 1 / 2) \pi} + O (n ^ {- 1 / 2}) = O \left(\frac {\sqrt {n}}{j - 1 / 2}\right) + O (n ^ {- 1 / 2}). \\ \end{array}
$$
By Lemma A.1,  $v_{j}^{T}\mathbf{k}_{n}(x_{i}) = \mu_{j}v_{j}^{i} = O(c_{n}\sqrt{n} j^{-2})$ . Since also  $\eta_j\lesssim 1$ , the second term in (22) is bounded by a multiple of
$$
\frac {c _ {n}}{\sqrt {n}} \sum_ {j = m + 1} ^ {n} j ^ {- 1} | \mathbf {v} _ {j} ^ {T} \mathbf {k} _ {n} (x _ {i}) | \lesssim c _ {n} ^ {2} \sum_ {j = m + 1} ^ {n} j ^ {- 3} \lesssim c _ {n} ^ {2} m ^ {- 2} = O (n c _ {n} ^ {2} m ^ {- 3}).
$$
The third term in (22) can similarly be shown to be  $O(c_n^2 n^{-1}m^{-1}) = O(nc_n^2 m^{-3})$ . We have thus shown that (22) is of size  $O(nc_n^2 m^{-3})$  exactly as in the case where  $x$  was a design point, which implies the desired bound for the SGPR variance  $\sigma_m^2(x)$ .
The case  $t_m^2 (x)$  follows in a similar fashion, simply replacing  $\eta_j$  by  $\eta_j^2$ , while the bias  $b_{m}(x)$  uses additionally the bound  $|\mathbf{v}_j^T\mathbf{f}_0|\lesssim \sqrt{n} (j - 1 / 2)^{-\alpha}$  derived above. In conclusion, the three quantities  $b_{m}(x),t_{m}^{2}(x)$  and  $\sigma_{m}^{2}(x)$  satisfy the same bounds as when  $x$  is a design point, which completes the proof.
The next standard bound controls the error when approximating a  $C^\alpha$  function by its Riemann sum.
Lemma A.4. If  $f \in C^{\alpha}[0,1]$  with  $\alpha \in (0,1]$ , then
$$
\left| \int_ {0} ^ {1} f (t) d t - \frac {1}{n} \sum_ {i = 1} ^ {n} f \left(x _ {i}\right) \right| \leq (n + 1 / 2) ^ {- \alpha},
$$
with  $x_{i} = \frac{i}{n + 1 / 2}$  for  $i = 1,\ldots ,n$
The following lemma shows that for  $m$  large enough, the bias, frequentist variance of the SGPR mean and SGPR variance can each be related to those of the full posterior derived in [29].
Lemma A.5. If  $m \gg n^{\frac{1}{1 + 2\gamma}\left(\frac{2 + \alpha}{1 + \alpha}\right)}$ , then the SGPR satisfies
$$
b _ {m} (x) = b _ {n} (x) (1 + o (1))
$$
$$
t _ {m} ^ {2} (x) = t _ {n} ^ {2} (x) (1 + o (1))
$$
$$
\sigma_ {m} ^ {2} (x) = \sigma_ {n} ^ {2} (x) (1 + o (1)).
$$
Proof. From [29], we have the following expressions for the full posterior:
$$
\left| b _ {n} (x) \right| \lesssim n ^ {- \frac {\alpha}{1 + 2 \gamma}}, \qquad t _ {n} ^ {2} (x) \asymp n ^ {- \frac {2 \gamma}{1 + 2 \gamma}}, \qquad \sigma_ {n} ^ {2} (x) \asymp n ^ {- \frac {2 \gamma}{1 + 2 \gamma}}.
$$