SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Thus, if we can show that the  $O$  terms given in equations (18)-(20) are smaller than their respective terms above, we have the result. By Lemma A.3, we have  $b_{m}(x) = b_{n}(x) + O(nc_{n}m^{-(1 + \alpha)})$ , and  $nc_{n}m^{-(1 + \alpha)} = n^{\frac{2}{1 + 2\gamma}}m^{-(1 + \alpha)} \lesssim n^{-\frac{\alpha}{1 + 2\gamma}}$  if and only if  $n^{\frac{1}{1 + 2\gamma}\left(\frac{2 + \alpha}{1 + \alpha}\right)} \ll m$ . Thus, if  $m \gg n^{\frac{1}{1 + 2\gamma}\left(\frac{2 + \alpha}{1 + \alpha}\right)}$  then  $nc_{n}m^{-(1 + \alpha)} = o(b_{n}(x))$  and  $b_{m}(x) = b_{n}(x)(1 + o(1))$ . Similarly,
$$
n c _ {n} ^ {2} m ^ {- 3} \ll t _ {n} ^ {2} (x) = \sigma_ {n} ^ {2} (x) \asymp n ^ {- \frac {2 \gamma}{1 + 2 \gamma}} \Leftrightarrow m \gg n ^ {\frac {1}{1 + 2 \gamma}}.
$$
Thus,  $t_m^2 (x) = t_n^2 (x)(1 + o(1))$  and  $\sigma_{m}^{2}(x) = \sigma_{n}^{2}(x)(1 + o(1))$  if  $m\gg n^{\frac{1}{1 + 2\gamma} (\frac{2 + \alpha}{1 + \alpha})}$  (in fact these are implied even at the smaller bound  $n^{\frac{1}{1 + 2\gamma}}$ ).
Proof of Theorem 3.5. The proof follows from Proposition 3.2 combined with the two groups of expressions (7)-(9) and (18)-(20).  $\square$
Proof of Theorem 3.6. The coverage equals
$$
P _ {f _ {0}} \left(f _ {0} (x) \in C _ {m} ^ {\delta}\right) = P _ {f _ {0}} \left(\left| \nu_ {m} (x) - f _ {0} (x) \right| \leq z _ {1 - \delta} \sigma_ {m} (x)\right) = P _ {f _ {0}} \left(\left| V _ {m} \right| \leq z _ {1 - \delta}\right),
$$
for  $V_{m}\sim \mathcal{N}\left(\frac{b_{m}(x)}{\sigma_{m}(x)},\frac{t_{m}^{2}(x)}{\sigma_{m}^{2}(x)}\right)$ . Now, since  $m\gg n^{\frac{1}{1 + 2\gamma}\left(\frac{2 + \alpha}{1 + \alpha}\right)}$  we have  $b_{m}(x) = b_{n}(x)(1 + o(1))$ ,  $t_m^2 (x) = t_n^2 (x)(1 + o(1))$  and  $\sigma_{m}^{2}(x) = \sigma_{n}^{2}(x)(1 + o(1))$  and thus,
$$
\lim _ {n \to \infty} P _ {f _ {0}} \left(| V _ {m} | \leq z _ {1 - \delta}\right) = \lim _ {n \to \infty} P _ {f _ {0}} \left(| V _ {n} | \leq z _ {1 - \delta}\right) = \lim _ {n \to \infty} P _ {f _ {0}} \left(f _ {0} (x) \in C _ {n} ^ {\delta}\right).
$$
Lemma A.6. The Kullback-Leibler divergence between the SGPR  $Q^{*}$  and the full posterior satisfies
$$
2 K L (Q ^ {*} | | \Pi (\cdot | Y)) = \mathbf {y} ^ {T} \left(\frac {1}{\sigma^ {2}} \sum_ {j = m + 1} ^ {n} \frac {\mu_ {j}}{\mu_ {j} + \sigma^ {2}} \mathbf {v} _ {j} \mathbf {v} _ {j} ^ {T}\right) \mathbf {y} + \sum_ {j = m + 1} ^ {n} \log \frac {\sigma^ {2}}{\sigma^ {2} + \mu_ {j}} + \frac {1}{\sigma^ {2}} \sum_ {j = m + 1} ^ {n} \mu_ {j}.
$$
Suppose further that  $f_0 \in C^\alpha$ ,  $\alpha \in (0,1]$ , and consider the fixed design setting with the rescaled Brown motion prior of regularity  $\gamma \in (0,1]$ . If  $m \gg n^{\frac{1}{1 + 2\gamma}}$ , then
$$
E _ {f _ {0}} K L (Q ^ {*} | | \Pi (\cdot | Y)) \asymp n ^ {\frac {2}{1 + 2 \gamma}} m ^ {- 1} + n ^ {\frac {3 + 2 \gamma}{1 + 2 \gamma}} m ^ {- 1 - 2 \alpha}.
$$
In particular, if  $n^{\frac{1}{1 + 2\gamma}} \ll m \ll n^{\frac{2}{1 + 2\gamma}}$  then  $E_{f_0}KL(Q^* ||\Pi (\cdot |Y)) \to \infty$ .
Proof. We have from [31]
$$
K L (Q | | \Pi (\cdot | Y)) = \frac {1}{2} \left(\mathbf {y} ^ {T} \left(\mathbf {Q} _ {n} ^ {- 1} - \mathbf {K} _ {n} ^ {- 1}\right) \mathbf {y}\right) + \log \frac {| \mathbf {Q} _ {n} |}{| \mathbf {K} _ {n} |} + \frac {1}{\sigma^ {2}} t r (\mathbf {K} _ {n} - \mathbf {Q} _ {n}),
$$
where  $\mathbf{K}_n = \mathbf{K}_{nn} + \sigma^2 I$ ,  $\mathbf{Q}_n = Q_{nn} + \sigma^2 I$ ,  $\mathbf{K}_{nn} = \sum_{j=1}^{n} \mu_j \mathbf{v}_j \mathbf{v}_j^T$  and  $Q_{nn} = \sum_{j=1}^{m} \mu_j \mathbf{v}_j \mathbf{v}_j^T$ .
$$
\mathbf {Q} _ {n} ^ {- 1} - \mathbf {K} _ {n} ^ {- 1} = \sum_ {j = m + 1} ^ {n} \left(\frac {1}{\sigma^ {2}} - \frac {1}{\sigma^ {2} + \mu_ {j}}\right) \mathbf {v} _ {j} \mathbf {v} _ {j} ^ {T} = \frac {1}{\sigma^ {2}} \sum_ {j = m + 1} ^ {n} \frac {\mu_ {j}}{\sigma^ {2} + \mu_ {j}} \mathbf {v} _ {j} \mathbf {v} _ {j} ^ {T},
$$
$\frac{|\mathbf{Q}_n|}{|\mathbf{K}_n|} = \prod_{j=m+1}^n \frac{\sigma^2}{\sigma^2 + \mu_j}$ , and  $tr(\mathbf{K}_n - \mathbf{Q}_n) = \sum_{j=m+1}^n \mu_j$ , giving the results. We now consider the three terms given in the first part of the lemma. Defining  $M := \sum_{j=m+1}^n \frac{\mu_j}{\mu_j + \sigma^2} \mathbf{v}_j \mathbf{v}_j^T$ , we have
$$
\begin{array}{l} E _ {f _ {0}} \left[ \mathbf {y} ^ {T} \left(\frac {1}{\sigma^ {2}} \sum_ {j = m + 1} ^ {n} \frac {\mu_ {j}}{\mu_ {j} + \sigma^ {2}} \mathbf {v} _ {j} \mathbf {v} _ {j} ^ {T}\right) \mathbf {y} \right] = \frac {1}{\sigma^ {2}} E _ {f _ {0}} \left[ (\mathbf {f} _ {0} + \boldsymbol {\varepsilon}) ^ {T} M (\mathbf {f} _ {0} + \boldsymbol {\varepsilon}) \right] \\ = \frac {1}{\sigma^ {2}} \mathbf {f} _ {0} ^ {T} M \mathbf {f} _ {0} + \frac {1}{\sigma^ {2}} E _ {f _ {0}} \left[ \boldsymbol {\varepsilon} ^ {T} M \boldsymbol {\varepsilon} \right]. \\ \end{array}
$$
Recalling from the proof of Lemma A.3 that  $|\mathbf{v}_j^T\mathbf{f}_0| \lesssim \sqrt{n} j^{-\alpha}$ , we can control
$$
\frac {1}{\sigma^ {2}} \mathbf {f} _ {0} ^ {T} M \mathbf {f} _ {0} = \frac {1}{\sigma^ {2}} \sum_ {j = m + 1} ^ {n} \frac {\mu_ {j}}{\mu_ {j} + \sigma^ {2}} (\mathbf {v} _ {j} ^ {T} \mathbf {f} _ {0}) ^ {2} \lesssim \frac {1}{\sigma^ {2}} \sum_ {j = m + 1} ^ {n} n \mu_ {j} j ^ {- 2 \alpha} \asymp n ^ {\frac {3 + 2 \gamma}{1 + 2 \gamma}} m ^ {- 1 - 2 \alpha}.
$$
$$
E _ {f _ {0}} [ \pmb {\varepsilon} ^ {T} M \pmb {\varepsilon} ] = E _ {f _ {0}} t r (\pmb {\varepsilon} ^ {T} M \pmb {\varepsilon}) = E _ {f _ {0}} t r (M \pmb {\varepsilon} \pmb {\varepsilon} ^ {T}) = t r (M E _ {f _ {0}} [ \pmb {\varepsilon} \pmb {\varepsilon} ^ {T} ]) = t r (M) \asymp n ^ {\frac {2}{1 + 2 \gamma}} m ^ {- 1}.
$$
For the second term in the first part of the lemma,
$$
\begin{array}{l} \sum_ {j = m + 1} ^ {n} \log \frac {\sigma^ {2}}{\sigma^ {2} + \mu_ {j}} = - \sum_ {j = m + 1} ^ {n} \log \left(1 + \frac {\mu_ {j}}{\sigma^ {2}}\right) = - \sum_ {j = m + 1} ^ {n} \left(\frac {\mu_ {j}}{\sigma^ {2}} + O \left(\frac {\mu_ {j} ^ {2}}{\sigma^ {4}}\right)\right) \\ = - \frac {1}{\sigma^ {2}} \sum_ {j = m + 1} ^ {n} \mu_ {j} - O \left(\sum_ {j = m + 1} ^ {n} \frac {\mu_ {j} ^ {2}}{\sigma^ {4}}\right) \\ = - \frac {1}{\sigma^ {2}} n ^ {\frac {2}{1 + 2 \gamma}} / m - O (n ^ {\frac {4}{1 + 2 \gamma}} m ^ {- 3}). \\ \end{array}
$$
Hence, adding these expressions to the third term in the first part of the lemma yields
$$
E _ {f _ {0}} K L (Q | | \Pi (\cdot | Y)) \asymp n ^ {\frac {2}{1 + 2 \gamma}} m ^ {- 1} + n ^ {\frac {3 + 2 \gamma}{1 + 2 \gamma}} m ^ {- 1 - 2 \alpha} - O (n ^ {\frac {4}{1 + 2 \gamma}} m ^ {- 3}).
$$
The last term is of strictly smaller order than the first two if  $m \gg n^{\frac{1}{1 + 2\gamma}}$ .
B Additional numerical simulations
We provide here additional simulations to those in Section 5, investigating further the minimal number of inducing variables needed for good pointwise UQ and the sensitivity of the results to misspecification of the noise distribution. To illustrate the pointwise approach to UQ using full posterior inference and SGPRs, one can refer to Figure 1 in the paper.
Threshold for the number of inducing variables. We showed in Sections 3 and 5 that for  $m \gg n^{\frac{1}{1 + 2\gamma} \left(\frac{2 + \alpha}{1 + \alpha}\right)}$  inducing variables, the SGPR behaves similarly to the true posterior for pointwise inference. When  $\alpha = \gamma$ , the threshold for minimax convergence rates for estimation in  $L^2$  is the smaller bound  $m \gg n^{\frac{1}{1 + 2\alpha}}$  [22]. We next investigate how the SGPR behaves near this threshold.
Return to the fixed  $(X_{i} = i / (n + 1 / 2))$  and uniform random  $(X_{i}\sim^{iid}U[0,1])$  design settings on [0, 1] and consider the same Gaussian processes as before: rescaled Brownian motion, Matérn and rescaled square exponential. We consider three different values of  $m$  given by  $m_{-}^{*} = n^{\frac{1}{1 + 2\gamma}} / \log n$ ,  $m_{+}^{*} = n^{\frac{1}{1 + 2\gamma}}\cdot \log n$ , so that these values of  $m$  are just below and above the threshold  $n^{\frac{1}{1 + 2\gamma}}$ , respectively, and the full posterior case  $m = n$ . We consider the 1-Hölder function  $f_0(x) = |x - 1 / 2|$  and 2-Hölder function  $f_0(x) = \mathrm{sign}(x - 1 / 2)|x - 1 / 2|^2$ .
Coverage	Length	RMSE	NLPD
GP	m-+	m+	n	m-+	m+	n	m-+	m+	n	m-+	m+	n
Fixed Design: n = 1000, (α,γ) = (1,0.5)
rBM	1.00	0.98	0.98	0.52	0.42	0.42	0.06	0.09	0.09	-0.85	-0.89	-0.89
Matérn	1.00	0.98	0.98	0.69	0.50	0.50	0.07	0.11	0.11	-0.59	-0.70	-0.70
SE	1.00	0.93	0.93	2.73	0.66	0.66	0.08	0.18	0.18	0.74	-0.28	-0.28
Fixed Design: n = 1000, (α,γ) = (2,1.5)
rBM	0.98	0.98	0.98	0.18	0.17	0.17	0.04	0.04	0.04	-1.72	-1.74	-1.74
Matérn	1.00	0.95	0.95	0.56	0.22	0.22	0.04	0.05	0.05	-0.82	-1.51	-1.51
SE	1.00	0.91	0.91	2.29	0.31	0.31	0.05	0.09	0.09	0.56	-0.99	-0.99
Random Design: n = 500, (α,γ) = (1,0.5)
rBM	1.00	0.97	0.97	0.59	0.50	0.50	0.08	0.10	0.10	-0.70	-0.75	-0.75
Matérn	1.00	0.96	0.96	0.87	0.59	0.59	0.07	0.12	0.12	-0.38	-0.58	-0.58
SE	1.00	0.94	0.94	2.70	0.77	0.77	0.10	0.21	0.21	0.73	-0.13	-0.13
Random Design: n = 500, (α,γ) = (2,1.5)
rBM	0.98	0.95	0.95	0.23	0.23	0.23	0.05	0.05	0.05	-1.49	-1.49	-1.49
Matérn	1.00	0.96	0.96	0.57	0.29	0.29	0.05	0.07	0.07	-0.79	-1.19	-1.19
SE	1.00	0.95	0.95	2.10	0.39	0.39	0.05	0.10	0.10	0.47	-0.86	-0.86
Table 3: Comparison of SGPR and full posterior for  $90\%$  pointwise credible intervals for different values of  $(n,\alpha ,\gamma)$  in dimension  $d = 1$ . The column headers describe how many inducing variables were used, with  $m_{-}^{*}\coloneqq n^{\frac{1}{1 + 2\gamma}} / \log (n)$  and  $m_{+}^{*}\coloneqq n^{\frac{1}{1 + 2\gamma}}\cdot \log (n)$  and the full posterior represented by  $n$ . The hypothesised bound  $n^{\frac{1}{1 + 2\gamma}}$  equals 32, 6, 22 and 5 in the four different regimes (top-to-bottom).
The results of our experiments for different values of  $(n, \alpha, \gamma)$  are given in Table 3; we note here that we exclude estimates of the standard error in this Table due to space constraints, but they are comparable to those presented in Table 1 in the main text. In all cases, one can see that the coverage, length and RMSE are practically indistinguishable between the cases  $m = m_{+}^{*}$  and the full posterior  $m = n$ , suggesting that the choice of  $m \gg n^{\frac{1}{1 + 2\gamma}}$  inducing variables is sufficient to achieve the same performance as the full posterior for pointwise inference. Furthermore, for fixed design, we again see the Brownian motion with both choices  $m = m_{+}^{*}$  and  $m = n$  achieves the expected coverage of 0.98 predicted by our theory.
Conversely, for  $m = m_{-}^{*}$  inducing variables, the coverage and credible interval length was larger in almost all cases, sometimes significantly so. This suggests that below this threshold, the SGPR is no longer a good approximation for the true posterior, matching the findings in [22, 23] for  $L^2$ -type inference. On the other hand, the RMSE is usually reduced, indicating that additional smoothing helps the sparse posterior mean for estimation, which is not surprising since in our simulation choices
the prior undersmooths the truth. The main message is that taking fewer inducing points than this threshold leads to a significant increase in the credible interval width, making the UQ less informative and overly conservative. It is interesting that this threshold shows so clearly, since  $m_{-}^{*}$  and  $m_{+}^{*}$  differ only by a subpolynomial factor.
The same story holds for random design, with some additional fluctuations due to this additional source of randomness. We recall that for fixed design, the credible intervals have deterministic length, whereas for random design these are also now random.
It would be interesting to extend our theoretical results to the threshold  $n^{\frac{1}{1 + 2\gamma}}$  as suggested by these simulations. However, the mismatch between pointwise loss and the  $L^2$ -distance induced by the likelihood makes using standard tools from Bayesian nonparametrics difficult, see [15] for further discussion.
Multidimensional setting. Consider again  $d = 10$  dimensional covariates with the Matérn kernel of smoothness  $1/2$ , and the corresponding threshold  $n^{\frac{d}{d + 2\gamma}}$  needed for the minimax contraction rate in  $L^2$  [22]. We take uniform random design  $X_i \sim^{iid} \mathcal{U}([-1/2, 1/2]^d)$ , true function  $f_0(x) = \|x\|^{\alpha}$  for  $\alpha = 0.9$  and investigate the pointwise  $90\%$  -credible intervals at the point  $x_0 = (0, \ldots, 0)$ . We now take three values of  $m$  given by  $m_{d-}^* = n^{\frac{d}{d + 2\gamma}} / \log n$ ,  $m_{d+}^* = n^{\frac{d}{d + 2\gamma}} \cdot \log n$  and  $m = n$ . The results are presented in Table 4. Indeed, one sees that the variational posterior performs identically to the full posterior once at least  $m_{d+}^* = n^{d/(d + 2\gamma)} \cdot \log n \gg n^{\frac{d}{d + 2\gamma}}$  inducing variables have been used, while  $m_{d-}^*$  inducing variables do not seem to be enough to get the same performance.
Coverage	Length	RMSE	NLPD
md-	md+	n	md-	md+	n	md-	md+	n	md-	md+	n
0.93	1.00	1.00	2.59	2.40	2.40	0.91	0.73	0.73	1.35	1.10	1.10
Misspecified noise. We now take the same fixed design setting as in Section 3 of the main paper, but sample  $Y_{i} = f_{0}(X_{i}) + \varepsilon_{i}$ , where  $\varepsilon_{i}$  are independent standard Laplace random variables. Based on Table 5, the results are much the same as in the well-specified case, so that the credible sets demonstrate some robustness to noise misspecification. Note that while the coverage remains broadly similar, the credible intervals are slightly wider and have larger RMSE reflecting the larger noise.
Table 4: Comparison of SGPR and full posterior with Matérn prior for  $90\%$  pointwise credible intervals for  $(d, \alpha, \gamma) = (10, 0.9, 0.5)$ . The column headers describe how many inducing variables were used with  $m_{d-}^{*} = n^{\frac{d}{d+2\gamma}} / \log(n)$  and  $m_{d+}^{*} = n^{\frac{d}{d+2\gamma}} \cdot \log(n)$  and the full posterior represented by  $n$ . Here,  $n^{\frac{d}{d+2\gamma}} = 284$ .
Smoothness	Coverage	Length	RMSE	NLPD
GP	α	γ	SGPR	GP	SGPR	GP	SGPR	GP	SGPR	GP
rBM	1	0.5	0.98	0.98	0.49	0.49	0.11	0.11	-0.71 (0.15)	-0.71 (0.15)
Matérn	1	0.5	0.98	0.98	0.58	0.58	0.12	0.12	-0.58 (0.17)	-0.58 (0.17)
rBM	0.5	0.5	0.69	0.69	0.49	0.49	0.22	0.22	0.11 (0.32)	0.11 (0.32)
Matérn	0.5	0.5	0.88	0.88	0.58	0.58	0.20	0.20	-0.17 (0.20)	-0.17 (0.20)
Table 5: Misspecified noise. SGPR summary statistics for  $90\%$  pointwise credible intervals for different values of  $(\alpha, \gamma)$  with regular fixed design on  $[0,1]$  with  $n = 1000$  and Laplace noise. The SGPR uses  $m^{*} = n^{\frac{1}{1 + 2\gamma}\frac{2 + \alpha}{1 + \alpha}}$  inducing variables, which is 126 and 316 in the two regimes.
B.1 Definitions of simulation statistics
For completeness, we define here the statistics reported in the simulation sections based on  $M$  Monte-Carlo samples. For  $Y_{i}^{(j)} = f_{0}(X_{i}) + \varepsilon_{i}^{(j)}$ ,  $i = 1,\dots ,n$  and  $j = 1,\ldots ,M$ , let  $C_m^{(j)}(x_0)$  denote the credible interval for  $f(x_0)$ ,  $\bar{f}_m^{(j)}(x_0)$  and  $\sigma_m^{(j)}(x_0)^2$  the posterior mean and variance from the SGPR with  $m$  inducing variables computed from the  $j^{th}$  realisation of the data. We then consider the Monte-Carlo estimates of the coverage, credible interval length, root-mean square error (RMSE)
and negative log-predictive density (NLPD) given by
$$
\widehat {\mathbf {C o v}} = \frac {1}{M} \sum_ {j = 1} ^ {M} \mathbf {1} \left\{f _ {0} \left(x _ {0}\right) \in C _ {m} ^ {(j)} \left(x _ {0}\right) \right\}, \quad \widehat {R M S E} = \sqrt {\frac {1}{M} \sum_ {j = 1} ^ {M} \left(\bar {f} _ {m} ^ {(j)} \left(x _ {0}\right) - f _ {0} \left(x _ {0}\right)\right) ^ {2}},
$$
$$
\widehat {\text {L e n}} = \frac {1}{M} \sum_ {j = 1} ^ {M} \operatorname {L e n g t h} \left(C _ {m} ^ {(j)} (x _ {0})\right), \quad \widehat {N L P D} = - \frac {1}{M} \sum_ {j = 1} ^ {M} \log \mathcal {N} \left(f _ {0} (x _ {0}) | f _ {m} ^ {(j)} (x _ {0}), \sigma_ {m} ^ {(j)} (x _ {0}) ^ {2}\right),
$$
respectively, where  $\mathcal{N}\left(f_0(x_0) | f_m^{(j)}(x_0), \sigma_m^{(j)}(x_0)^2\right)$  denotes the density of the Gaussian distribution with mean  $f_m^{(j)}(x_0)$  and variance  $\sigma_m^{(j)}(x_0)^2$  evaluated at the point  $f_0(x_0)$ .