SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathbf {1} ^ {T} (K ^ {\infty}) ^ {- 1} \mathbf {1} = \frac {m}{\sigma_ {\xi} ^ {2}} \left(1 - \frac {m \sigma_ {f} ^ {2}}{\sigma_ {\xi} ^ {2} + m \sigma_ {f} ^ {2}}\right) \\ = \frac {m}{\sigma_ {\xi} ^ {2}} \left(1 - m \sigma_ {f} ^ {2} \frac {1}{m \sigma_ {f} ^ {2}} \left(1 - \frac {\sigma_ {\xi} ^ {2}}{m \sigma_ {f} ^ {2}} + \frac {\sigma_ {\xi} ^ {4}}{m ^ {2} \sigma_ {f} ^ {4}} - \mathcal {O} (m ^ {- 3})\right)\right) \\ = \frac {1}{\sigma_ {f} ^ {2}} - \frac {\sigma_ {\xi} ^ {2}}{m \sigma_ {f} ^ {4}} + \mathcal {O} (m ^ {- 2}). \tag {11} \\ \end{array}
$$
$$
\lim  _ {n \rightarrow \infty} \boldsymbol {k} _ {N} ^ {* T} K _ {N} ^ {- 1} \boldsymbol {k} _ {N} ^ {*} = \sigma_ {f} ^ {4} \mathbf {1} ^ {T} (K ^ {\infty}) ^ {- 1} \mathbf {1} = \sigma_ {f} ^ {2} - \sigma_ {\xi} ^ {2} m ^ {- 1} + \mathcal {O} (m ^ {- 2}). \tag {12}
$$
Lemma 13 (WSRF expectations). Under (A3),  $\mathrm{E_{y,y^*}}\{\pmb{y}\pmb{y}^*\} = \pmb{k}^*$  and  $\mathrm{E_y}\{\pmb{y}\pmb{y}^T\} = K$
Proof. By assumption on the covariance properties of  $y$  and the independence and zero-mean of the additive noise,  $\mathrm{E_y}\{y_i y_j\} = k(\pmb{x}_i, \pmb{x}_j)$ . Extending this to the joint distribution over  $\pmb{y}, \pmb{y}^*$  is straightforward and gives the results stated.
Lemma 13 is subsequently assumed to be in use throughout A.2.
A.2 Limit proofs
In the following statements only misspecification of type (d) and/or (e) (subsection 4.2) is considered to be at work.
Lemma 14 (MSE limit). Under the assumptions (A1-3), for fixed  $m < \infty$ , the predictive GPnn given in subsection 4.1 converges pointwise in the sense of MSE wrt  $P_{\mathbf{x}}$ -a.e. as
$$
\lim  _ {n \rightarrow \infty} f _ {n} ^ {\mathrm {M S E}} (\boldsymbol {\theta}) = \sigma_ {\xi} ^ {2} (1 + m ^ {- 1}) - \mathcal {O} \big (m ^ {- 2} \big).
$$
Proof. This follows from Lemma 12 by expanding the definition of MSE:
$$
\begin{array}{l} \lim  _ {n \rightarrow \infty} f _ {n} ^ {\mathrm {M S E}} (\boldsymbol {\theta}) = \lim  _ {n \rightarrow \infty} \operatorname {E} _ {\mathbf {y}, \mathrm {y} ^ {*}} \left\{\left| y ^ {*} - \mu_ {N} ^ {*} \right| ^ {2} \right\} \\ = \lim  _ {n \rightarrow \infty} \left[ \mathrm {E} _ {\mathbf {y} ^ {*}} \left\{y ^ {* 2} \right\} + \mathrm {E} _ {\mathbf {y}} \left\{\mu_ {N} ^ {* 2} \right\} - 2 \mathrm {E} _ {\mathbf {y}, \mathbf {y} ^ {*}} \left\{\boldsymbol {k} _ {N} ^ {* T} K _ {N} ^ {- 1} \boldsymbol {y} _ {N} y ^ {*} \right\}\right] \\ = \sigma_ {f} ^ {2} + \sigma_ {\xi} ^ {2} - \lim _ {n \rightarrow \infty} \mathrm {E} _ {\mathbf {y}} \left\{\mu_ {N} ^ {*} ^ {2} \right\} \\ = \sigma_ {\xi} ^ {2} (1 + m ^ {- 1}) - \mathcal {O} \left(m ^ {- 2}\right). \\ \end{array}
$$
Since  $\mathrm{E_y}\{\mu_N^{*2}\} = \mathrm{E_y}\{\pmb{k}_N^{*T}K_N^{-1}\pmb{y}_N\pmb{y}_N^T K_N^{-1}\pmb{k}_N^*\} = \pmb{k}_N^{*T}K_N^{-1}\pmb{k}_N^*$ , and by assumption  $\mathrm{E_{y,y^*}}\{\pmb{y}_Ny^*\} = \pmb{k}_N^*$  even under a  $\mathcal{WSR}\mathcal{F}$  generative model (Lemma 13).
Corollary 15 (NLL limit).
$$
\lim  _ {n \rightarrow \infty} f _ {n} ^ {\mathrm {N L L}} (\boldsymbol {\theta}) = \frac {1}{2} \log \left(\sigma_ {\xi} ^ {2} (1 + m ^ {- 1})\right) + \frac {1}{2} + \frac {1}{2} \log 2 \pi - \mathcal {O} \left(m ^ {- 2}\right).
$$
Proof. The proof follows straightforwardly from Lemma 12 and because  $\sigma_N^{*2} = \sigma_f^2 +\sigma_\xi^2 -$ $\pmb{k}_{N}^{*T}\pmb{K}_{N}^{-1}\pmb{k}_{N}^{*}$
$$
\begin{array}{l} 2 \operatorname {E} _ {\mathbf {y}, \mathrm {y} ^ {*}} \left\{l _ {N} ^ {*} \right\} = \operatorname {E} _ {\mathbf {y}, \mathrm {y} ^ {*}} \left\{\log \sigma_ {N} ^ {* 2} + \frac {(y ^ {*} - \mu_ {N} ^ {*}) ^ {2}}{\sigma_ {N} ^ {* 2}} + \log 2 \pi \right\} \\ = \log \sigma_ {N} ^ {* 2} + 1 + \log 2 \pi \\ \end{array}
$$
$$
\begin{array}{l} \lim  _ {n \rightarrow \infty} 2 \operatorname {E} _ {\mathbf {y}, \mathrm {y} ^ {*}} \left\{l _ {N} ^ {*} \right\} = \log \left(\sigma_ {f} ^ {2} + \sigma_ {\xi} ^ {2} - \left(\sigma_ {f} ^ {2} - \sigma_ {\xi} ^ {2} m ^ {- 1} + \mathcal {O} (m ^ {- 2})\right)\right) + 1 + \log 2 \pi \\ = \log \left(\sigma_ {\xi} ^ {2} \left(1 + m ^ {- 1}\right) - \mathcal {O} \left(m ^ {- 2}\right)\right) + 1 + \log 2 \pi \\ = \log \sigma_ {\xi} ^ {2} + m ^ {- 1} + 1 + \log 2 \pi - \mathcal {O} (m ^ {- 2}). \\ \end{array}
$$
A.2.1 Full misspecification
For the remainder of A.2 we assume that the full range of possible misspecifications ((a)-(e)) outlined in subsection 4.2 are in action. We refer to this case as "fully-misspecified" and introduce the notation  $\hat{\mu}_N^*$ ,  $\hat{\sigma}_N^{*2}$  to be understood to mean the predictive mean and variance under these misspecifications.
Lemma 16 (Fully misspecified MSE limit). For a fully misspecified model, asymptotically
$$
\lim  _ {n \rightarrow \infty} f _ {n} ^ {\mathrm {M S E}} (\hat {\boldsymbol {\theta}}) = \sigma_ {\xi} ^ {2} (1 + m ^ {- 1}) \pm \mathcal {O} \left(m ^ {- 2}\right).
$$
provided the misspecified kernel distance metric is weakly faithful in the sense that the  $m^{th}$  nearest neighbour converges under both the true and misspecified metrics (Definition 10).
$$
\begin{array}{l} \mathrm {E} _ {\mathbf {y}} \left\{\mathrm {E} _ {\mathrm {y} ^ {*}} \left[ (y ^ {*} - \hat {\mu} _ {N} ^ {*}) ^ {2} \mid \mathbf {y} \right] \right\} = \mathrm {E} _ {\mathbf {y}} \left\{\mathrm {E} _ {\mathrm {y} ^ {*}} \left[ y ^ {* 2} - 2 y ^ {*} \hat {\mu} _ {N} ^ {*} + (\hat {\mu} _ {N} ^ {*}) ^ {2} \mid \mathbf {y} \right] \right\} \\ = \mathrm {E} _ {\mathbf {y}} \left\{\sigma_ {N} ^ {* 2} + \mu_ {N} ^ {* 2} - 2 \mu_ {N} ^ {*} \hat {\mu} _ {N} ^ {*} + (\hat {\mu} _ {N} ^ {*}) ^ {2} \right\} \\ = \underbrace {\sigma_ {N} ^ {* 2}} _ {(a)} + \underbrace {\boldsymbol {k} _ {N} ^ {* T} K _ {N} ^ {- 1} \boldsymbol {k} _ {N} ^ {*}} _ {(b)} - 2 \underbrace {\boldsymbol {k} _ {N} ^ {* T} \hat {K} _ {N} ^ {- 1} \hat {\boldsymbol {k}} _ {N} ^ {*}} _ {(c)} + \underbrace {\hat {\boldsymbol {k}} _ {N} ^ {* T} \hat {K} _ {N} ^ {- 1} K _ {N} \hat {K} _ {N} ^ {- 1} \hat {\boldsymbol {k}} _ {N} ^ {*}} _ {(d)}. \\ \end{array}
$$
We can use standard results to state that  $(a) + (b) = \sigma_f^2 +\sigma_\xi^2$  . Then we define  $\hat{\gamma} = \frac{\hat{\sigma}_f^2}{\hat{\sigma}_\xi^2 + m\hat{\sigma}_f^2}$  and expand it in terms of  $m^{-1}$  ..
$$
1 - m \hat {\gamma} = \frac {\hat {\sigma} _ {\xi} ^ {2}}{m \hat {\sigma} _ {f} ^ {2}} - \frac {\hat {\sigma} _ {\xi} ^ {4}}{m ^ {2} \hat {\sigma} _ {f} ^ {4}} + \mathcal {O} (m ^ {- 3}).
$$
In a manner similar to Lemma 12 we use this result to compute:
$$
\begin{array}{l} \lim _ {n \rightarrow \infty} (c) = \sigma_ {f} ^ {2} \mathbf {1} ^ {T} \hat {\sigma} _ {\xi} ^ {- 2} (I - \hat {\gamma} \mathbf {1} ^ {T}) \mathbf {1} \hat {\sigma} _ {f} ^ {2} \\ = \frac {\sigma_ {f} ^ {2} \hat {\sigma} _ {f} ^ {2}}{\hat {\sigma} _ {\xi} ^ {2}} m (1 - m \hat {\gamma}) \\ = \frac {\sigma_ {f} ^ {2} \hat {\sigma} _ {f} ^ {2}}{\hat {\sigma} _ {\xi} ^ {2}} \left(\frac {\hat {\sigma} _ {\xi} ^ {2}}{\hat {\sigma} _ {f} ^ {2}} - \frac {\hat {\sigma} _ {\xi} ^ {4}}{m \hat {\sigma} _ {f} ^ {4}}\right) + \mathcal {O} (m ^ {- 2}) \\ = \sigma_ {f} ^ {2} - \frac {\sigma_ {f} ^ {2} \hat {\sigma} _ {\xi} ^ {2}}{m \hat {\sigma} _ {f} ^ {2}} + \mathcal {O} (m ^ {- 2}) \\ \end{array}
$$
$$
\begin{array}{l} \lim _ {n \to \infty} (d) = \frac {\hat {\sigma} _ {f} ^ {4}}{\hat {\sigma} _ {\xi} ^ {4}} \mathbf {1} ^ {T} (I - \hat {\gamma} \mathbf {1} \mathbf {1} ^ {T}) (\sigma_ {\xi} ^ {2} I + \sigma_ {f} ^ {2} \mathbf {1} \mathbf {1} ^ {T}) (I - \hat {\gamma} \mathbf {1} \mathbf {1} ^ {T}) \mathbf {1} \\ = \frac {\hat {\sigma} _ {f} ^ {4}}{\hat {\sigma} _ {\xi} ^ {4}} \mathbf {1} ^ {T} \left[ \sigma_ {\xi} ^ {2} I + \sigma_ {f} ^ {2} \mathbf {1 1} ^ {T} - 2 \sigma_ {\xi} ^ {2} \hat {\gamma} \mathbf {1 1} ^ {T} + \hat {\gamma} ^ {2} \sigma_ {\xi} ^ {2} m \mathbf {1 1} ^ {T} - 2 \sigma_ {f} ^ {2} \hat {\gamma} m \mathbf {1 1} ^ {T} + \sigma_ {f} ^ {2} \hat {\gamma} ^ {2} m ^ {2} \mathbf {1 1} ^ {T} \right] \mathbf {1} \\ = \frac {\hat {\sigma} _ {f} ^ {4}}{\hat {\sigma} _ {\xi} ^ {4}} m \left(\sigma_ {\xi} ^ {2} + m \sigma_ {f} ^ {2}\right) \left[ 1 - 2 m \hat {\gamma} + m ^ {2} \hat {\gamma} ^ {2} \right] \\ = \frac {\hat {\sigma} _ {f} ^ {4}}{\hat {\sigma} _ {\xi} ^ {4}} m (\sigma_ {\xi} ^ {2} + m \sigma_ {f} ^ {2}) (1 - m \hat {\gamma}) ^ {2} \\ = \frac {\hat {\sigma} _ {f} ^ {4}}{\hat {\sigma} _ {\xi} ^ {4}} m \left(\sigma_ {\xi} ^ {2} + m \sigma_ {f} ^ {2}\right) \left(\frac {\hat {\sigma} _ {\xi} ^ {4}}{m ^ {2} \hat {\sigma} _ {f} ^ {4}} - 2 \frac {\hat {\sigma} _ {\xi} ^ {6}}{m ^ {3} \hat {\sigma} _ {f} ^ {6}} + \mathcal {O} \left(m ^ {- 4}\right)\right) \\ = \sigma_ {f} ^ {2} + \frac {\sigma_ {\xi} ^ {2}}{m} - 2 \frac {\sigma_ {f} ^ {2}}{\hat {\sigma} _ {f} ^ {2}} \frac {\hat {\sigma} _ {\xi} ^ {2}}{m} \pm \mathcal {O} (m ^ {- 2}), \\ \end{array}
$$
where we have used the expansion of  $1 - m\hat{\gamma}$  given earlier. Putting these results together gives
$$
\begin{array}{l} \lim  _ {n \rightarrow \infty} f _ {n} ^ {\mathrm {M S E}} (\hat {\boldsymbol {\theta}}) = \lim  _ {n \rightarrow \infty} [ (a) + (b) - 2 (c) + (d) ] \\ = \sigma_ {f} ^ {2} + \sigma_ {\xi} ^ {2} - 2 \left(\sigma_ {f} ^ {2} - \frac {\sigma_ {f} ^ {2} \hat {\sigma} _ {\xi} ^ {2}}{m \hat {\sigma} _ {f} ^ {2}}\right) + \sigma_ {f} ^ {2} + \frac {\sigma_ {\xi} ^ {2}}{m} - 2 \frac {\sigma_ {f} ^ {2}}{\hat {\sigma} _ {f} ^ {2}} \frac {\hat {\sigma} _ {\xi} ^ {2}}{m} \pm \mathcal {O} (m ^ {- 2}) \\ = \sigma_ {\xi} ^ {2} (1 + m ^ {- 1}) \pm \mathcal {O} (m ^ {- 2}). \\ \end{array}
$$
Lemma 17 (Calibration limit under full misspecification).
$$
\lim _ {n \to \infty} f _ {n} ^ {\mathrm {C A L}} (\hat {\boldsymbol {\theta}}) = \frac {\sigma_ {\xi} ^ {2}}{\hat {\sigma} _ {\xi} ^ {2}} \pm \mathcal {O} \big (m ^ {- 2} \big).
$$
Proof. We use continuity to write
$$
\lim  _ {n \to \infty} \operatorname {E} _ {\mathbf {y}, y ^ {*}} \left\{\frac {(y ^ {*} - \hat {\mu} _ {N} ^ {*}) ^ {2}}{\hat {\sigma} _ {N} ^ {* 2}} \right\} = \left(\lim  _ {n \to \infty} \frac {1}{\hat {\sigma} _ {N} ^ {* 2}}\right) \left(\lim  _ {n \to \infty} f _ {n} ^ {\mathrm {M S E}} (\hat {\boldsymbol {\theta}})\right).
$$
By direct application of Lemma 12  $\hat{\sigma}_N^{*2}\xrightarrow{n\to\infty}\hat{\sigma}_\xi^2(1+m^{-1})-\mathcal{O}\big(m^{-2}\big)$  and thus
$$
\lim  _ {n \to \infty} f _ {n} ^ {\mathrm {C A L}} (\hat {\boldsymbol {\theta}}) = \frac {\sigma_ {\xi} ^ {2}}{\hat {\sigma} _ {\xi} ^ {2}} \pm \mathcal {O} \left(m ^ {- 2}\right).
$$
Corollary 18 (NLL limit under full misspecification).
$$
\lim  _ {n \to \infty} f _ {n} ^ {\mathrm {N L L}} (\hat {\boldsymbol {\theta}}) = \frac {1}{2} \log \left(\hat {\sigma} _ {\xi} ^ {2} (1 + m ^ {- 1})\right) + \frac {1}{2} \frac {\sigma_ {\xi} ^ {2}}{\hat {\sigma} _ {\xi} ^ {2}} + \frac {1}{2} \log 2 \pi \pm \mathcal {O} \left(m ^ {- 2}\right).
$$
Proof. We start with
$$
2 f _ {n} ^ {\mathrm {N L L}} (\hat {\boldsymbol {\theta}}) = \operatorname {E} _ {\mathbf {y}, \mathbf {y} ^ {*}} \left\{\log \hat {\sigma} _ {N} ^ {* 2} + \frac {(y ^ {*} - \hat {\mu} _ {N} ^ {*}) ^ {2}}{\hat {\sigma} _ {N} ^ {* 2}} + \log 2 \pi \right\}.
$$
For the second term we use Lemma 17 so that we have
$$
\lim  _ {n \to \infty} 2 f _ {n} ^ {\mathrm {N L L}} (\hat {\boldsymbol {\theta}}) = \log \hat {\sigma} _ {\xi} ^ {2} + m ^ {- 1} + \frac {\sigma_ {\xi} ^ {2}}{\hat {\sigma} _ {\xi} ^ {2}} + \log 2 \pi \pm \mathcal {O} (m ^ {- 2}).
$$
Proof of Theorem 1. We construct the proof using all of the intermediate results given above. In particular item (i) follows from Lemma 16, item (ii) from Lemma 17 and item (iii) from Corollary 18.
B Validity of Algorithm 1 (Proof of Lemma 4)
Proof of Lemma 4. We prove the result for MSE, the proofs for NLL and calibration being essentially identical.
The proof involves showing that Algorithm 1 is exactly equivalent to Algorithm 1b, an alternative scheme which self-evidently provides valid estimates of MSE on size- $n$  synthetic datasets:
Algorithm 1b Simulation of GPnn Robustness and Limiting Behaviour (Expensive)
1. Generate a size- $n$  set of i.i.d. training  $\pmb{x}$ -values  $X = \{\pmb{x}_i\}_{i=1}^n$  with  $\pmb{x}_i \sim P_{\mathbf{x}}$  ( $X$  is held constant henceforth).
2. Generate  $n^*$  i.i.d. test points  $\pmb{x}_i^* \sim P_{\mathbf{x}}$
3. Generate  $n^*$  separate independent synthetic GP samples  $\pmb{y}_i \in \mathbb{R}^{(n+1)}$  corresponding to each of the values  $(\pmb{x}_i^*, X)$ .
4. From within  $X$  take the  $m$  nearest-neighbours  $N(\pmb{x}_i^*)$  to the test point  $\pmb{x}_i^*$ .
5. Corresponding to each test point  $\pmb{x}_i^*$  evaluate the function  $e_i^{*\prime} = f_i(\pmb{y}_i) = (y_i^* - \mu_N^*(\pmb{y}_i'))^2$ . Where, as defined in Equation 6,  $\mu_N^* = \hat{\pmb{k}}_N^{*T}\hat{K}_N^{-1}\pmb{y}_i'$  with  $N = N(\pmb{x}_i)$  and where  $\pmb{y}_i' \in \mathbb{R}^m$  is formed from the components of  $\pmb{y}_i$  corresponding to  $N = N(\pmb{x}_i)$ .
6. Compute the average  $\frac{1}{n^*}\sum_{i=1}^{n^*}e_i^{*'}$  to obtain the MSE statistic.
Algorithm 1b clearly provides a valid evaluation of the MSE arising from the full GPnn prediction process described in section 3 (albeit at prohibitive expense for large  $n$ ). Hence it is sufficient to prove that Algorithm 1 is equivalent to Algorithm 1b. We do so by applying two minor alterations to Algorithm 1b in succession whose combined effect is to convert it to Algorithm 1. We also show that throughout this process equivalence is maintained with Algorithm 1b thus completing the proof:
Change 1: Let  $\pmb{y}_i = (y_i^*, y_i', y_i'')$  where  $(y_i^*, y_i') \in \mathbb{R}^{(m+1)}$  with  $\pmb{y}_i'$  corresponding to the  $\pmb{x}$ -values in  $N(\pmb{x}_i^*)$ . Since each function  $f_i(y_i)$  in line 5 above only depends on the components  $(y_i^*, y_i')$  of the  $(n+1)$ -long vector  $\pmb{y}_i$  we can equally write  $e_i^{*'} = f_i(y_i^*, y_i')$  and hence truncate each of the vectors  $\pmb{y}_i$  to  $(y_i^*, y_i')$  before evaluating  $e_i^{*'}$  in 5 and this clearly leaves the output of Algorithm 1b completely unchanged.
Change 2: Note (by a standard property of multivariate normal distributions) that each of the truncated vectors  $(y_i^*, y_i')$  are i.i.d. from the multivariate normal distribution whose covariance  $\Sigma$  is the kernel gram matrix corresponding to just  $(x_i^*, N(x_i^*))$ . Hence it is legitimate to change the (inefficient) way of sampling  $(y_i^*, y_i')$  (via the large sample  $\mathbf{y}$ ) to direct sampling from  $\mathcal{N}(\mathbf{0}, \Sigma)$  whilst still maintaining equivalence with Algorithm 1b.
In this way we arrive at Algorithm 1.
C Parameter Calibration (Proof of Lemma 5)
Proof of Lemma 5. (a) Replacing parameters  $\hat{\pmb{\theta}} = (\hat{l},\hat{\sigma}_{\xi}^{2},\hat{\sigma}_{f}^{2})$  with  $\hat{\pmb{\theta}}^{\prime} = (\hat{l},\alpha \hat{\sigma}_{\xi}^{2},\alpha \hat{\sigma}_{f}^{2})$  changes all of the  $\sigma_{i}^{*2}$  values to  $\alpha \sigma_{i}^{*2}$  and therefore changes the calibration value on  $C$  from  $\alpha = \frac{1}{c}\sum_{i = 1}^{c}\frac{(y_i^* - \mu_i^*)^2}{\sigma_i^{*2}}$  to  $\alpha /\alpha = 1$ . (b) The NLL on  $C$  arising from parameters  $(\hat{l},\alpha \hat{\sigma}_{\xi}^{2},\alpha \hat{\sigma}_{f}^{2})$  is  $\frac{1}{2c}\sum_{i = 1}^{c}\{\log \Bigl (\alpha \hat{\sigma}_{\xi}^{2}\Bigr) + (y_{i}^{*} - \mu_{i}^{*})^{2} / (\alpha \sigma_{i}^{*2}) + \log 2\pi)\}$  which, on taking first and second derivatives w.r.t.  $\alpha$ , is found to be uniquely minimised by  $\alpha = \frac{1}{c}\sum_{i = 1}^{c}\frac{(y_i^* - \mu_i^*)^2}{\sigma_i^{*2}}$ . (c) It is easily shown that replacing parameters  $(\hat{\sigma}_{\xi}^{2},\hat{\sigma}_{f}^{2})$  by  $(k\hat{\sigma}_{\xi}^{2},k\hat{\sigma}_{f}^{2})$  (for any  $k > 0$ ) in the formula for  $\mu^{*}$  (Equation 3 and Equation 6) does not
alter  $\mu^{*}$ . Hence the value of  $\mathrm{MSE} = \frac{1}{n^*}\sum_{i=1}^{n^*}(y_i^* - \mu_i^*)^2$  on any size- $n^*$  test set is unchanged when parameters  $\hat{\pmb{\theta}}'$  are used in place of  $\hat{\pmb{\theta}}$ .
D Real world datasets
We consider a variety of datasets from the standard UCI machine learning repository<sup>2</sup>. These datasets are commonly used in the GP literature (see [10] for instance) and are, in principle, easily available online. In practice, we encountered some difficulties: the dataset documentation is often limited; the dataset names commonly used in other published papers do not always match the UCI database naming and important details about data pre-processing, which features to use etc, are often omitted. There are numerous attempts on GitHub and elsewhere at cataloguing these datasets along with any pre-processing, however we had limited success using them, with many appearing unmaintained. Our focus in this work is on testing our methods on a variety of real world datasets and in a way that is, as far as possible, consistent with other papers. We therefore rejected datasets about which there is ambiguity over the correct features to use, or even which column to regress on or for which outlier rejection is required but undocumented elsewhere.
Referring to the datasets used in [10], we were able to locate the following:
- Song (https://archive.ics.uci.edu/ml/machine-learning-databases/00203/YearPredictionMSD.txt.zip)
- Bike (https://archive.ics.uci.edu/ml/machine-learning-databases/00275/Bike-Sharing-Dataset.zip)
- Poletele (https://archive.ics.uci.edu/ml/machine-learning-databases/parkinsons/telemonitoring/parkinsons_updrs.data)
- Keggdirected (https://archive.ics.uci.edu/ml/machine-learning-databases/00220/Relation%20Network%20(Directed).data)
- Keggundirected (https://archive.ics.uci.edu/ml/machine-learning-databases/00221/Reaction%20Network%20(Undirected).data)
- CTSlice (https://archive.ics.uci.edu/ml/machine-learning-databases/00206/slice_localization_data.zip)
- Road3d (https://archive.ics.uci.edu/ml/machine-learning-databases/00246/3D_spatial_network.txt)
- Protein (https://archive.ics.uci.edu/ml/machine-learning-databases/00265/CASP.csv)
- Buzz (https://archive.ics.uci.edu/ml/machine-learning-databases/00248/regression.tar.gz)
- HouseElectric (HouseE) (https://archive-beta.ics.uci.edu/dataset/235/individual+household+electric+power+consumption)
We were unable to find any documentation on the Kegg datasets to indicate which of the columns should be used as the independent variable (the regressor) and neither is this mentioned in any literature of which we are aware. Initial runs of standard exact GP training and prediction produced RMSEs much higher that reported in [10]. Combining these two observations, we chose to exclude both Kegg datasets. Likewise we faced problems with Buzz. An analysis of the  $y$  values revealed a small proportion of extremely large outliers that we found could unduly distort performance results (e.g. depending on whether these outliers appeared in the test set for some of the random splits). With the lack of documentation we were unable to identify an outlier rejection scheme that we were confident would be consistent with results quoted in other papers. For this reason we have excluded Buzz.
The choice of  $(x,y)$  value that we applied for each of the used datasets is as follows:
$^{2}$ https://archive-beta.ics.uci.edu, accessed April 2023.
- Song. The first column is  $y$ , all remaining columns are  $x$ .
- Bike. We use hour.csv. The  $y$  value is cnt. dtdeday (the date) is transformed to just be the integer representation of the day. instant is just an index so is dropped. registered and casual are dropped as registered + casual = cnt.
- Poletele. The  $y$  value is total_UPDRS. The columns subject# and test_time are not relevant to the problem so are dropped.
- CTSlice.  $y$  value is the final column. The first column is dropped as it is just an index. We additionally drop six columns which are constant over the majority of the dataset, namely columns 59, 69, 179, 189, 279 and 351.
- Road3d.  $y$  value is the final column. The first column is dropped as it is just an index.
- Protein. This dataset was processed as per https://github.com/hughsalimbeni/ bayesian_benchmarks, whereafter we used our own random (seeded) train/test split.
- HouseElectric.  $y$  value is the column labelled "Global active power", rescaled by 1000/60 and with "Sub metering 1,2,3" columns subtracted. We convert the date column into day-of-year/365 and the time column into time of day in minutes. Further, we remove any rows with null entries.
We note that although we are using a standard set of real-world datasets, it is not always clear exactly how others in the field have carried out their own preprocessing, limiting the ability to make direct comparisons to other results reported in the literature.
E Additional Implementation Details
E.1 Pre-whitening of Data
For all datasets covered in subsection 7.1 the following "whitening" preprocessing step is adopted: Let  $\mathbf{y}$  be the vector of all regressor values in the training dataset only, and  $X$  the matrix of all regressands in the training dataset only, where each row of  $X$  is a feature. Let  $\mu_y, \sigma_y^2$  be the sample mean and variance of  $\mathbf{y}$  respectively in the training dataset, then the whitened  $y$  values used in both the training and test set are simply  $\sigma_y^{-1}(y - \mu_y)$ . Let  $\boldsymbol{\mu}_X, \Sigma_X$  be the sample mean and covariance matrix of  $X$  respectively. Let  $\Sigma_X = MM^T$ , then the whitened  $x$  values in both the training and test data are  $\frac{1}{\sqrt{d}} M^{-1}(\boldsymbol{x} - \boldsymbol{\mu}_X)$ , where  $d$  is the feature dimension of  $X$ . Note: the performance metrics given in subsection 7.1 are expressed in terms of the whitened  $y$  values rather than the  $y$  values in their original form. This appears to be common practice in the literature and has no bearing on the comparative performance of the different methods within this paper.
E.2 Test-Set Batching
To prevent excessive memory consumption, we perform all predictions for the distributed and variational methods in batches of 1000 points at a time. Where this is not possible (e.g. for especially large datasets), we use smaller batches of 500 or 250 points, as appropriate.
E.3 Additional Implementation Details for SVGP
We use the sparse variational inducing point approach of [14], following the implementation provided by GPyTorch, which in particular uses a Cholesky decomposition to parameterise the covariance matrix of the variational prior. We broadly follow the SVGP implementation example provided by https://docs.gpytorch.ai/en/stable/examples/04_Variational_and_Aproximate_GPs/SVGP_Regression_CUDA.html. In particular, we follow their example in using the Adam optimiser to train our model over 100 epochs with a minibatch size of 1024 and a learning rate of 0.01. We opt to use 1024 inducing points. All experiments under this method are run on a SageMaker ml.p3.2xlarge instance, consisting of a single Tesla V100 GPU with 16GB of memory.
E.4 Additional Implementation Details for Distributed methods