SlowGuess/NeurIPS_2023_OCR huggingface dataset:
A good introduction to distributed methods for Gaussian process inference is [7]. Here we run the product-of-experts (PoE) [15], generalised product-of-experts (gPoE) [2], Bayesian committee machine (BCM) [33], robust Bayesian committee machine (rBCM) [7] and generalised robust Bayesian committee machine (GrBCM) [18] following the recommendation in [4] to aggregate in  $f$ -space. There are three components to any distributed method: the hyperparameter inference, the partitioner and the aggregator. Hyperparameter estimation is the same for all of the methods: we use the method in section 3.1 of [7], randomly partitioning the entire training set into subsets of size 625 (or as close as possible with equal-sized experts given that in general  $n$  is not a multiple of 625). A block diagonal approximation (with  $n / 625$  blocks) is then used to approximate to the full  $n \times n$  gram kernel matrix. To recover hyperparameters with this we use Gaussian Process models with a zero prior mean and a scaled square-exponential kernel. Training is conducted using the Adam optimiser with a learning rate of 0.1 over 100 optimiser iterations. Once the hyperparameters are trained, we run our distributed prediction mechanism to evaluate performance against the test-set. The 625-sized partitioned blocks are referred to as "experts" and the shared hyperparameter values are distributed to each expert and held fixed thereafter. In the aggregator, or distributed prediction phase, each expert produces an individual predictive distribution and these are then aggregated to a final predictive mean and variance for each of our test points. GRBCM prediction is a little more complex than this as it makes use of an additional "communications" expert as explained in [18], aggregating in  $f$ -space as recommended in [4]. We provide timing statistics for training these models.
We use our own GPyTorch-based implementation of distributed GP approximations. All exact GP calculations are performed using GPyTorch using the default settings (so 20 Lanczos iterations throughout and a CG tolerance of 1 for hyperparameter inference, and  $10^{-3}$  for posterior predictions). For all of our experiments, we utilise an AWS t3.2xlarge instance (consisting of 8 Intel Skylake Processors and 32 GB of RAM).
E.5 Reproducibility
All code used to generate tables and figures in this document can be found at https://github. com/ant-stephenson/gpnn-experiments/.
F Related Work
F.1 NNGP
An hierarchical Bayesian approach to nearest neighbour GPs is derived in [5], who construct a full stochastic process allowing an end-to-end probabilistic approach they term 'NNGP' derived from a 'parent' GP using collections of nearest neighbour sets forming a 'reference' set. This work is modified in various ways in [8] including by adaption to a hybrid empirical-Bayes/fully-Bayesian approach to improve scalability. This is described in Algorithm 5 which presents an MCMC-free approach ('conjugate NNGP') where some parameters are estimated via K-fold cross-validation and the remainder are given conjugate priors to allow exact posterior inference. Under this model, the marginal predictive distribution is Student- $t$  with mean and variance expressions that match that given by GPnn up to a scaling factor on the variance. Here we will go into more detail to make this connection explicit. First of all, note that they use an alternative parameterisation to us, specifying an inverse lengthscale  $(\phi)$ , a ratio  $(\alpha)$  of noise variance  $(\tau^2)$  to kernelscale  $(\sigma^2)$  and a vector of coefficients for their linear mean function  $(\beta$ , which we can neglect since we focus exclusively on mean-zero GPs).
In Algorithm 5 they go on to give the following expressions for the predictive mean and variance:
$$
z = M (s, N (s, k))
$$
$$
w = \operatorname {s o l v e} (M [ N (s, k), N (s, k) ], z)
$$
$$
m _ {0} = \widehat {y (s)} = \operatorname {d o t} (x (s), g (k)) + \operatorname {d o t} (w, (y [ N (s, k) ] - \operatorname {d o t} (X [ N (s, k), ], g (k))))
$$
$$
u = x (s) - \operatorname {d o t} (X [ N (s, k), ], w)
$$
$$
v _ {0} = \operatorname {d o t} (u, \operatorname {g e m v} (V (k), u)) + 1 + \alpha - \operatorname {d o t} (w, z)
$$
$$
\widehat {\operatorname {V a r} (y (s))} = b _ {\sigma} ^ {*} (k) v _ {0} / \left(a _ {\sigma} ^ {*} (k) - 1\right)
$$
We can see that the predictive mean  $m_0$  matches our own in the mean-zero setting. In particular, in this case the only non-zero term is  $\mathrm{dot}(w,y[N(s,k)])$  which can be re-written in our notation as  $\pmb{k}_N^{*T}K_N^{-1}\pmb{y}_N$ . Note that this expression is, for fixed  $\alpha$ , independent of choice of kernelscale. This can be seen using the notation given in section 1:  $\pmb{k}_N^{*T}K_N^{-1}\pmb{y}_N = \sigma_f^2\pmb{c}_N^{*T}(\sigma_f^2 C_N + \sigma_\xi^2 I)^{-1}\pmb{y}_N = \pmb{c}_N^{*T}(C_N + \alpha I)^{-1}\pmb{y}_N$ .
In addition to this, the predictive variance is equivalent to ours up to multiplicative scaling  $S = b_{\sigma}^{*}(k) / [\sigma_{f}^{2}(a_{\sigma}^{*}(k) - 1)]$ ; i.e.  $\widehat{\operatorname{Var}(y(s))} = S\sigma_N^{*2}$ , which can be seen as follows: In the mean-zero case we have  $v_0 = 1 + \alpha - \mathrm{dot}(w,z)$  which can again be re-written in our notation as  $1 + \sigma_{\xi}^{2} / \sigma_{f}^{2} - c_{N}^{*T}C_{\alpha ,N}^{-1}c_{N}^{*}$  (where  $C_{\alpha ,N} = C_N + \alpha I$ ). Hence, if we rescale this by a factor of  $\sigma_f^2$  we exactly obtain the GPnn predictive variance, so that  $\widehat{\operatorname{Var}(y(s))} = b_{\sigma}^{*}(k)v_{0} / (a_{\sigma}^{*}(k) - 1) = b_{\sigma}^{*}(k)\sigma_{N}^{*2} / [\sigma_{f}^{2}(a_{\sigma}^{*}(k) - 1)]$  as claimed. Since GPnn uses an additional recalibration step to rescale all predictive variances by a single shared multiplicative factor, the factor  $S$  between the two methods can be made effectively redundant: i.e. if our recalibration step (Algorithm 2) were applied to their method, the two methods would become equivalent with respect to RMSE and weak-calibration (setting aside differences in parameter estimation and their associated costs).
We reemphasise that whilst GPnn and Conjugate NNGP pointwise predictions can be related as above, the latter is given in the form of a  $t$ -distribution which, even with matched first two moments, will have a different shape to that of a Normal distribution. As such, the (Gaussian) NLL performance measure is no longer a valid choice to assess this model. RMSE and weak-calibration remain distribution-agnostic however.
G Further simulation results
Figure 6: Behaviour of performance metrics as functions of kernel hyperparameters for increasing training set sizes  $n$ . The black dashed line denotes the true parameter value; the red dashed line shows the limiting behaviour as  $n \to \infty$  and the green dashed line shows the limiting behaviour when the hyperparameters are correct. Simulations run with  $d = 20, l = 0.5, \sigma_{\xi}^{2} = 0.1, \sigma_{f}^{2} = 0.9$ . Assumed parameters when constant:  $\hat{\sigma}_{\xi}^{2} = 0.2, \hat{\sigma}_{f}^{2} = 0.8, \hat{l} = 0.5$ .
H.1 Results for all distributed methods
Table 3: Results for all methods on all metrics.
Dataset	n	d	Model	Calibration	NLL	RMSE
Bike	1.4e+04	13	BCM	1.02 ± 0.02	1.0 ± 0.0065	0.66 ± 0.0043
GPOE	0.873 ± 0.012	1.03 ± 0.0069	0.664 ± 0.0054
GRBCM	0.893 ± 0.014	0.977 ± 0.0057	0.634 ± 0.004
OURS	0.974 ± 0.087	0.953 ± 0.013	0.624 ± 0.0079
POE	1.03 ± 0.022	1.01 ± 0.0083	0.664 ± 0.0054
RBCM	1.01 ± 0.02	1.0 ± 0.0065	0.659 ± 0.0043
SVGP	0.898 ± 0.011	0.93 ± 0.0043	0.606 ± 0.0033
Ctslice	4.2e+04	378	BCM	5.04 ± 0.28	1.43 ± 0.13	0.311 ± 0.0052
GPOE	0.435 ± 0.013	0.422 ± 0.0015	0.347 ± 0.0027
GRBCM	1.13 ± 0.11	-0.159 ± 0.052	0.237 ± 0.012
OURS	1.04 ± 0.085	-1.26 ± 0.01	0.132 ± 0.00062
POE	6.39 ± 0.27	2.08 ± 0.12	0.347 ± 0.0027
RBCM	4.16 ± 0.25	0.987 ± 0.11	0.28 ± 0.0048
SVGP	0.865 ± 0.026	0.467 ± 0.016	0.384 ± 0.0064
Houseelectric	1.6e+06	8	BCM	1.27 ± 0.0046	-1.33 ± 0.0009	0.0634 ± 3.5e-05
GPOE	0.908 ± 0.0065	-1.43 ± 0.0016	0.0638 ± 7.7e-05
GRBCM	1.25 ± 0.011	-1.34 ± 0.0039	0.063 ± 0.00026
OURS	1.08 ± 0.21	-1.56 ± 0.0065	0.0506 ± 0.00072
POE	1.28 ± 0.006	-1.32 ± 0.0018	0.0638 ± 7.7e-05
RBCM	1.24 ± 0.0054	-1.34 ± 0.0013	0.0626 ± 5.2e-05
SVGP	0.911 ± 0.038	-1.46 ± 0.0046	0.0566 ± 0.00011
Poletele	4.6e+03	19	BCM	1.07 ± 0.029	0.00035 ± 0.019	0.243 ± 0.0048
GPOE	0.917 ± 0.02	0.0344 ± 0.013	0.246 ± 0.0038
GRBCM	0.872 ± 0.024	0.0091 ± 0.015	0.241 ± 0.0033
OURS	1.03 ± 0.073	-0.214 ± 0.019	0.195 ± 0.0042
POE	1.1 ± 0.036	0.00772 ± 0.016	0.246 ± 0.0038
RBCM	1.08 ± 0.029	0.00309 ± 0.018	0.243 ± 0.0048
SVGP	0.862 ± 0.035	-0.0667 ± 0.017	0.226 ± 0.0059
Protein	3.6e+04	9	BCM	1.04 ± 0.0097	1.14 ± 0.003	0.754 ± 0.0022
GPOE	0.925 ± 0.007	1.15 ± 0.0035	0.763 ± 0.0024
GRBCM	0.95 ± 0.012	1.11 ± 0.0051	0.733 ± 0.0038
OURS	0.991 ± 0.029	1.01 ± 0.0016	0.666 ± 0.0014
POE	1.07 ± 0.0088	1.15 ± 0.0033	0.763 ± 0.0024
RBCM	1.03 ± 0.0096	1.13 ± 0.003	0.752 ± 0.0022
SVGP	0.908 ± 0.016	1.05 ± 0.0059	0.688 ± 0.0043
Road3D	3.4e+05	2	BCM	1.01 ± 0.017	0.753 ± 0.007	0.514 ± 0.0035
GPOE	0.756 ± 0.012	0.819 ± 0.0054	0.529 ± 0.0037
GRBCM	0.873 ± 0.011	0.685 ± 0.0041	0.478 ± 0.0023
OURS	0.991 ± 0.041	0.371 ± 0.004	0.351 ± 0.0014
POE	1.07 ± 0.019	0.783 ± 0.0076	0.529 ± 0.0037
RBCM	0.976 ± 0.016	0.735 ± 0.0066	0.505 ± 0.0034
SVGP	0.9 ± 0.00094	0.608 ± 0.018	0.443 ± 0.008
Song	4.6e+05	90	BCM	1.56 ± 0.0063	1.32 ± 0.0012	0.851 ± 6.7e-05
GPOE	0.926 ± 0.00049	1.27 ± 3.4e-05	0.864 ± 7.5e-05
GRBCM	1.61 ± 0.11	1.46 ± 0.058	0.961 ± 0.035
OURS	0.99 ± 0.037	1.18 ± 0.0045	0.787 ± 0.0045
POE	1.61 ± 0.0067	1.34 ± 0.0013	0.864 ± 7.5e-05
RBCM	1.56 ± 0.0062	1.31 ± 0.0011	0.851 ± 6.4e-05
SVGP	0.991 ± 0.02	1.24 ± 0.0012	0.834 ± 0.0011
H Further Results on UCI Datasets
H.2 Performance of different kernels
Table 4: Results on the UCI datasets using different kernel choices for our method and demonstrating the apparent superiority of the exponential kernel in these cases.
Dataset	n	d	Calibration Distributed	Ours (Exp)	Ours (Matérn)	Ours (RBF)	SVGP
Poletele	4.6e+03	19	0.872 ± 0.024	0.994 ± 0.15	0.971 ± 0.13	1.03 ± 0.073	0.862 ± 0.035
Bike	1.4e+04	13	0.893 ± 0.014	0.988 ± 0.098	0.971 ± 0.086	0.974 ± 0.087	0.898 ± 0.011
Protein	3.6e+04	9	0.95 ± 0.012	0.995 ± 0.038	0.993 ± 0.031	0.991 ± 0.029	0.908 ± 0.016
Ctslice	4.2e+04	378	1.13 ± 0.11	0.912 ± 0.071	1.04 ± 0.082	1.04 ± 0.085	0.865 ± 0.026
Road3D	3.4e+05	2	0.873 ± 0.011	1.09 ± 0.065	1.0 ± 0.054	0.991 ± 0.041	0.9 ± 0.00094
Song	4.6e+05	90	1.56 ± 0.0063	0.995 ± 0.033	0.994 ± 0.035	0.99 ± 0.037	0.991 ± 0.02
Houseelectric	1.6e+06	8	1.24 ± 0.0054	1.11 ± 0.29	1.08 ± 0.27	1.08 ± 0.21	0.911 ± 0.038
RMSE
Distributed	Ours (Exp)	Ours (Matérn)	Ours (RBF)	SVGP
Dataset	n	d
Poletele	4.6e+03	19	0.241 ± 0.0033	0.169 ± 0.0076	0.17 ± 0.0076	0.195 ± 0.0042	0.226 ± 0.0059
Bike	1.4e+04	13	0.634 ± 0.004	0.565 ± 0.0036	0.6 ± 0.0044	0.624 ± 0.0079	0.606 ± 0.0033
Protein	3.6e+04	9	0.733 ± 0.0038	0.58 ± 0.0068	0.629 ± 0.004	0.666 ± 0.0014	0.688 ± 0.0043
Ctslice	4.2e+04	378	0.237 ± 0.012	0.123 ± 0.004	0.126 ± 0.0024	0.132 ± 0.00062	0.384 ± 0.0064
Road3D	3.4e+05	2	0.478 ± 0.0023	0.0976 ± 0.013	0.27 ± 0.01	0.351 ± 0.0014	0.443 ± 0.008
Song	4.6e+05	90	0.851 ± 6.7e-05	0.776 ± 0.004	0.778 ± 0.0045	0.787 ± 0.0045	0.834 ± 0.0011
Houseelectric	1.6e+06	8	0.0626 ± 5.2e-05	0.045 ± 0.00025	0.0485 ± 0.0004	0.0506 ± 0.00072	0.0566 ± 0.0001
NLL
Distributed	Ours (Exp)	Ours (Matérn)	Ours (RBF)	SVGP
Dataset	n	d
Poletele	4.6e+03	19	0.0091 ± 0.015	-0.397 ± 0.028	-0.346 ± 0.032	-0.214 ± 0.019	-0.0667 ± 0.017
Bike	1.4e+04	13	0.977 ± 0.0057	0.854 ± 0.004	0.915 ± 0.0077	0.953 ± 0.013	0.93 ± 0.0043
Protein	3.6e+04	9	1.11 ± 0.0051	0.853 ± 0.013	0.95 ± 0.0061	1.01 ± 0.0016	1.05 ± 0.0059
Ctslice	4.2e+04	378	-0.159 ± 0.052	-1.05 ± 0.027	-1.31 ± 0.017	-1.26 ± 0.01	0.467 ± 0.016
Road3D	3.4e+05	2	0.685 ± 0.0041	-0.931 ± 0.14	0.109 ± 0.039	0.371 ± 0.004	0.608 ± 0.018
Song	4.6e+05	90	1.32 ± 0.0012	1.16 ± 0.0046	1.17 ± 0.0051	1.18 ± 0.0045	1.24 ± 0.0012
Houseelectric	1.6e+06	8	-1.34 ± 0.0013	-1.95 ± 0.028	-1.62 ± 0.0095	-1.56 ± 0.0065	-1.46 ± 0.0046
H.3 Prediction times
Table 5: Prediction times (in seconds) for GPnn and SVGP with 400 nearest-neighbours and 1024 inducing points respectively, over a small range of dataset sizes and dimensions.
n	d	GPnn	SVGP
4.2e4	378	0.06	0.02
3.6e4	9	0.02	0.02
1.1e5	50	0.03	0.06
1.1e5	10	0.02	0.06
I Overall Computational Expenditure
Our distributed and variational method experiments were conducted using cloud computing resources. Experiments using our own method have been carried out on an author's laptop. SVGP experiments were run using a SageMaker virtual machine on a single Nvidia Tesla V100 GPU with 16GB memory. Distributed method experiments were run using eight Intel Xeon Platinum 8000 CPU cores (t3.2xlarge EC2 instances).
Below we will attempt to give reasonable indications of the amount of computational work expended to obtain the results in this paper, though note that we are neglecting the work expended in the development and research stages that did not directly contribute to the runs in the paper. As such, the costs presented are representative of the costs of replicating our paper, not repeating the research from scratch. Instead of reporting costs in dollars, we will report approximate computing hours for each instance type. The reader can then estimate their own costs using the current instance costs in the region of their choice, or under other cloud providers or even using on-premise compute.
Dataset	Billed hours (1 GPU, 3 runs)	Billed hours (8 CPUs, 3 runs of 5 methods)
bike	0.027	0.222
ctslice	0.082	0.546
houseelectric	3.713	256.776
poletele	0.010	0.079
protein	0.068	0.680
road3d	0.635	31.674
song	0.904	12.237
This gives a total of around 5.4 hours of compute time on a 1 GPU VM and 302.2 hours on an 8 CPU VM.