SlowGuess/NeurIPS_2023_OCR huggingface dataset:
2. Select the nearby locations which have the highest acquisition function values. To find the top nearby locations, we first try 50k nearby locations and then identify the location with the highest acquisition function value. We repeat this process 30 times, finding and evaluating a total of  $1.5\mathrm{m}$  nearby locations and obtaining a total of 30 top nearby locations.
3. Maximize the acquisition function with gradient-based optimization, using the top nearby locations as initialization. After optimization, the best location becomes  $x_{\mathrm{new}}$ , the maximizer at which the true objective function will be evaluated in the next acquisition step. Initializing at the top nearby locations, we perform 100 steps of Adam on the sampled acquisition function, with a learning rate of 0.001 to find the maximizer.
In every Thompson step, we perform this three step process in parallel for 1000 random acquisition functions sampled from the GP posterior, resulting in a total of  $1000x_{\mathrm{new}}$ , which will be added to the training data and evaluated at the objective function. Note that, although we share the initial nearby locations between sampled acquisition functions, each acquisition function will, in general, produce distinct top nearby locations and maximizers. Figure 7 illustrates a single Thompson step on a 1D problem using 3 acquisition functions, 7 nearby locations and 3 top nearby locations.
Figure 8 provides optimization traces for SGD, CG and their low noise variants on all datasets with under  $50\mathrm{k}$  points. For these, we can compute the exact GP predictions, allowing us to evaluate error with respect to the exact GP in prediction space, representer weight space and in the RKHS. We observe the same trends reported in the main text. SGD decreases its prediction error monotonically, while CG does not. However, both take a similar amount of time to converge on smaller datasets and CG obtains lower error in the end. While SGD makes negligible progress in representer weight space, CG finds the optimal representative weights. CG's time to convergence is greatly increased in the low noise setting, but SGD is practically unaffected.
Table 2 provides quantitative results for inducing point SGD on the HOUSEELECTRIC dataset. SGD's time to convergence is shown to scale roughly linearly in the number of (inducing) points observed. However, for this dataset, keeping only  $10\%$  of observations and thus obtaining  $10\times$  faster
Table 2: Time to convergence and predictive performance for all methods under consideration, including inducing point SGD, on the HOUSEELECTRIC dataset.
Model M	Inducing SGD	Standard SGD 1844352	CG 1844352	SVGP 4096
218782	431489	1099206
RMSE	0.08±0.00	0.08±0.00	0.08±0.01	0.09±0.00	0.87±0.14	0.11±0.01
Minutes	16.7±0.54	24.6±5.40	49.6±10.2	162±54.3	157±0.41	20.0±0.03
NLL	-1.10±0.05	-1.11±0.04	-1.13±0.04	-1.09±0.04	2.07±0.58	-0.89±0.10
Figure 8: Convergence of GP posterior mean with SGD and CG as a function of time (on an A100 GPU) on the POL  $(N\approx 15\mathrm{k})$ , ELEVATORS  $(N\approx 16\mathrm{k})$ , BIKE  $(N\approx 17\mathrm{k})$  and PROTEIN  $(N\approx 46\mathrm{k})$  datasets while setting the noise scale to (i) maximize exact GP marginal likelihood and (ii) to  $10^{-3}$ , labeled low noise. We plot, in left-to-right order, test RMSE, RMSE to the exact GP mean at the test inputs, representer weight error  $\| \pmb {v} - \pmb{v}^{*}\|_{2}$ , and RKHS error  $\| \mu_{f|\pmb{y}} - \mu_{\mathrm{SGD}}\|_{H_k}$ .
Table 3: UCI benchmark with subset of data on the smaller datasets, using a randomly-sampled subset with size that varies as a percentage of the full data.
Dataset N	POL 15000	ELEVATORS 16599	BIKE 17379	PROTEIN 45730	KEGGDIR 48827
RMSE	5%	0.20 ± 0.01	0.44 ± 0.00	0.18 ± 0.01	0.71 ± 0.01	0.11 ± 0.00
10%	0.16 ± 0.00	0.41 ± 0.01	0.13 ± 0.00	0.66 ± 0.00	0.11 ± 0.00
NLL	5%	-0.43 ± 0.02	0.57 ± 0.01	-0.61 ± 0.09	0.99 ± 0.01	-0.76 ± 0.10
10%	-0.66 ± 0.02	0.51 ± 0.01	-1.28 ± 0.07	0.91 ± 0.01	-0.82 ± 0.09
convergence leaves performance unaffected. This suggests the dataset can be summarized well by a small number of points. Indeed, SVGP obtains almost as strong performance as SGD in terms of RMSE with only 4096 inducing points. SVGP's NLL is weaker however, which is consistent with known issues of uncertainty overestimation when using a too small amount of inducing points. On the other hand, the large and potentially redundant nature of this dataset makes the corresponding optimization problem ill-conditioned, hurting CG's performance. Tables 3 and 4 provide additional baseline results for the subset of data approximation, which behaves similarly to SVGP.
Finally, Figure 9 shows all methods' Thompson sampling performance as a function of both compute time and acquisition steps for each individual true function length scale considered in our experiments. Differences among methods are more pronounced in the small compute budget setting. Here, SVGP performs well—on par with SGD—in the large length scale setting, where many observations can likely be summarized with 1024 inducing points. CG suffers from slow convergence due to ill-conditioning here. On the other hand, CG performs on par with SGD in the better-conditioned small length scale setting, while SVGP suffers. In the large compute setting, all methods perform similarly per acquisition step for all length scales except the small one, where SVGP suffers.
Table 4: UCI benchmark with subset of data on the larger datasets, using a randomly-sampled subsets of fixed size which are chosen to reflect typical GPU memory limitations. Numbers which are better than or competitive with respect to Table 1 are printed in boldface.
Dataset N	3DROAD 434874	SONG 515345	BUZZ 583250	HOUSEELEC 2049280
RMSE	25k	0.23 ± 0.00	0.82 ± 0.00	0.33 ± 0.00	0.06 ± 0.01
50k	0.16 ± 0.00	0.81 ± 0.00	0.32 ± 0.00	0.06 ± 0.00
NLL	25k	-0.42 ± 0.01	1.22 ± 0.00	0.30 ± 0.06	-0.53 ± 0.33
50k	-0.78 ± 0.01	1.20 ± 0.00	0.26 ± 0.06	-0.49 ± 0.39
Figure 9: Maximum function values (mean and std. err.) obtained by Thompson sampling with our approximate inference methods as a function of acquisition steps and of compute time on an A100 GPU. The latter includes time spent drawing function samples and finding their maxima. All methods share a starting dataset of  $50\mathrm{k}$  points and we take 30 Thompson steps, acquiring 1000 points in each. Different length scales  $l$  exhibit different behaviors: CG performs better in settings with smaller length scales and better conditioning, SVGP tends to perform better in settings with larger setting and increased smoothness, SGD performs well in both settings.
C Inducing Points
Let  $\mathbf{z} \in X^M$  be a set of  $M > 0$  inducing point locations. The variational inducing point framework of Titsias [43, 44] substitutes our observed targets  $\mathbf{y}$  with the inducing targets  $\mathbf{u} \in \mathbb{R}^M$ . Our Gaussian process, conditional on a set of inducing locations and targets, is given by the mean and covariance functions
$$
\mu_ {f \mid u} (\cdot) = \mathbf {K} _ {(\cdot) z} \mathbf {K} _ {z z} ^ {- 1} \mathbf {u} \quad k _ {f \mid u} (\cdot , \cdot^ {\prime}) = \mathbf {K} _ {(\cdot^ {\prime}, \cdot^ {\prime})} - \mathbf {K} _ {(\cdot) z} \mathbf {K} _ {z z} ^ {- 1} \mathbf {K} _ {z (\cdot^ {\prime})}. \tag {17}
$$
Titsias [43] provides a closed-form expression for the variational distribution  $q = \mathrm{N}(\pmb{\mu}^{(z)},\mathbf{K}^{(z)})$  over inducing targets  $\pmb{u}$  which minimizes the associated Kullback-Leibler divergence, namely
$$
\boldsymbol {\mu} ^ {(z)} = \mathbf {K} _ {z z} \left(\mathbf {K} _ {z z} + \mathbf {K} _ {z x} \boldsymbol {\Sigma} ^ {- 1} \mathbf {K} _ {x z}\right) ^ {- 1} \mathbf {K} _ {z x} \boldsymbol {\Sigma} ^ {- 1} \boldsymbol {y} \tag {18}
$$
$$
\mathbf {K} ^ {(z)} = \mathbf {K} _ {z z} \left(\mathbf {K} _ {z z} + \mathbf {K} _ {z x} \boldsymbol {\Sigma} ^ {- 1} \mathbf {K} _ {x z}\right) ^ {- 1} \mathbf {K} _ {z z}. \tag {19}
$$
In turn, Matthews et al. [30] shows that these expressions lift to a respective infinite-dimensional Kullback-Leibler minimization problem between the variational and true posterior Gaussian processes. Using this, we obtain a variational posterior with mean and covariance functions
$$
\mu_ {f | \boldsymbol {y}} ^ {(z)} (\cdot) = \mathbf {K} _ {(\cdot) z} \left(\mathbf {K} _ {z z} + \mathbf {K} _ {z x} \boldsymbol {\Sigma} ^ {- 1} \mathbf {K} _ {x z}\right) ^ {- 1} \mathbf {K} _ {z x} \boldsymbol {\Sigma} ^ {- 1} \boldsymbol {y} \tag {20}
$$
$$
k _ {f | \boldsymbol {y}} ^ {(z)} (\cdot , \cdot^ {\prime}) = \mathbf {K} _ {(\cdot , \cdot^ {\prime})} + \mathbf {K} _ {(\cdot) z} \left(\left(\mathbf {K} _ {z z} + \mathbf {K} _ {z x} \boldsymbol {\Sigma} ^ {- 1} \mathbf {K} _ {x z}\right) ^ {- 1} - \mathbf {K} _ {z z} ^ {- 1}\right) \mathbf {K} _ {z (\cdot^ {\prime})}. \tag {21}
$$
These expressions will be our starting point. We now write this Gaussian process posterior in a pathwise manner and derive stochastic estimators for the corresponding representative weights.
C.1 Pathwise Representation of the Kullback-Leibler-optimal Variational Distribution
The pathwise expression of Section 3.3 can be derived by restricting the domain of the respective optimization problem which describes the exact posterior samples. This is done by replacing the reproducing kernel Hilbert space with a subset consisting of the span of a set of canonical basis functions centered at the inducing points, as described by Wild et al. [47], Theorem 5 for the posterior mean. Rather than deriving the respective result from scratch, we will verify its correctness by means of computing the relevant means and covariances. Define
$$
\left(f ^ {(z)} \mid \boldsymbol {y}\right) (\cdot) = f (\cdot) + \mathbf {K} _ {(\cdot) z} \mathbf {K} _ {z z} ^ {- 1} \mathbf {K} _ {z x} \left(\mathbf {K} _ {x z} \mathbf {K} _ {z z} ^ {- 1} \mathbf {K} _ {z x} + \boldsymbol {\Sigma}\right) ^ {- 1} \left(\boldsymbol {y} - f ^ {(z)} (\boldsymbol {x}) - \varepsilon\right) \tag {22}
$$
$$
\varepsilon \sim \mathrm {N} (\mathbf {0}, \boldsymbol {\Sigma}) \quad f \sim \operatorname {G P} (0, k) \quad f ^ {(z)} (\cdot) = \mathbf {K} _ {(\cdot) z} \mathbf {K} _ {z z} ^ {- 1} f (z).
$$
We now calculate the moments of this Gaussian process' marginal distributions and show them to match those of the KL-optimal variational Gaussian process given in (20). Write
$$
\begin{array}{l} \mathbb {E} \left[ \left(f ^ {(z)} \mid \boldsymbol {y}\right) (\cdot) \right] = \mathbf {K} _ {(\cdot) z} \mathbf {K} _ {z z} ^ {- 1} \mathbf {K} _ {z x} \left(\mathbf {K} _ {x z} \mathbf {K} _ {z z} ^ {- 1} \mathbf {K} _ {z x} + \boldsymbol {\Sigma}\right) ^ {- 1} \boldsymbol {y} (23) \\ = \mathbf {K} _ {(\cdot) z} \mathbf {K} _ {z z} ^ {- 1} \mathbf {K} _ {z x} \boldsymbol {\Sigma} ^ {- 1} \left(\mathbf {K} _ {x z} \mathbf {K} _ {z z} ^ {- 1} \mathbf {K} _ {z x} \boldsymbol {\Sigma} ^ {- 1} + \mathbf {I}\right) ^ {- 1} \boldsymbol {y} (24) \\ = \mathbf {K} _ {(\cdot) z} \left(\mathbf {K} _ {z x} \boldsymbol {\Sigma} ^ {- 1} \mathbf {K} _ {x z} + \mathbf {K} _ {z z}\right) ^ {- 1} \mathbf {K} _ {z x} \boldsymbol {\Sigma} ^ {- 1} \boldsymbol {y} (25) \\ = \mu_ {f | \boldsymbol {y}} ^ {(\boldsymbol {z})} (\cdot) (26) \\ \end{array}
$$
$$
\begin{array}{l} \operatorname {C o v} \left(\left(f ^ {(z)} \mid \boldsymbol {y}\right) (\cdot) - \mu_ {f \mid \boldsymbol {y}} ^ {(z)} (\cdot)\right) (27) \\ = \mathbb {E} \left(\left(f ^ {(z)} \mid \boldsymbol {y}\right) (\cdot) - \mu_ {f | \boldsymbol {y}} ^ {(z)} (\cdot), \left(f ^ {(z)} \mid \boldsymbol {y}\right) \left(\cdot^ {\prime}\right) - \mu_ {f | \boldsymbol {y}} ^ {(z)} \left(\cdot^ {\prime}\right)\right) (28) \\ = \mathbf {K} _ {(\cdot , \cdot^ {\prime})} - \mathbf {K} _ {(\cdot) z} \mathbf {K} _ {z z} ^ {- 1} \mathbf {K} _ {z x} \left(\mathbf {K} _ {x z} \mathbf {K} _ {z z} ^ {- 1} \mathbf {K} _ {z x} + \boldsymbol {\Sigma}\right) ^ {- 1} \mathbf {K} _ {x z} \mathbf {K} _ {z z} ^ {- 1} \mathbf {K} _ {z (.) ^ {\prime}} (29) \\ = \mathbf {K} _ {(\cdot , \cdot^ {\prime})} + \mathbf {K} _ {(\cdot) z} \mathbf {K} _ {z z} ^ {- 1} (- \mathbf {I} + \mathbf {I} - \mathbf {K} _ {z x} \boldsymbol {\Sigma} ^ {- 1} (\mathbf {K} _ {x z} \mathbf {K} _ {z z} ^ {- 1} \mathbf {K} _ {z x} \boldsymbol {\Sigma} ^ {- 1} + \mathbf {I}) ^ {- 1} \mathbf {K} _ {x z} \mathbf {K} _ {z z} ^ {- 1}) \mathbf {K} _ {z (\cdot^ {\prime})} (30) \\ = \mathbf {K} _ {(\cdot , \cdot^ {\prime})} + \mathbf {K} _ {(\cdot) z} \mathbf {K} _ {z z} ^ {- 1} \left(- \mathbf {I} + \left(\mathbf {K} _ {z x} \boldsymbol {\Sigma} ^ {- 1} \mathbf {K} _ {x z} \mathbf {K} _ {z z} ^ {- 1} + \mathbf {I}\right) ^ {- 1}\right) \mathbf {K} _ {z (\cdot^ {\prime})} (31) \\ = \mathbf {K} _ {(\cdot , \cdot^ {\prime})} + \mathbf {K} _ {(\cdot) z} \left(- \mathbf {K} _ {z z} ^ {- 1} + \left(\mathbf {K} _ {z x} \boldsymbol {\Sigma} ^ {- 1} \mathbf {K} _ {x z} + \mathbf {K} _ {z z}\right) ^ {- 1}\right) \mathbf {K} _ {z (\cdot^ {\prime})} (32) \\ = k _ {f | \boldsymbol {y}} ^ {(z)} (\cdot , \cdot^ {\prime}) (33) \\ \end{array}
$$
which recovers (21), as claimed.
C.2 Derivation of inducing point optimization objectives
We now derive the optimization objectives we use for inducing points.
The MAP objective (12). To find the optimization objective for the inducing point mean function's representative weights, we apply (23) and begin from
$$
\mu_ {f | \boldsymbol {y}} ^ {(z)} (\cdot) = \mathbf {K} _ {(\cdot) z} \left(\mathbf {K} _ {\boldsymbol {z x}} \boldsymbol {\Sigma} ^ {- 1} \mathbf {K} _ {\boldsymbol {x z}} + \mathbf {K} _ {\boldsymbol {z z}}\right) ^ {- 1} \mathbf {K} _ {\boldsymbol {z x}} \boldsymbol {\Sigma} ^ {- 1} \boldsymbol {y} = \mathbf {K} _ {(\cdot) z} \boldsymbol {v} ^ {*}. \tag {34}
$$
Now, we recognize  $(\mathbf{K}_{zx}\pmb{\Sigma}^{-1}\mathbf{K}_{xz} + \mathbf{K}_{zz})^{-1}\mathbf{K}_{zx}\pmb{\Sigma}^{-1}\pmb{y} = \pmb{v}^{*}$  as the expression for the optimizer of a ridge-regularized linear regression problem—see Bishop [7], Chapter 3—with parameters  $\pmb{v}$ , features  $\mathbf{K}_{xz}$ , Gaussian noise of covariance  $\pmb{\Sigma}$ , and regularizer  $\mathbf{K}_{zz}$ . This means that its respective optimization problem is
$$
\underset {\boldsymbol {v} \in \mathbb {R} ^ {M}} {\arg \min } \| \boldsymbol {y} - \mathbf {K} _ {\boldsymbol {x z}} \boldsymbol {v} \| _ {\boldsymbol {\Sigma} ^ {- 1}} ^ {2} + \| \boldsymbol {v} \| _ {\mathbf {K} _ {z z}} ^ {2}. \tag {35}
$$
The sampling objective (13). For the uncertainty reduction term's representative weights with inducing points, we also leverage (23), giving
$$
\begin{array}{l} \mathbf {K} _ {(\cdot) z} \mathbf {K} _ {z z} ^ {- 1} \mathbf {K} _ {z x} \left(\mathbf {K} _ {x z} \mathbf {K} _ {z z} ^ {- 1} \mathbf {K} _ {z x} + \boldsymbol {\Sigma}\right) ^ {- 1} \left(f ^ {(z)} (\boldsymbol {x}) + \varepsilon\right) (36) \\ = \mathbf {K} _ {(\cdot) z} \left(\mathbf {K} _ {z x} \boldsymbol {\Sigma} ^ {- 1} \mathbf {K} _ {x z} + \mathbf {K} _ {z z}\right) ^ {- 1} \mathbf {K} _ {z x} \boldsymbol {\Sigma} ^ {- 1} \left(f ^ {(z)} (\boldsymbol {x}) + \varepsilon\right) (37) \\ = \mathbf {K} _ {(\cdot) z} \boldsymbol {\alpha} ^ {*}. (38) \\ \end{array}
$$
This is again therefore the solution to a ridge-regularized quadratic problem, but now the targets are given by the random variable  $(f^{(z)}(\pmb{x}) + \varepsilon)$ . We thus write the objective
$$
\underset {\boldsymbol {\alpha} \in \mathbb {R} ^ {M}} {\arg \min } \| f ^ {(z)} (\boldsymbol {x}) + \varepsilon - \mathbf {K} _ {\boldsymbol {x} z} \boldsymbol {\alpha} \| _ {\Sigma^ {- 1}} ^ {2} + \| \boldsymbol {\alpha} \| _ {\mathbf {K} _ {z z}} ^ {2}. \tag {39}
$$
C.3 On the Error in the Nyström Approximation  $\mathbf{K}_{xz}\mathbf{K}_{zz}^{-1}\mathbf{K}_{zx}\approx \mathbf{K}_{xx}$
Figure 10 compares the KL-optimal inducing point posterior GP with that obtained when applying the approximation presented in Section 3.3. That is, taking the prior function samples which we fit with the representative weighed canonical basis functions to be  $f(x)$  with  $f \sim \mathrm{GP}(0,k)$  instead of  $f^{(z)}(x) = \mathbf{K}_{xz}\mathbf{K}_{zz}^{-1}f(z)$ . This amounts to approximating the Nyström-type matrix  $\mathbf{K}_{xz}\mathbf{K}_{zz}^{-1}\mathbf{K}_{zx}$  with its exact-posterior counterpart  $\mathbf{K}_{xx}$ . The difference between them is exactly equal to the posterior covariance: by posterior contraction, both of these matrices become very similar if there is an inducing point placed sufficiently close to every data point. In practice, this tends to occur when an inducing point is placed within roughly a half-length-scale of every observation. This is effectively what is needed for inducing point methods to provide a good approximation of the exact GP. This is reflected in Figure 10, where we see that our approximate inducing point posterior differs from the exact inducing point posterior only in situations where the latter fails to be a good approximation to the exact GP in the first place. This manifests as the approximate method providing larger error bars. When the number of inducing points increases, both methods become indistinguishable from each other and the exact GP. Fortunately, the linear cost of SGD in the number of inducing points allows us to use a very large number of these in practice.
D Low-variance Sampling Objectives
Here, we show that the modified sampling objective (10) matches the standard sample-then-optimize objective (9) up to a constant. We then compare both objectives' minibatch gradient variance.
D.1 Standard and Modified Sample-then-Optimize Objectives: Equivalent Gradients
Let  $\mathbf{LL}^T = \boldsymbol{\Sigma}$  be the Cholesky factorization of the noise covariance, let  $f(\pmb{x})\sim \mathrm{N}(\mathbf{0},\mathbf{K}_{\pmb{x}\pmb{x}})$ , and let  $\pmb{s}\sim \mathrm{N}(\mathbf{0},\mathbf{I})$ . Consider the two objectives at hand, namely
$$
\left\| f (\boldsymbol {x}) + \mathbf {L} s - \mathbf {K} _ {\boldsymbol {x x}} \boldsymbol {\alpha} \right\| _ {\boldsymbol {\Sigma} ^ {- 1}} ^ {2} + \left\| \boldsymbol {\alpha} \right\| _ {\mathbf {K} _ {\boldsymbol {x x}}} ^ {2} \tag {40}
$$
Figure 10: Comparison of exact and approximate inducing point posteriors for a GP with squared exponential kernel and  $10\mathrm{k}$  data points generated using the true regression function  $\sin (2x) + \cos (5x)$  under two different data-generation schemes: infill asymptotics, which considers  $x_{i} \sim \mathrm{N}(0,1)$ , and large-domain asymptotics, which considers  $x_{i}$  on an evenly spaced grid with fixed spacing. We see that the approximation needed to apply inducing points is only inaccurate in situations where the inducing point posterior itself has significant error, which generally manifests itself as error bars that are larger than those of the exact posterior.
$$
\left\| f (\boldsymbol {x}) - \mathbf {K} _ {\boldsymbol {x x}} \boldsymbol {\alpha} \right\| _ {\boldsymbol {\Sigma} ^ {- 1}} ^ {2} + \left\| \boldsymbol {\alpha} - \mathbf {L} ^ {- T} \boldsymbol {s} \right\| _ {\mathbf {K} _ {\boldsymbol {x x}}} ^ {2}. \tag {41}
$$
We show these are equal up to a constant by showing they have the same derivatives. Taking derivative with respect to  $\alpha$ , we have
$$
\begin{array}{l} \frac {\partial}{\partial \boldsymbol {\alpha}} \| f (\boldsymbol {x}) + \mathbf {L} s - \mathbf {K} _ {\boldsymbol {x x}} \boldsymbol {\alpha} \| _ {\boldsymbol {\Sigma} ^ {- 1}} ^ {2} + \| \boldsymbol {\alpha} \| _ {\mathbf {K} _ {\boldsymbol {x x}}} ^ {2} (42) \\ = - 2 \mathbf {K} _ {\boldsymbol {x x}} \boldsymbol {\Sigma} ^ {- 1} (f (\boldsymbol {x}) + \mathbf {L} s - \mathbf {K} _ {\boldsymbol {x x}} \boldsymbol {\alpha}) + 2 \mathbf {K} _ {\boldsymbol {x x}} \boldsymbol {\alpha} (43) \\ = - 2 \mathbf {K} _ {\boldsymbol {x x}} \left(\boldsymbol {\Sigma} ^ {- 1} f (\boldsymbol {x}) - \boldsymbol {\Sigma} ^ {- 1} \mathbf {K} _ {\boldsymbol {x x}} \boldsymbol {\alpha} + \mathbf {L} ^ {- T} \boldsymbol {s} - \boldsymbol {\alpha}\right), (44) \\ \end{array}
$$
$$
\begin{array}{l} \frac {\partial}{\partial \boldsymbol {\alpha}} \| f (\boldsymbol {x}) - \mathbf {K} _ {\boldsymbol {x x}} \boldsymbol {\alpha} \| _ {\boldsymbol {\Sigma} ^ {- 1}} ^ {2} + \left\| \boldsymbol {\alpha} - \mathbf {L} ^ {- T} \boldsymbol {s} \right\| _ {\mathbf {K} _ {\boldsymbol {x x}}} ^ {2} (45) \\ = - 2 \mathbf {K} _ {\boldsymbol {x x}} \boldsymbol {\Sigma} ^ {- 1} (f (\boldsymbol {x}) - \mathbf {K} _ {\boldsymbol {x x}} \boldsymbol {\alpha}) + 2 \mathbf {K} _ {\boldsymbol {x x}} (\boldsymbol {\alpha} - \mathbf {L} ^ {- T} \boldsymbol {s}) (46) \\ = - 2 \mathbf {K} _ {\boldsymbol {x x}} \left(\boldsymbol {\Sigma} ^ {- 1} f (\boldsymbol {x}) - \boldsymbol {\Sigma} ^ {- 1} \mathbf {K} _ {\boldsymbol {x x}} \boldsymbol {\alpha} + \mathbf {L} ^ {- T} \boldsymbol {s} - \boldsymbol {\alpha}\right) (47) \\ \end{array}
$$
respectively. These expressions match, giving the claim.
D.2 Minibatch Gradient Variance
Following Antorán et al. [3], we consider single sample minibatch gradient estimators for the data fit term in (7), that is  $D = 1$ . For each of the two sampling objectives under consideration, the gradient estimators are
$$
\partial L _ {\text {o l d}} = - N \mathbb {E} _ {i \sim U (1, \dots , N)} \mathbf {K} _ {\boldsymbol {x}, x _ {i}} \left(f \left(x _ {i}\right) + \varepsilon_ {i} - \mathbf {K} _ {x _ {i}, \boldsymbol {x}} \boldsymbol {\alpha}\right) \tag {48}
$$
$$
\partial L _ {\text {n e w}} = - N \mathbb {E} _ {i \sim U (1, \dots , N)} \mathbf {K} _ {\boldsymbol {x}, x _ {i}} \left(f \left(x _ {i}\right) - \mathbf {K} _ {x _ {i}, \boldsymbol {x}} \boldsymbol {\alpha}\right) \tag {49}
$$
for (9) and (10), respectively. Since both objectives use the same Fourier feature estimator for the gradients of the regularizer, we omit these from our analysis. The variances for single sample gradient estimators of both expressions are given by
$$
\operatorname {C o v} \left(\partial L _ {\text {o l d}}\right) = N \operatorname {C o v} \left(\mathbf {K} _ {\boldsymbol {x x}} (f (\boldsymbol {x}) + \varepsilon - \mathbf {K} _ {\boldsymbol {x}, \boldsymbol {x}} \boldsymbol {\alpha})\right) \tag {50}
$$
$$
\operatorname {C o v} \left(\partial L _ {\text {n e w}}\right) = N \operatorname {C o v} \left(\mathbf {K} _ {\boldsymbol {x x}} \left(f (\boldsymbol {x}) - \mathbf {K} _ {\boldsymbol {x}, \boldsymbol {x}} \boldsymbol {\alpha}\right)\right), \tag {51}
$$
respectively. Expanding both expressions and subtracting them we arrive at
$$
\operatorname {C o v} \left(\partial L _ {\text {o l d}}\right) - \operatorname {C o v} \left(\partial L _ {\text {n e w}}\right) = N \operatorname {C o v} \left(\mathbf {K} _ {\boldsymbol {x x}} \varepsilon\right) - 2 N \operatorname {C o v} \left(\mathbf {K} _ {\boldsymbol {x x}} \mathbf {K} _ {\boldsymbol {x x}} \boldsymbol {\alpha}, \mathbf {K} _ {\boldsymbol {x x}} \varepsilon\right). \tag {52}
$$
Whether our proposed sampling objective presents lower variance rests on the positivity of the above matrix. To analyze this, we must give concrete values to  $\alpha$ . Following Antorán et al. [3], we consider two settings, initialization and convergence.
(I) Initialization. Here,  $\alpha = 0$  and therefore
$$
\operatorname {C o v} \left(\partial L _ {\text {o l d}}\right) - \operatorname {C o v} \left(\partial L _ {\text {n e w}}\right) = N \operatorname {C o v} \left(\mathbf {K} _ {\boldsymbol {x x}} \boldsymbol {\varepsilon}\right) = N \left(\mathbf {K} _ {\boldsymbol {x x}} \boldsymbol {\Sigma} \mathbf {K} _ {\boldsymbol {x x}}\right) \tag {53}
$$
is a positive definite matrix. Thus, our proposed objective has lower variance.
(II) Convergence. Here,  $\pmb {\alpha} = (\mathbf{K}_{\pmb {x}\pmb{x}} + \pmb {\Sigma})^{-1}(f(\pmb {x}) + \pmb {\varepsilon})$  , so we have