SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $\Phi^r (x) = \Pi^{r + 1}(\phi_{u,r}(x)\otimes x^{\otimes r})$ ,  $\Pi^r$  is a TREESKETCH (see appendix C) and  $\phi_{u,r}(x)$  is the random feature expansion for the prefactor defined in Lemma 4.5. TREESKETCH (Ahle et al., 2020), must be instantiated with the base sketches OSNAP (Nelson and Nguyen, 2013) at the leaves  $(T_{\mathrm{base}})$ , and TENSORSRHT (Ahle et al., 2020) at internal nodes  $(S_{\mathrm{base}})$ . The choices for  $R$ , and the dimensions of  $\Pi^r$  and  $\phi_{u,r}$  will be made explicit in the proof.
For each  $r \in [R]$ , define matrices  $K^r$ ,  $\tilde{K}^r$ , and  $\widehat{K}^r$  in  $\mathbb{R}^{n \times n}$  by
$$
K _ {i, j} ^ {r} = \left(\frac {x ^ {(i)} \cdot x ^ {(j)}}{\| x ^ {(i)} \| ^ {2} + \| x ^ {(j)} \| ^ {2}}\right) ^ {r},
$$
$$
\tilde {K} _ {i, j} ^ {r} = \left(x ^ {(i) \otimes r} \otimes \phi_ {u, r} \left(x ^ {(i)}\right)\right) \cdot \left(x ^ {(i) \otimes r} \otimes \phi_ {u, r} \left(x ^ {(i)}\right)\right),
$$
$$
\widehat {K} _ {i, j} ^ {r} = \Phi_ {r} (x ^ {(i)}) \cdot \Phi_ {r} (x ^ {(j)}),
$$
Then, by the triangle inequality, we have
$$
\begin{array}{l} \| \widehat {K} - K \| _ {\mathrm {o p}} \leq \sum_ {r = 1} ^ {R} \| \widehat {K} ^ {r} - \tilde {K} ^ {r} \| _ {\mathrm {o p}} + \sum_ {r = 1} ^ {R} \| \tilde {K} ^ {r} - K ^ {r} \| _ {\mathrm {o p}} + \left\| \sum_ {r = 1} ^ {R} K ^ {r} - K \right\| _ {\mathrm {o p}} \\ \leq \sum_ {r = 1} ^ {R} \| \widehat {K} ^ {r} - \tilde {K} ^ {r} \| _ {\mathrm {o p}} + \sum_ {r = 1} ^ {R} \| \tilde {K} ^ {r} - K ^ {r} \| _ {\mathrm {o p}} + \sum_ {r = R + 1} ^ {\infty} \| K ^ {r} \| _ {\mathrm {o p}}. \tag {23} \\ \end{array}
$$
We shall find high-probability bounds for each of the terms on the right in turn. For the last term, we can choose  $R = C \log (\tilde{\mathrm{sr}} (K)\varepsilon^{-1})$  with  $C$  large enough, and apply the following upper bound
$$
\begin{array}{l} \sum_ {r = R + 1} ^ {\infty} \| K ^ {r} \| _ {\mathrm {o p}} \leq \sum_ {r = R + 1} ^ {\infty} \| K ^ {r} \| _ {F} \\ \leq \sum_ {r = R + 1} n \left(\max _ {i, j \in [ n ]} \frac {\| x ^ {(i)} \cdot x ^ {(j)} \| ^ {r}}{(\| x ^ {(i)} \| ^ {2} + \| x ^ {(j)} \| ^ {2}) ^ {r}}\right) ^ {1 / 2} \\ \leq \sum_ {r = R + 1} n 2 ^ {- r / 2} \\ \leq n 2 ^ {- (R - 1) / 2} \leq \frac {n \varepsilon}{3 \tilde {\mathrm {s r}} (K)} \leq \frac {\varepsilon \| K \| _ {\mathrm {o p}}}{3}, \tag {24} \\ \end{array}
$$
where the final inequality follows from the fact that  $\tilde{\mathrm{sr}}(K) = n / \| K\|_{\mathrm{op}}$ , as the kernel matrix  $K$  has ones on the diagonal.
The terms in the second sum of (23) may be written as
$$
\| \tilde {K} ^ {r} - K ^ {r} \| _ {\mathrm {o p}} = \| \tilde {E} \odot K ^ {r} \| _ {\mathrm {o p}}
$$
where  $\odot$  denotes the Hadamard product and  $E$  is the error matrix defined in (10). Noting that  $E$  is symmetric, we can apply Corollary 11 in Ando et al. (1987) to assert that
$$
\| \tilde {K} ^ {r} - K ^ {r} \| _ {\mathrm {o p}} \leq \max  _ {i} | E _ {i, i} | \| K ^ {r} \| _ {\mathrm {o p}}.
$$
By the triangle inequality, for each  $r \geq 1$ , we have  $\| K^r\|_{\mathrm{op}} \leq \| K\|_{\mathrm{op}}$ . Hence, with the choice  $M = \Omega (R\zeta^{-1}\varepsilon^{-1})$  for the dimension of  $\phi_{u,r}$ , and applying Lemma 4.6 we obtain
$$
\sum_ {r = 1} ^ {R} \| \tilde {K} ^ {r} - K ^ {r} \| _ {\mathrm {o p}} \leq \frac {\varepsilon \| K \| _ {\mathrm {o p}}}{3}. \tag {25}
$$
Finally, we turn to the first term in (23). Let  $A_r$  be a matrix with columns  $x^{(i)\otimes r} \otimes \phi_{u,r}(x^{(i)})$  for  $i = 1,\dots ,n$ . We can rewrite the error in question
$$
\| \widehat {K} ^ {r} - \tilde {K} ^ {r} \| _ {\mathrm {o p}} = \| \left(\Pi^ {r + 1} A _ {r}\right) ^ {T} \Pi^ {r + 1} A _ {r} - A _ {r} ^ {T} A _ {r} \| _ {\mathrm {o p}}
$$
In Lemma D.3, we establish that  $\| A_r\| _F^2\leq (1 + \varepsilon)\mathrm{Tr}(K)$  and  $\| A_r\|_{\mathrm{op}}^2\leq (1 + \varepsilon)\| K\|_{\mathrm{op}}$  . Let  $\delta = 1 / \mathrm{poly}(n)$  denote the error tolerance. Choose the dimension of the sketch  $\Pi^{r + 1}$  to be  $m_r = \Omega \left(\frac{R^2r^4\tilde{\mathrm{s}}\Gamma(K)}{\varepsilon^2}\log^3\left(\frac{n(d\lor M)}{\varepsilon\delta}\right)\right)$  and the sparsity parameter of OSNAP as  $s = \frac{R^2r^4}{\varepsilon^2}\log^3\left(\frac{n(d\lor M)}{\varepsilon\delta}\right)$  Then, Lemmata 32-34 in Ahle et al. (2020) ensure that,
$$
\mathbb {P} \left(\| \widehat {K} ^ {r} - \tilde {K} ^ {r} \| _ {\mathrm {o p}} \geq \frac {\varepsilon \| K \| _ {\mathrm {o p}}}{3 R}\right) = \mathbb {P} \left(\| (\Pi^ {r + 1} A _ {r}) ^ {T} \Pi^ {r + 1} A _ {r} - A _ {r} ^ {T} A _ {r} \| _ {\mathrm {o p}} \geq \frac {\varepsilon \| K \| _ {\mathrm {o p}}}{3 R}\right) \leq \frac {1}{\operatorname {p o l y} (n)}.
$$
Taking a union bound, we obtain
$$
\mathbb {P} \left(\sum_ {r = 1} ^ {R} \| \hat {K} ^ {r} - \tilde {K} ^ {r} \| _ {\mathrm {o p}} \geq \frac {\varepsilon \| K \| _ {\mathrm {o p}}}{3}\right) \leq \frac {1}{\operatorname {p o l y} (n)}. \tag {26}
$$
Finally, combining the bounds (26), (25), and (24), we obtain
$$
\mathbb {P} \left(\| \widehat {K} - K \| _ {\mathrm {o p}} \geq \varepsilon \| K \| _ {\mathrm {o p}}\right) \leq \frac {1}{\operatorname {p o l y} (n)}.
$$
Dimension of the sketch  $\Phi$ : The total dimension of the sketch is  $m = m_1 + \dots + m_R$ , where  $m_r = \Omega\left(\frac{R^2r^4\tilde{\mathrm{sr}}(K)}{\varepsilon^2}\log^3\left(\frac{n(d\vee M)}{\varepsilon\delta}\right)\right)$  with  $R = C\log (\tilde{\mathrm{sr}} (K)\varepsilon^{-1})$ . Hence, ignoring polylogarithmic factors, the sketch has dimension  $m = \tilde{\Omega} (\tilde{\mathrm{sr}} (K) / \varepsilon^2)$ .
Runtime: We first estimate the runtime of applying  $\Phi^r$ . Applying OSNAP with sparsity parameter  $s$  to a vector  $w \in \mathbb{R}^d$  takes  $O(s\mathrm{nnz}(w))$  time. Thus, applying  $r$  independent OSNAP sketches to each  $x^{(i)}$  for  $i \in [n]$  takes  $O(sr\mathrm{nnz}(X))$  time. Applying OSNAP to  $\phi_{u,r}(x^{(i)})$  for  $i \in [n]$  takes  $O(nsM)$  time. Then, applying one TENSORSRHT sketch takes  $O(m_r\log m_r)$  time, and therefore applying all the necessary copies of  $S_{\mathrm{base}}$  requires  $O(rm_r\log m_r)$ . In total, the sketch  $\Phi^a tior$  may be computed in  $O(nrm_r\log m_r + s\mathrm{nnz}(X) + nsM)$ . Adding these runtimes over  $r \in [R]$  gives us a total runtime which is  $\tilde{O}(\mathrm{sr}(K)n\varepsilon^{-2} + \mathrm{nnz}(X)\varepsilon^{-2} + n\zeta^{-1}\varepsilon^{-3})$ .
Lemma D.3. For each  $r = 1,\dots ,R$  we have  $\| A_r\|_{op}^2\leq (1 + \varepsilon)\| K\|_{op}$  and  $\| A_r\| _F^2\leq (1 + \varepsilon)\mathrm{Tr}(K)$
Proof. Note that  $A_r^T A_r = \tilde{K}^r$ . Hence, we aim to show that  $\| \tilde{K}^r \|_{\mathrm{op}} \leq (1 + \varepsilon) \| K \|_{\mathrm{op}}$ , and that  $\operatorname{Tr}(\tilde{K}^r) \leq (1 + \varepsilon) \operatorname{Tr}(K)$ . By the triangle inequality,
$$
\| \tilde {K} ^ {r} \| _ {\mathrm {o p}} \leq \| \tilde {K} ^ {r} \| _ {\mathrm {o p}} + \| \tilde {K} ^ {r} - K ^ {r} \| _ {\mathrm {o p}} \leq (1 + \varepsilon) \| K ^ {r} \| _ {\mathrm {o p}} \leq (1 + \varepsilon) \| K \| _ {\mathrm {o p}},
$$
where the penultimate inequality was shown in the proof of Theorem 4.7, and the final inequality is due to the fact that  $K^r \preceq K$ .
Then, by linearity of the trace,
$$
\operatorname {T r} \left(\tilde {K} ^ {r}\right) = \operatorname {T r} \left(K ^ {r}\right) + \operatorname {T r} \left(\tilde {K} ^ {r} - K ^ {r}\right) = \operatorname {T r} \left(K ^ {r}\right) + \operatorname {T r} \left(E \odot K ^ {r}\right). \tag {27}
$$
And applying once more Corollary 11 of Ando et al. (1987), we have
$$
\operatorname {T r} (E \odot K ^ {r}) = \sum_ {i = 1} ^ {n} \sigma_ {i} (E \odot K ^ {r}) \leq \max  _ {i \in [ n ]} | E _ {i, i} | \sum_ {i = 1} ^ {n} \sigma_ {i} (K ^ {r}) \leq \varepsilon \sum_ {i = 1} ^ {n} \sigma_ {i} (K ^ {r}),
$$
where  $\sigma_{i}(\cdot)$  denotes the  $i$ th singular value. Using once more that  $K^{r} \preceq K$  and hence  $\sigma_{i}(K^{r}) \leq \sigma_{i}(K)$  for all  $i \in [n]$ , we obtain,
$$
\operatorname {T r} (E \odot K ^ {r}) \leq \varepsilon \operatorname {T r} (K).
$$
Plugging this into (27) yields  $\mathrm{Tr}(\tilde{K}^r)\leq (1 + \varepsilon)\mathrm{Tr}(K)$
D.5 Rank of Tanimoto Kernels
Lemma D.4. There does not exist an exact finite-dimensional feature map for either  $T_{MM}$  or  $T_{DP}$ .
Proof. We will use a proof by contradiction, first focusing on  $T_{MM}$ . The setup for the contradiction is as follows: suppose there existed an exact feature map  $f: \mathbb{R}^d \mapsto \mathbb{R}^M$  such that  $T_{MM}(x,y) = f(x) \cdot f(y) \forall x,y \in \mathbb{R}^d$ . This would imply that any kernel matrix between  $n$  points  $X$  could be written as an inner product
$$
T _ {M M} (X, X) = f (X) ^ {T} f (X).
$$
Because  $f$  outputs  $M$  dimensional vectors this would imply the resulting matrix has rank at most  $M$ . Therefore, under this hypothesis it should not be possible to form a kernel matrix of more than  $M$  inputs which is full-rank (invertible). Our contradiction will be to construct such a matrix.
We now present a way to construct a full-rank  $T_{MM}$  kernel matrix with any number of points  $n$ . Consider the set of points  $\{a^{(i)}\}_{i=1}^n$  where  $a^{(i)} = (2n)^i \in \mathbb{R}$ . For any  $i$ , we have that  $T_{MM}(a^{(i)}, a^{(i)}) = 1$  and if  $n \geq 2$  then
$$
\sum_ {j \neq i} \left| T _ {M M} \left(a ^ {(i)}, a ^ {(j)}\right) \right| = \sum_ {j \neq i} (2 n) ^ {- | i - j |} \leq \sum_ {j \neq i} \frac {1}{2 n} \leq \frac {1}{2},
$$
meaning the kernel matrix is strictly diagonally dominant and therefore non-singular. Since such a construction exists for any  $n$ , setting  $n = M + 1$  contradicts the assumption of an  $M$ -dimensional
feature map, and repeating this argument for all finite  $M$  proves that no finite-dimensional feature map can exist for  $T_{MM}$ .
The proof for  $T_{DP}$  is almost identical. Consider the same sequence of points as in the preceding paragraph. For any  $i$ , we have that  $T_{DP}(a^{(i)}, a^{(i)}) = 1$  and
$$
\sum_ {j \neq i} | T _ {D P} (a ^ {(i)}, a ^ {(j)}) | = \sum_ {j \neq i} \frac {1}{(2 n) ^ {| i - j |} + (2 n) ^ {- | i - j |} - 1} \leq \sum_ {j \neq i} \frac {1}{2 n - 1} <   1,
$$
meaning the kernel matrix for  $T_{DP}$  is also strictly diagonally dominant, thereby also precluding the existence of an  $M$ -dimensional feature map for finite  $M$ .
This result suggests that the best we can hope for is an approximate finite-dimensional feature map for both kernels, which is exactly what is provided in this paper.
E Methods to correct the bias of Tanimoto dot product random features
Here, we describe the bias correction strategies mentioned in Section 4.2 and tested experimentally in section 6. We begin by noting that when  $x, x' \in \mathbb{R}_{\geq 0}^{d}$ ,  $x \cdot x' \geq 0$ , and therefore, the power series
$$
T _ {D P} (x, x ^ {\prime}) = \sum_ {r = 1} ^ {\infty} (x \cdot x ^ {\prime}) ^ {r} (\| x \| ^ {2} + \| x ^ {\prime} \| ^ {2}) ^ {- r}
$$
is monotone. Therefore, if we use an unbiased sketch for the truncated series
$$
\sum_ {r = 1} ^ {R} (x \cdot x ^ {\prime}) ^ {r} (\| x \| ^ {2} + \| x ^ {\prime} \| ^ {2}) ^ {- r}
$$
as the one constructed in Section 4.2, the final sketch will be biased downward:
$$
\mathbb {E} (\Phi (x) \cdot \Phi (x ^ {\prime})) <   T _ {D P} (x, x ^ {\prime})
$$
for all  $x, x' \in \mathbb{R}_{\geq 0}^d$ . Below we introduce two strategies to remedy this issue.
E.1 Bias correction strategy 1: normalize the features
Empirically, we observe that the highest bias occurs in the diagonal elements of the kernel matrix. As the kernel satisfies  $T_{DP}(x,x) = 1$  for all  $x\in \mathbb{R}^d$ , one possible correction is to normalize the sketch to obtain
$$
\tilde {\Phi} (x) = \frac {\Phi (x)}{\| \Phi (x) \|}.
$$
This remains an oblivious sketch, and ensures that the diagonal of the kernel matrix is estimated exactly, at the expense of possibly introducing some bias in off-diagonal elements of  $K$ .
E.2 Bias correction strategy 2: sketch the residual
To simplify the algebra, let  $t_{x,y} = \frac{x\cdot y}{\|x\|^2 + \|y\|^2}$ . The power series for  $T_{DP}$  can then be re-written as:
$$
\begin{array}{l} T _ {D P} (x, y) = \sum_ {r = 1} ^ {\infty} \left(t _ {x, y}\right) ^ {r} \\ = \sum_ {r = 1} ^ {R} \left(t _ {x, y}\right) ^ {r} + \left(t _ {x, y}\right) ^ {R + 1} + \sum_ {r = R + 2} ^ {\infty} \left(t _ {x, y}\right) ^ {r} \\ = \sum_ {r = 1} ^ {R} \left(t _ {x, y}\right) ^ {r} + \left(t _ {x, y}\right) ^ {R + 1} + \left(t _ {x, y}\right) ^ {R + 1} \sum_ {r = 1} ^ {\infty} \left(t _ {x, y}\right) ^ {r} \\ = \underbrace {\sum_ {r = 1} ^ {R} \left(t _ {x , y}\right) ^ {r}} _ {k ^ {R}} + \underbrace {\left(t _ {x , y}\right) ^ {R + 1} \left(1 + T _ {D P} (x , y)\right)} _ {\text {t r u n c a t i o n e r r o r}} \tag {28} \\ \end{array}
$$
Critically, the truncation error can be written in terms of the kernel value itself, without any infinite sums. This enables a simple procedure to generate random features for both the truncated power series and the remainder:
1. Compute and store  $\Phi (x)$  as random features for the truncated kernel  $k^R$
2. Concatenate a single 1 onto  $\Phi(x)$  to produce features  $\Phi_{+1}(x) = (1, \Phi(x))$  which approximate the kernel  $1 + k^R$  using only a single extra dimension. Treat this as approximate random features for the kernel  $1 + k$ .
3. Compute random features  $\Phi_{r + 1}(x)$  for  $t_{x,y}^{R + 1}$ .
4. Apply SKETCH to the tensor product  $\Phi_{r+1}(x) \otimes \Phi_{+1}(x)$  to obtain  $\Delta(x)$ , which approximates random features of the truncation error  $t_{x,y}^{R+1}\left(1 + T_{DP}(x,y)\right)$ .
5. Concatenate the random features  $\Phi(x)$  and  $\Delta(x)$ , to obtain bias corrected random features  $\Phi_{\mathrm{bc}}(x) = \Phi \oplus \Delta(x)$ .
Overall, these random features are essentially a concatenation of the random features for the truncated power series with an additional random feature estimate of the truncation error, which is formed using both the random features for the first  $R$  terms and the random features for the  $(R + 1)$ th term. A nice property of the procedure above is that it re-uses the random features  $\Phi(x)$  to estimate the error.
F Experimental details and Additional Results
F.1 Datasets and featurization
The 1000 molecules from GuacaMol are included in our code. The molecules from the DOCKSTRING dataset are available online (https://github.com dockstring/dataset).
Note that when using count fingerprints for  $T_{DP}$  we used the square root of the counts as the fingerprint. This was done for two reasons:
1. To reduce the norms of the vectors (and thereby increase  $\zeta$ ).
2. To roughly make their interpretation consistent with  $T_{MM}$ : i.e. the "weight" of a fragment in which occurs  $n$  times in the numerator/denominator of  $T_{DP}$  is  $n$  times the weight of a fragment which occurs once.
F.2 Details of  $T_{MM}$  random features
We use the random hash from Ioffe (2010) in our implementation. To hash vectors in  $\mathbb{R}^D$  random variables  $r_i, c_i \sim \Gamma(2, 1)$ ,  $\beta_i \sim U(0, 1)$  are drawn i.i.d. for  $i = 1, \ldots, D$ , and then the following<sup>3</sup> are computed for all  $i$ :
$$
t _ {i} (x) = \left\lfloor (\ln x _ {i}) / r _ {i} + \beta_ {i} \right\rfloor , \tag {29}
$$
$$
y _ {i} (x) = r _ {i} \left(t _ {i} (x) - \beta_ {i}\right), \tag {30}
$$
$$
a _ {i} (x) = \ln c _ {i} - y _ {i} (x) - \ln r _ {i}, \tag {31}
$$
$$
i ^ {*} (x) = \arg \min  _ {i = 1, \dots , D} a _ {i} (x), \tag {32}
$$
$$
h _ {r, c, \beta} (x) = \left(i ^ {*} (x), t _ {i ^ {*} (x)} (x)\right). \tag {33}
$$
Note that equation 29 uses the convention that  $\ln 0 = -\infty$ . These variables do not have a clear interpretation in isolation so we refer the reader to Ioffe (2010) for an explanation of why this hashing procedure produces a random hash for  $T_{MM}$ . The hash itself is formed of 2 integers:  $i^{*} \in \{1, \dots, D\}$  and  $t_{i^{*}} \in \mathbb{Z}$ . This unfortunately means that the number of possible outputs is potentially infinite, which would require us to potentially sample an arbitrarily large vector  $\Xi$ . To avoid this, we first use python's built-in hash function to map this pair of integers to a single integer, then take the result modulo  $2^{12} = 4096$ . This allows us to sample a small (finite) vector  $\Xi$ , and although it introduces a small amount of bias this does not appear to be an issue in practice.
Elements of  $\Xi$  are always sampled i.i.d. from either a Rademacher or Gaussian distribution (the distribution should always be clear from context).
F.3 Additional results for  $T_{DP}$  random features
Figure F.1 shows that the error for both polynomial random features from TENSORSKETCH and random features for individual terms in  $T_{DP}^{\prime}s$  power series decreases with  $O(1 / M)$ . Figure F.2 shows the overall error for count fingerprints.
F.4 Gaussian process training details from section 6.2
Our GP models use a constant mean and Gaussian noise. Specifically, the model of the observed labels  $y$  is:
$$
f (X) \sim \mathcal {G P} (\mu , a k (X, X) ])
$$
$$
y (X) \sim \mathcal {N} \left(f (X), \sigma^ {2} I\right)
$$
Therefore, the GP hyperparameters are three scalars:
The constant mean,  $\mu$
Note that the presentation of equations 29-33 differs slightly from Ioffe (2010) who defines  $y_{i}$  and  $a_{i}$  to be the exponential of equations 30/31: we wrote it this way because in practice working in log space avoids numerical stability issues. This is explicitly suggested in their paper.
Figure F.1: MSEs for approximating  $(x\cdot x^{\prime})^{r}$  (left) and  $((x\cdot x^{\prime}) / (\| x\|^{2} + \| x^{\prime}\|^{2}))^{r}$  (right) using TENSORSKETCH for various  $r$  on count and binary fingerprints as a function of the random feature dimension  $M$ .
Figure F.2: Same as Figure 3 but with count fingerprints.
The kernel amplitude/outputscale  $a$
The observation noise  $\sigma^2$
GP performance will be greatly affected by the choice of kernel hyperparameters. To ensure that the difference in performance is not due to differences in kernel hyperparameter settings we fit them in a consistent way for all methods. Specifically, for all methods, we start by fitting an exact GP to a random subset of  $M = 5000$  data points by maximizing the marginal likelihood with L-BFGS. The different approximations are as follows:
- Random subset: gpytorch's exact GP inference is applied on a random subset of  $M$  data points (a different subset then was used to fit the hyperparameters).
- SVGP: scikit-learn's K-means clustering is run with  $M$  clusters to produce an initialization of the inducing points. These inducing points are used to initialize a sparse variational GP (Hensman et al., 2013), implemented in gpytorch. The variational parameters are optimized via natural gradient descent with a learning rate of  $10^{-1}$  and a batch size of  $2M = 10000$  for one pass through the dataset. Although the inducing point locations themselves could be further optimized with gradient descent, we chose not to do so for this experiment.
- RFGP: First, the training and test sets are converted into  $M$  dimensional random features. Then, the posterior equations for inference in Bayesian linear models are used to make predictions on the test set given the training set (Bishop and Nasrabadi, 2006, equations 3.49-3.51). The computation is done in a specific order to avoid forming any  $n \times n$  matrices (the Woodbury matrix identity is used extensively for this). This is fairly clearly documented in the code.
Table F.1: Average  $R^2$  score for approximate GPs on DOCKSTRING dataset. Attentive FP and MPNN results are taken from Garcia-Ortegón et al. (2022). Other details are the same as Table 1.
KERNEL	METHOD	ESR2	F2	KIT	PARP1	PGR
TM	RAND SUBSET GP	0.514±0.002	0.810±0.002	0.695±0.002	0.849±0.001	0.426±0.007
SVGP	0.578±0.001	0.861±0.000	0.749±0.000	0.889±0.000	0.542±0.002
RFGP (Ξ RAD.)	0.518±0.002	0.838±0.001	0.703±0.002	0.864±0.001	0.465±0.003
RFGP (Ξ GAUSS.)	0.517±0.002	0.837±0.000	0.702±0.001	0.864±0.001	0.467±0.004
TDP	RAND SUBSET GP	0.513±0.003	0.817±0.001	0.696±0.002	0.851±0.001	0.384±0.011
SVGP	0.581±0.001	0.865±0.000	0.753±0.001	0.889±0.000	0.543±0.002
RFGP (PLAIN)	0.546±0.001	0.852±0.001	0.716±0.002	0.876±0.000	0.512±0.002
RFGP (NORM)	0.546±0.001	0.852±0.001	0.715±0.002	0.876±0.000	0.513±0.002
RFGP (SKETCH)	0.545±0.001	0.852±0.001	0.716±0.002	0.876±0.000	0.510±0.002
N/A	MPNN	0.506±0.001	0.798±0.005	0.755±0.005	0.815±0.010	0.324±0.096
N/A	ATTENTIVE FP	0.627±0.010	0.880±0.001	0.806±0.008	0.910±0.002	0.678±0.008
Note that for  $T_{MM}$  it was vital to implement the kernel as
$$
T _ {M M} (x, x ^ {\prime}) = \frac {\| x \| _ {1} + \| x ^ {\prime} \| _ {1} - \| x - x ^ {\prime} \| _ {1}}{\| x \| _ {1} + \| x ^ {\prime} \| _ {1} + \| x - x ^ {\prime} \| _ {1}} \tag {34}
$$
instead of a naive implementation which follows equation 1. This is because such an implementation requires forming a tensor of shape  $N \times M \times d$  when calculating the kernel between  $N$  and  $M$  points in  $\mathbb{R}^d$  (for example  $T_{ijk} = \min(x_{ik}, y_{jk})$ ) which can exceed memory limits for modest  $N, M, d$ . This identity allows us to use the relatively efficient torch.cdist function. Note that we did not invent this identity ourselves; we discovered it in Ioffe (2010).
Even with this implementation,  $M = 5000$  inducing points did not fit into GPU memory for  $T_{MM}$ , so all experiments were run on CPU.
F.5 Metrics and additional results from section 6.2
Metrics Table 1 reports average log probability, which calculated by first calculating the log probability of each test point individually (i.e. marginally, not jointly), then averaging these values. We also report the coefficient of determination ( $R^2$  score), calculated using the function sklearn.metrics.r2_score. This measures only the error of the GP mean. A value of 1 indicates perfect prediction, while a value of 0 can be achieved by predicting the sample mean for every data point.
Additional results Table F.1 reports the average  $R^2$ . Trends are similar to Table 1. We also include baselines for two types of graph neural network: Attentive FP (Xiong et al., 2019) and MPNN (Gilmer et al., 2017). Although the performance of GP methods does not match that of Attentive FP, it is often close, suggesting there is potential for approximate GPs to be competitive with graph neural networks for molecular property prediction.