SlowGuess/NeurIPS_2023_OCR huggingface dataset:
4.4 Asymptotic error bound for  $T_{DP}$  random features
Because many kernel methods use kernel matrices as linear operators, it is natural to examine the error of kernel approximations in the operator norm. Previous works have produced asymptotic error bounds of the random feature dimension  $m$  required to achieve a relative approximation error of  $\varepsilon$  in the operator norm. Defining  $\tilde{\mathrm{sr}}(K) = \mathrm{Tr}(K) / \| K\|_{\mathrm{op}}$ , Cohen et al. (2015) show that  $m = \tilde{\Omega}(\tilde{\mathrm{sr}}(K) / \varepsilon^2)$  is essentially optimal for data-oblivious random features of linear kernels, even though it is possible to eliminate logarithmic factors. Our main theoretical result is that with the correct choices of base random features, the random features for  $T_{DP}$  presented in section 4.2 achieve similar scaling. We now state this as a theorem.
Theorem 4.7. For any  $n \geq 1$ , let  $x^{(1)}, \ldots, x^{(n)} \in \mathbb{R}^d$  be a set of inputs with  $\min_i \|x^{(i)}\|^2 \geq \zeta$ . Let  $K$  be the matrix with entries  $K_{i,j} = T_{DP}(x^{(i)}, x^{(j)})$ . For all  $\varepsilon > 0$ , there exists an oblivious sketch  $\Phi: \mathbb{R}^d \to \mathbb{R}^m$  with  $m = \tilde{\Omega}(\tilde{s}r(K) / \varepsilon^2)$ , such that
$$
\mathbb {P} _ {\Phi} \left(\| \widehat {K} - K \| _ {o p} \geq \varepsilon \| K \| _ {o p}\right) \leq \frac {1}{p o l y (n)} \tag {14}
$$
where  $\widehat{K}_{i,j} = \Phi(x^{(i)}) \cdot \Phi(x^{(j)})$ . Furthermore, the sketch can be computed in time  $\tilde{O}(\tilde{s}r(K)n\varepsilon^{-2} + nnz(X)\varepsilon^{-2} + n\zeta^{-1}\varepsilon^{-3})$ .
The random features in the theorem follow equation 13, with specific choices for  $M, R$ , and  $\{\mathrm{SKETCH}_r, \tilde{m}_r, \phi_r, \psi_r\}_{r=1}^R$  given in the proof in Appendix D.4. Theorem 4.7 essentially suggests that, with the correct settings, the error of the random features proposed in section 4.2 scales as well as one could reasonably expect for a kernel of this type. We would highlight that the computational cost of the sketch is sub-quadratic in  $n$ , and compares favourably with the cost of data-dependent low-rank approximation methods.
Figure 1: Left: MSE of  $T_{MM}$  matrix reconstruction as a function of number of random features (median over 5 trials, shaded regions are first/third quartiles). Right: empirical variance of scalar  $T_{MM}$  random feature estimates for  $M = 10^5$ , closely matching theoretical predictions (dashed lines).
Our work on random features fits into a large body of literature random features for kernels (Liu et al., 2021). The majority of work in this area focuses on stationary kernels (i.e.  $k(x, x') = f(x - x')$ ), because the Fourier transform can be applied to any stationary kernel to produce random features in a systematic way (Rahimi and Recht, 2007). There is however no analogous universal formula to produce random features for non-stationary kernels like  $T_{MM}$  and  $T_{DP}$ ; therefore each kernel requires a bespoke approach. Although our random features are novel, they build upon ideas present in prior works. Our random features for  $T_{MM}$  critically rely on previously-proposed random hashes for  $T_{MM}$ . Our approach to create random features for  $T_{DP}$  via approximating its power series follows was inspired by the random features for the Gaussian kernel from Cotter et al. (2011), which were subsequently improved upon by Ahle et al. (2020). Similar techniques have also been used to create random features for the neural tangent kernel (Zandieh et al., 2021). However, to the best of our knowledge no prior works have proposed random features specifically for the Tanimoto kernel or its variants.
Other works have proposed other types of scalable approximations for Tanimoto coefficients which are not based on random features. Haque and Pande (2010) propose SCISSORS, an optimization-based approach to estimate Tanimoto coefficients which is akin to a data-dependent sketch. A large number of works use hash-based techniques to find approximate nearest neighbours with the Tanimoto distance metric (Nasr et al., 2010; Kristensen et al., 2011; Tabei and Tsuda, 2011; Anastasiu and Karypis, 2017). Although these techniques are useful for information retrieval, unlike random features they cannot be used to directly scale kernel methods to larger datasets.
In this section we apply the techniques in this paper to realistic datasets of molecular fingerprints. All experiments were performed in python using the numpy (Harris et al., 2020), pytorch (Paszke et al., 2019), gpytorch (Gardner et al., 2018), and rdkit (Landrum et al., 2023) packages. Molecules were represented with both binary (B) and count (C) Morgan fingerprints (Rogers and Hahn, 2010) of dimension 1024 (additional details in Appendix F.1). These vectors indicate the presence (B) or count (C) of different subgraphs in a molecule. Code to reproduce all experiments is available at: https://github.com/AustinT/tanimoto-random-features-neurips23.
6.1 Errors of random features on real datasets
Here we study the error of approximating matrices of Tanimoto coefficients using our random features, with the general goal of verifying the claims in sections 3-4 on a realistic dataset of molecules. We choose to study a sample of 1000 small organic molecules from the GuacaMol dataset (Brown et al., 2019; Mendez et al., 2019) which exemplify the types of molecules typically considered in drug discovery projects. We use both binary (B) and count (C) fingerprints of radius 2.
Figure 2: Left: Contour plots of MSE for prefactor random features with  $M = 10^{4}$  with varying  $s, c$ . Right: MSE vs number of prefactor random features with  $s, c$  values from Lemma 4.6. As in Figure 1, lines are medians over 5 trials, shaded regions are first/third quartiles.
Figure 3: Left: MSE for  $M = 10^4$  dimensional random features when allocating features by  $\tilde{m}_r \propto r^p$ ,  $r = 1, \ldots, 4$ . Right: MSE of  $T_{DP}$  random features using  $R = 4$ ,  $p = -1$  and various bias correction strategies. Both subplots use binary fingerprints (the equivalent plot for count fingerprints is Figure F.2). As in Figure 1, lines are medians over 5 trials, shaded regions are first/third quartiles.
First, we investigate the random features for  $T_{MM}$  proposed in section 3. We instantiate these features using the random hash from Ioffe (2010) (explained further in Appendix F.2) with  $\Xi$  both Gaussian and Rademacher distributed. The results are shown in Figure 1. The left subplot shows the median mean squared error (MSE), i.e.  $\mathbb{E}_{i,j}\left[\left(T_{MM}(x^{(i)},x^{(j)}) - \phi (x^{(i)})\cdot \phi (x^{(j)})\right)^{2}\right]$  as a function of the random feature dimension  $M$ . As expected for a Monte Carlo estimator, the square error decreases with  $O(1 / M)$  (i.e. increasing the number of random features by 10 reduces the MSE by a factor of 10). As predicted by Theorem 3.1, the estimation error seems to depend only on the distribution of  $\Xi$  and not on the input vectors themselves; therefore the error curves for count and binary fingerprints overlap completely. The error is lowest when  $\Xi$  is Rademacher distributed, although the empirical difference in error seems small. The right subplot looks at the variance across scalar random features, showing close matching with the predictions of Theorem 3.1. Overall these features behave exactly as expected.
Next, we investigate the random features for  $T_{DP}$  from section 4. These features are more complex, so we start by studying the random features for the "prefactor" from section 4.1. Recall that these features had free parameters  $s, c > 0$ . Fixing the number of features  $M = 10^4$ , Figure 2 (left) shows the MSE for the  $r = 1$  and  $r = 3$  terms for both the binary and count fingerprints (which have different norms) as a function of  $s$  and  $c$ . The values which minimize the relative error bound from lemma 4.6, denoted  $s^*, c^*$ , seem to lie in a broad plateau of low error in all settings, suggesting that these values of  $s, c$  are a prudent choice. Using these values of  $s, c$ , Figure 2 (right) shows the MSE with respect to the number of features  $M$ . As expected for a QMC method, the error dependence appears to be quadratic  $O(1 / M^2)$  (i.e. a ten-fold increase in  $M$  reduces MSE by 100-fold).
Table 1: Average log probability of test set labels with various approximate GPs for 5 targets from DOCKSTRING dataset (García-Ortegón et al., 2022). ± values are standard deviations over 5 trials.
KERNEL	METHOD	ESR2	F2	KIT	PARP1	PGR
TMM	RAND SUBSET GP	-1.084±0.004	-0.951±0.002	-1.094±0.002	-0.999±0.002	-1.183±0.005
SVGP	-0.908±0.005	-0.502±0.005	-0.846±0.002	-0.606±0.005	-1.030±0.005
RFGP (Ξ RAD.)	-0.954±0.009	-0.658±0.013	-1.222±0.044	-0.968±0.037	-1.127±0.023
RFGP (Ξ GAUSS.)	-0.956±0.010	-0.663±0.015	-1.230±0.048	-0.967±0.036	-1.124±0.025
TDP	RAND SUBSET GP	-1.073±0.002	-0.940±0.001	-1.077±0.002	-0.988±0.001	-1.187±0.006
SVGP	-0.880±0.004	-0.459±0.002	-0.804±0.002	-0.568±0.002	-1.010±0.004
RFGP (PLAIN)	-0.902±0.003	-0.513±0.004	-0.979±0.015	-0.690±0.022	-1.029±0.002
RFGP (NORM)	-0.902±0.003	-0.515±0.003	-0.980±0.015	-0.691±0.022	-1.028±0.002
RFGP (SKETCH)	-0.904±0.002	-0.515±0.004	-0.979±0.014	-0.690±0.021	-1.030±0.002
Because it is implemented in scikit-learn (Pedregosa et al., 2011), we use TENSORSKETCH (Pagh, 2013) both as the polynomial random feature map and to combine the polynomial and prefactor random features. We fix the number of random features for the prefactor to be  $10^{4}$  (recall it can be chosen freely without impacting the final random feature dimension). Figure F.1 shows the MSE of approximating both  $(x\cdot x^{\prime})^{r}$  and  $((x\cdot x^{\prime}) / (\| x\|^{2} + \| x^{\prime}\|^{2}))^{r}$ . In both cases, the MSE decreases approximately with  $O(1 / M)$ . Finally, we empirically examine how to allocate  $M$  random features across  $R$  terms. Using  $R = 4$ , Figure 3 (left) shows that allocating most of the features to the terms with small  $R$  results in lower error. We therefore heuristically suggest allocating features according to  $\tilde{m}_r\propto r^{-1}$ . Recall that truncating the power series biases the random features downward, and in section 4.2 two bias correction techniques were proposed. Figure 3 (right) studies the overall MSE for the plain features and both bias correction techniques. It appears that, in practice, neither technique is particularly helpful (normalization in fact appears harmful for large  $M$ ). All techniques show an error dependence of approximately  $O(1 / M)$ .
6.2 Molecular property prediction and uncertainty quantification
To evaluate their efficacy in practice, we use our random features to approximate large-scale Gaussian processes (GPs) (Williams and Rasmussen, 2006) for molecular property prediction. Specifically, we study 5 tasks from the DOCKSTRING benchmark which entail predicting protein binding affinity from a molecular graph structure (García-Ortegón et al., 2022). Each task contains  $250\mathrm{k}$  molecules, making exact GP regression infeasible. We represent molecules with count fingerprints of radius 1.
We use  $M = 5000$  random features for all methods. We compare to two approximate GP baselines. The first is an exact GP on a random subset of size  $M$ . Since this approach ignores most of the dataset, one should expect a reasonable approximate GP to perform better. The second is a sparse variational GP (SVGP) which approximates the dataset using  $M$  pseudo-data points  $Z$  (Titsias, 2009; Hensman et al., 2013). The locations of  $Z$  are typically chosen based on the input dataset (we use K-means clustering), making this method effectively a data-dependent sketch. Accordingly, one might expect the performance of this approximation to be better than data-oblivious random features. Details of Gaussian process training are given in appendix F.4.
Table 1 shows the average log probability of test set labels for all types of GP with  $T_{MM}$  and  $T_{DP}$  kernels. Several trends are evident. First, for each kernel random feature GPs (RFGPs) consistently outperform random subset GPs, but underperform SVGP. Second, for each kernel the difference between the RFGP varieties is small (generally less than the standard deviation). Third, on most targets  $T_{DP}$  seems to perform better than the  $T_{MM}$  kernel. The reason for this is unclear. Similar trends can be see in the  $R^2$  metric (Table F.1). This suggests that the RFGPs in this paper can be used in large-scale regression, although it seems in practice that data-dependent approximations are more accurate.
6.3 Bayesian optimization in molecule space via Thompson sampling
Bayesian optimization (BO) uses a probabilistic surrogate model to guide optimization and is generally considered one of the most promising techniques for sample-efficient optimization (Shahriari et al., 2015). Because wet-lab experiments are expensive and time-consuming, there is considerable interest in using BO for experiment design. Chemistry experiments are often done in large batches, and
Figure 4: Run time and docking scores of BO using exact and approximate Thompson sampling. Solid lines are means over 5 trials, shaded regions are standard errors.
therefore algorithms which use functions sampled from a probabilistic model are of particular interest (Hernández-Lobato et al., 2017). To sample from a normal distribution  $\mathcal{N}(\mu, K)$  one typically transforms i.i.d. samples  $Z \sim \mathcal{N}(0,1)$  via  $\mu + K^{1/2}Z$ . This requires computing  $K^{1/2}$ , and thereby causes exact GP sampling to scale cubically in the number of evaluation points. By approximating  $K \approx \Phi^T\Phi$ , our random features allow for approximate sampling in linear time. In this section we apply this to a real-world dataset.
As a demonstration, we consider a single round of selecting 100 molecules from a random sub-sample of  $n$  molecules using Thompson sampling, a procedure for Bayesian optimization which selects molecules that maximize a function sampled from a GP prior using the  $T_{MM}$  and  $T_{DP}$  kernels. Similar to the setup from section 6.2, we use molecules and labels from the DOCKSTRING dataset. Molecules are represented as count fingerprints.  $M = 5000$  random features used, with Rademacher  $\Xi$  for  $T_{MM}$  and no bias correction for  $T_{DP}$ . All other implementation details are the same as in the previous subsection. Figure 4 (left) shows that, as expected, exact Thompson sampling scales worse than approximate Thompson sampling with random features. Figure 4 (right) shows that using approximate instead of exact Thompson sampling does not seem to change the average F2 docking scores of the molecules chosen. This suggests that approximate Thompson sampling could fruitfully be applied to large datasets of molecules in Bayesian optimization tasks.
7 Discussion and conclusion
In this paper we presented two kinds of random features to estimate Tanimoto kernel matrices: one based on random hashes and another based on a power series expansion. To our knowledge, this is the first investigation into random features for the Tanimoto kernel. We theoretically analyze their approximation quality and demonstrate that they can effectively approximate the Tanimoto kernel matrices on realistic molecular fingerprint data. In the process we discovered a new Tanimoto-like kernel over all of  $\mathbb{R}^d$  which is a promising substitute for the more established  $T_{MM}$  on regression and optimization tasks.
Despite promising theoretical and experimental results, our random features do have some limitations. We found that it was difficult to efficiently vectorize the computation of the random features for  $T_{MM}$ , making them undesirably slow to compute. For  $T_{DP}$ , we were able to exhibit an error bound on the spectral norm which depends on certain choices for the base sketch and sketch dimensions; however, it is unclear whether these choices are optimal in practice. Nonetheless, in Appendix D.5 we prove that exact low-rank factorizations of  $T_{MM}$  and  $T_{DP}$  kernel matrices are not possible; this means that follow-up works could reduce but never eliminate the approximation error.
We are most optimistic about the potential of our random features to be applied in Bayesian optimization, in particular by enabling scalable approximate sampling from GP posteriors (Wilson et al., 2020). Although we briefly explored this technique in section 6.3, in the future it could allow for sample-efficient Bayesian algorithms for complex tasks like Pareto frontier exploration and diverse optimization using the Bayesian algorithm execution framework (Neiswanger et al., 2021). These tasks are highly relevant to real-world drug discovery and there are scant new methods poised to solve them in a sample-efficient way. We hope that the methods presented in this paper enable impactful, large-scale applications of the Tanimoto kernel and its two extensions in chemoinformatics.
We thank Isaac Reid and Zhen Ning David Liu for helpful discussions. Austin Tripp acknowledges funding via a C T Taylor Cambridge International Scholarship and the Canadian Centennial Scholarship Fund. Sukriti Singh acknowledges funding from the UK Engineering and Physical Sciences Research Council. José Miguel Hernández-Lobato acknowledges support from a Turing AI Fellowship under grant EP/V023756/1.
Author contributions: the initial idea of using random hashes to produce random features for the Tanimoto kernel came from Sergio. Austin came across Ioffe (2010) and realized his proposed hash could be used to extend the scheme to  $T_{MM}$  for general non-negative vectors. Austin derived and proved the variance statement from Theorem 3.1. Sergio proposed and proved Theorem 4.1 and Corollary 4.2 for the  $T_{DP}$  kernel. Austin proposed and proved Proposition 4.3 after reading Charikar (2002). Sergio developed the random features for the prefactor (section 4.1) and the overall schema for the random features (section 4.2). Sergio proposed and proved Theorem 4.7. All experiments were designed and performed by Austin. Sukriti helped with some experiments which ultimately did not appear in this version of the manuscript. Sergio and José Miguel provided advising throughout the project. Writing was done jointly, but mostly by Austin and Sergio.
Ahle, T. D., Kapralov, M., Knudsen, J. B., Pagh, R., Velingker, A., Woodruff, D. P., and Zandieh, A. (2020). Oblivious sketching of high-degree polynomial kernels. In Proceedings of the Fourteenth Annual ACM-SIAM Symposium on Discrete Algorithms, pages 141-160. SIAM.
Anastasiu, D. C. and Karypis, G. (2017). Efficient identification of tanimoto nearest neighbors: All-pairs similarity search using the extended jaccard coefficient. International Journal of Data Science and Analytics, 4:153-172.
Ando, T., Horn, R. A., and Johnson, C. R. (1987). The singular values of a hadamard product: A basic inequality. Linear and Multilinear Algebra, 21(4):345-365.
Bajusz, D., Rácz, A., and Heberger, K. (2015). Why is tanimoto index an appropriate choice for fingerprint-based similarity calculations? Journal of cheminformatics, 7(1):1-13.
Bishop, C. M. and Nasrabadi, N. M. (2006). Pattern recognition and machine learning, volume 4. Springer.
Broder, A. Z. (1997). On the resemblance and containment of documents. In Proceedings. Compression and Complexity of SEQUENCES 1997 (Cat. No. 97TB100171), pages 21-29. IEEE.
Broder, A. Z., Charikar, M., Frieze, A. M., and Mitzenmacher, M. (1998). Min-wise independent permutations. In Proceedings of the thirtieth annual ACM symposium on Theory of computing, pages 327-336.
Brown, N., Fiscato, M., Segler, M. H., and Vaucher, A. C. (2019). Guacamol: benchmarking models for de novo molecular design. Journal of chemical information and modeling, 59(3):1096-1108.
Cereto-Massague, A., Ojeda, M. J., Valls, C., Mulero, M., Garcia-Vallve, S., and Pujadas, G. (2015). Molecular fingerprint similarity search in virtual screening. Methods, 71:58-63.
Charikar, M. S. (2002). Similarity estimation techniques from rounding algorithms. In Proceedings of the thirty-fourth annual ACM symposium on Theory of computing, pages 380-388.
Cohen, M. B., Nelson, J., and Woodruff, D. P. (2015). Optimal approximate matrix product in terms of stable rank. arXiv preprint arXiv:1507.02268.
Cortes, C. and Vapnik, V. (1995). Support-vector networks. Machine learning, 20:273-297.
Costa, L. d. F. (2021). Further generalizations of the jaccard index. arXiv preprint arXiv:2110.09619.
Cotter, A., Keshet, J., and Srebro, N. (2011). Explicit approximations of the gaussian kernel. arXiv preprint arXiv:1109.4603.
Shrivastava, A. (2016). Simple and efficient weighted minwise hashing. Advances in Neural Information Processing Systems, 29.
Stanley, M., Bronskill, J. F., Maziarz, K., Misztela, H., Lanini, J., Segler, M., Schneider, N., and Brockschmidt, M. (2021). Fs-mol: A few-shot learning dataset of molecules. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).
Tabei, Y. and Tsuda, K. (2011). Sketchsort: Fast all pairs similarity search for large databases of molecular fingerprints. Molecular informatics, 30(9):801-807.
Tan, P.-N., Steinbach, M., and Kumar, V. (2016). Introduction to data mining. Pearson Education India.
Tanimoto, T. T. (1958). Elementary mathematical theory of classification and prediction.
Titsias, M. (2009). Variational learning of inducing variables in sparse gaussian processes. In Artificial intelligence and statistics, pages 567-574. PMLR.
Wacker, J., Kanagawa, M., and Filippone, M. (2022). Improved random features for dot product kernels. arXiv preprint arXiv:2201.08712.
Wacker, J., Ohana, R., and Filippone, M. (2023). Complex-to-real sketches for tensor products with applications to the polynomial kernel. In Ruiz, F., Dy, J., and van de Meent, J.-W., editors, Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, volume 206 of Proceedings of Machine Learning Research, pages 5181-5212. PMLR.
Walters, W. P. and Barzilay, R. (2020). Applications of deep learning in molecule generation and molecular property prediction. Accounts of chemical research, 54(2):263-270.
Williams, C. K. and Rasmussen, C. E. (2006). Gaussian processes for machine learning, volume 2. MIT press Cambridge, MA.
Wilson, J., Borovitskiy, V., Terenin, A., Mostowsky, P., and Deisenroth, M. (2020). Efficiently sampling functions from gaussian process posteriors. In International Conference on Machine Learning, pages 10292-10302. PMLR.
Xiong, Z., Wang, D., Liu, X., Zhong, F., Wan, X., Li, X., Li, Z., Luo, X., Chen, K., Jiang, H., et al. (2019). Pushing the boundaries of molecular representation for drug discovery with the graph attention mechanism. Journal of medicinal chemistry, 63(16):8749-8760.
Zandieh, A., Han, I., Avron, H., Shoham, N., Kim, C., and Shin, J. (2021). Scaling neural tangent kernels via sketching and random features. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W., editors, Advances in Neural Information Processing Systems, volume 34, pages 1062-1073. Curran Associates, Inc.
Outline of appendices
- Appendix A comments on the items in the NeurIPS Paper Checklist.
- Appendix B summarizes the notation used in this paper.
- Appendix C explains TREESKETCH from Ahle et al. (2020), which is used in the proof of Theorem 4.7.
- Appendix D provides proofs for all theoretical results in the paper.
- Appendix E explains possible techniques to correct the bias of the  $T_{DP}$  kernel which are mentioned briefly in section 4.
- Appendix F gives additional details about the experiments from section 6 and presents some supplementary results.
A Paper Checklist
Here we explicitly comment on all areas of the NeurIPS paper checklist.
**Claims** The key claims in this paper are the creation of random features for the Tanimoto kernel and the creation of a new continuous kernel. The claims are mainly supported theoretically with proofs, but we also test our random features experimentally and show that they are usable with real-world data.