Title: Tanimoto Random Features for Scalable Molecular Machine Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: Tanimoto

Kernel

MinMax

Gaussian process

molecule

chemistry

random features
Abstract: The Tanimoto coefficient is commonly used to measure the similarity between molecules represented as discrete fingerprints,
either as a distance metric or a positive definite kernel. While many kernel methods can be accelerated using random feature approximations, at present there is a lack of such approximations for the Tanimoto kernel. In this paper we propose two kinds of novel random features to allow this kernel to scale to large datasets, and in the process discover a novel extension of the kernel to real-valued vectors. We theoretically characterize these random features, and provide error bounds on the spectral norm of the Gram matrix. Experimentally, we show that these random features are effective at approximating the Tanimoto coefficient of real-world datasets
and are useful for molecular property prediction and optimization tasks. Future updates to this work will be available at http://arxiv.org/abs/2306.14809.

Tanimoto Random Features for Scalable Molecular Machine Learning

Abstract

The Tanimoto coefficient is commonly used to measure the similarity between molecules represented as discrete fingerprints, either as a distance metric or a positive definite kernel. While many kernel methods can be accelerated using random feature approximations, at present there is a lack of such approximations for the Tanimoto kernel. In this paper we propose two kinds of novel random features to allow this kernel to scale to large datasets, and in the process discover a novel extension of the kernel to real-valued vectors. We theoretically characterize these random features, and provide error bounds on the spectral norm of the Gram matrix. Experimentally, we show that these random features are effective at approximating the Tanimoto coefficient of real-world datasets and are useful for molecular property prediction and optimization tasks. Future updates to this work will be available at http://arxiv.org/abs/2306.14809 .

1 Introduction

In recent years there have been notable advances in the use of machine learning (ML) for drug discovery, including molecule generation and property prediction (Dara et al., 2022). Despite ceaseless progress in deep learning, conventional methods such as support vector machines or random forest trained on molecular fingerprints are still competitive in the low-data regime (Walters and Barzilay, 2020; Stanley et al., 2021). These fingerprints essentially encode fragments from a molecule into a sparse vector, thereby compactly representing a large number of molecular substructures (David et al., 2020). They are extensively used in virtual screening for substructure and similarity searches as well as an input for ML models (Cereto-Massagué et al., 2015; Granda et al., 2018).

The Tanimoto coefficient (also known as the Jaccard index ) stands out as a natural way to compare such fingerprints. This coefficient is most commonly expressed as a function on sets T S or as a function of non-negative vectors T MM (Jaccard, 1912; Tanimoto, 1958; Ralaivola et al., 2005; Costa, 2021; Tan et al., 2016):

If a and b are binary indicator vectors representing sets A and B respectively, then T S ( A,B ) = T MM ( a, b ) . Therefore T MM can be viewed as a generalization of T S ; for this reason it is sometimes called the 'weighted Jaccard coefficient' or min-max coefficient.

The Tanimoto coefficient is widely used in machine learning and cheminformatics to compute similarities between molecular fingerprints (Bajusz et al., 2015; O'Boyle and Sayle, 2016; MirandaQuintana et al., 2021), chiefly because of the following properties:

Clear Interpretation: The value of T MM ( x, x ′ ) represents the degree of overlap between x and x ′ and is always between 0 and 1. T ( x, x ′ ) = 1 only when x = x ′ .

Kernel: T MM ( · , · ) is positive definite (Gower, 1971; Ralaivola et al., 2005), meaning it can be used as the kernel for algorithms like support vector machines or Gaussian processes.

Metric: 1 -T MM ( x, x ′ ) is a valid distance metric (typically called Jaccard/Soergel distance ) and can therefore be used in nearest-neighbour and clustering algorithms (Marczewski and Steinhaus, 1958; Levandowsky and Winter, 1971).

In this paper, we present and characterize two efficient low-rank approximations for large matrices of Tanimoto coefficients. The first method, presented in section 3, uses a random hash function to index a random tensor and enjoys exceptionally low variance. The second method, presented in section 4, uses a power series expansion of the Tanimoto similarity for binary vectors. This line of research also unexpectedly led to the discovery of a new generalization of the Tanimoto coefficient to arbitrary vectors in R d , T DP , which is also a kernel and can be used to form a distance metric. In section 6 we demonstrate experimentally that our random features are effective at approximating Tanimoto matrices of real-world fingerprint data and demonstrate its application to molecular property prediction and optimization problems.

2 Background: kernel methods and random features

Kernel methods are a broad class of machine learning algorithms which make predictions using a positive definite kernel function k : X × X ↦→ R (Schölkopf et al., 2002). Common methods in this class are support vector machines (Cortes and Vapnik, 1995) and Gaussian processes (Williams and Rasmussen, 2006). Given a dataset of n data points, training most kernel methods requires computing the n × n kernel matrix 1 K i,j = k ( x ( i ) , x ( j ) ) (with O ( n 2 ) time complexity) and possibly inverting it (with O ( n 3 ) time complexity). Because of this, applying kernel methods to large datasets generally requires approximations.

Given a kernel k , a random features map is a random function f : X ↦→ R M , with the property that f ( x ) · f ( x ′ ) approximates k ( x, x ′ ) for every pair x, x ′ ∈ X . The approximation is often exact in expectation:

Random features allow the kernel matrix to be approximated as ̂ K i,j = f ( x ( i ) ) · f ( x ( j ) . Because this matrix has rank at most M , this approximation generally reduces the cost of O ( n 3 ) /O ( n 2 ) computations to O ( M 3 ) /O ( nM 2 ) , i.e. at most linear in n .

The seminal work of Rahimi and Recht (2007), which coined the term random features, gave a general method based on Fourier analysis to construct random features for any Bochner or stationary kernel , for which k ( x, x ′ ) is a function of x -x ′ . This class includes many common kernels including the RBF and Matérn kernels, but excludes T MM . Subsequent works have proposed random features for other kernels including the polynomial kernel and the arc-cosine kernel (Liu et al., 2021). However, there is no general formula to define random features for non-stationary kernels, such as T MM .

A random features map is sometimes called a data-oblivious sketch , to distinguish it from other data-dependent low-rank approximation methods which depend on a given dataset x (1) , . . . , x ( n ) . Examples of data-dependent low rank sketches are the Nyström approximation and leverage-score sampling (Drineas et al., 2005, 2012). Although data-dependent methods may result in lower approximation errors for a given dataset, data-oblivious sketches are naturally parallelizable and useful in cases where the dataset changes over time (e.g. streaming or optimization) or for ultra-large datasets which may not fit in memory.

3 Low-variance random features for Tanimoto and MinMax kernels

Outside of chemistry, the Tanimoto coefficient has been widely used to measure the similarity between text documents and rank results in search engines. To quickly find documents with high similarity to a user's query, many prior works have studied random hashes for the Tanimoto coefficient, i.e. a family of random functions h : X ↦→ { 1 , . . . , K } such that

Although initially these hashes were only applicable to binary inputs (Broder, 1997; Broder et al., 1998; Charikar, 2002), more recent work has produced efficient random hashes for arbitrary nonnegative vectors (Manasse et al., 2010; Ioffe, 2010; Shrivastava, 2016). In this section we propose a novel family of low-variance random features for T MM (and by extension T S ) which is based on random hashes.

It is important to clarify that although the definition of random hashes in equation 3 resembles the definition of random features in equation 2, they are actually distinct. Random hash functions output discrete objects (typically an integer or tuple of integers) whose probability of equality is T MM , while random features must output vectors in R M whose expected inner product is T MM . If a random hash maps to { 1 , . . . , K } , a naive approach may be to use a K -dimensional indicator vector as a random feature. Because hash equality is a binary outcome, the variance of such random features would be T MM (1 -T MM ) . Realistic hash functions like that of Ioffe (2010) use K ≥ 10 3 , implying a 'variance per feature' of ≈ 10 2 , which is undesirably high.

Our main insight is that low-variance scalar random features can be created by using a random hash to index a suitably distributed random vector. In the following theorem, we show that a vector of i.i.d. samples from any distribution with the correct first and second moments can be combined with random hashes to produce random features for T MM .

Theorem 3.1. Let h : X → Y be a random hash for T MM satisfying equation 3, with |Y| = K . Furthermore, let ξ be a random variable such that E [ ξ ] = 0 and E [ ξ 2 ] = 1 , and let Ξ = [ ξ 1 , . . . , ξ K ] be a vector of independent copies of ξ . Then the 1D random features

estimate T MM without bias: E Ξ ,h ( ϕ Ξ ,h ( x ) · ϕ Ξ ,h ( x ′ )) = T MM ( x, x ′ ) , and with variance

V Ξ ,h [ ϕ Ξ ,h ( x ) · ϕ Ξ ,h ( x ′ )] = 1+ T MM ( x, x ′ ) ( E [ ξ 4 ] -1 -T MM ( x, x ′ ) ) ≥ 1 -T MM ( x, x ′ ) 2 . (5) Furthermore, the lower bound is tight and achieved when ξ is Rademacher distributed (i.e. uniform in {-1 , 1 } ).

The proof is given in Appendix D.1. This theorem shows that Rademacher ξ yields the smallest possible variance in the class of random features defined in eq. (4).

These random features have many desirable properties. First, unlike random features for many other kernels such as the Gaussian kernel (Liu et al., 2021), the variance does not depend on the dimension of the input data or norms of the input vectors. Second, because these random features are 1-dimensional scalars, M independent random feature functions can be concatenated to produce M -dimensional random feature vectors with variance at most 1 /M . This suggests that as few as ≈ 10 3 random features could be used in practical problems. Third, although each instance of Ξ requires storing a K dimensional random vector, if ξ is chosen to be Rademacher distributed, then each entry can be stored with a single bit, requiring just ≈ 100 kB of memory when K = 10 6 .

One disadvantage of these random features is that they are not continuous or differentiable with respect to their inputs. For applications such as Bayesian optimization which require optimizing over model inputs this would create difficulties as gradient-based optimization could no longer be done. It was this disadvantage which motivated us to search for other random features, leading to the discoveries in the following section.

4 Tanimoto dot product kernel and its random features

Ralaivola et al. (2005) gave a definition for the Tanimoto coefficient involving dot products:

with T DP ( x, x ′ ) = 1 when x, x ′ = 0 . It is easy to check that T DP ( x, x ′ ) = T MM ( x, x ′ ) on binary vectors, which was used by Ralaivola et al. (2005) to prove that T DP is a kernel on the space { 0 , 1 } d , referencing prior work by Gower (1971). However, T DP is not identical to T MM for general inputs x, x ′ ∈ R d ≥ 0 . Here, we give the first proof that T DP is a positive definite function in R d and thus, also a valid kernel in this space.

̸

Theorem 4.1. For x, x ′ = 0 in R d , we have

where the series is absolutely convergent. The function T DP is a positive definite kernel in R d .

It has been noticed previously that, unlike 1 -T MM , the function 1 -T DP is not a distance metric on non-binary inputs (Kosub, 2019). Indeed, when d = 1 , the inputs { 1 , 2 , 4 } violate the triangle inequality. However, we can easily derive a distance metric from T DP .

Corollary 4.2. d DP ( x, x ′ ) = √ 1 -T DP ( x, x ′ ) corresponds to the RKHS norm of the function 1 2 [ T DP ( x, · ) -T DP ( x ′ , · )] and is therefore a valid distance metric on R d .

Proofs are given in Appendix D.2. These results imply that T DP , like T MM , is an extension of the set-valued Tanimoto coefficient (equation 1) to real vectors and can be used as a substitute for T MM in machine learning algorithms that require a kernel or distance metric. Unlike T MM , the kernel T DP is differentiable everywhere with respect to its inputs. It can also be computed in batches using matrix-matrix multiplication, allowing for efficient vectorized computation.

We now consider producing a random features approximation to T DP for large-scale applications. Motivated by the close relationship between T DP and T MM , one may be tempted to find a random hash for T DP and apply the techniques developed in section 3. Unfortunately, we are able to prove that this is not possible.

Proposition 4.3. There exists no random hash function for T DP over non-binary vectors.

Proof. Charikar (2002) proved that if s ( x, x ′ ) is a similarity function for which there exists a random hash, then 1 -s ( x, x ′ ) must satisfy the triangle inequality (see their Lemma 1). Because 1 -T DP ( x, x ′ ) does not satisfy the triangle inequality it follows by contradiction that there does not exist a random hash for T DP .

Therefore producing random features for T DP will require another approach. In the remainder of this section we present a framework to produce random features for T DP by directly approximating its power series (equation 7). We first describe a method to produce random features for ( ∥ x ∥ 2 + ∥ x ′ ∥ 2 ) -r (4.1). Then we describe how these features can be combined with existing random features for the polynomial kernel to approximate T DP 's truncated power series (4.2-4.3). Lastly, we present an error bound for the kernel matrix of a dataset in the spectral norm, showing that the required dimension for the sketch scales optimally with the stable rank of the kernel matrix (4.4).

4.1 Random features for the 'prefactor' ( ∥ x ∥ 2 + ∥ x ′ ∥ 2 ) -r

In this section we present a random feature map for the positive definite kernel ( x, x ′ ) ↦→ ( ∥ x ∥ 2 + ∥ x ′ ∥ 2 ) -r , which we will refer to as the the prefactor . We defer all proofs to Appendix D.3. We begin with the following lemma, which defines scalar random features for the prefactor:

Lemma 4.4. If Z ∼ Gamma( s, c ) (where c is a rate parameter), then

is an unbiased scalar random feature for the prefactor ( ∥ x ∥ 2 + ∥ x ′ ∥ 2 ) -r for all s, c > 0 .

Although independent copies of Z could be combined to form an M -dimensional sketch, we instead propose to use a dependent point set Z 1 , . . . , Z M where each element Z i has a Gamma ( s, c ) distribution whilst maximally covering the real line. This is a well-established Quasi-Monte Carlo (QMC) technique which generally attains lower variance. We define our M -dimensional QMC features in the following lemma:

Lemma 4.5. Let γ s,c be the inverse cumulative distribution function of a Gamma( s, c ) random variable. Fix M,r ∈ N , u ∈ (0 , 1) , c, s > 0 and let u i = u + i/M -⌊ u + i/M ⌋ for i = 1 , . . . , M . Define ϕ u,r ( x ) = ( ϕ u,r, 1 ( x ) , . . . , ϕ u,r,M ( x )) , where:

If u ∼ U (0 , 1) then ϕ u,r ( x ) forms unbiased random features of the prefactor ( ∥ x ∥ 2 + ∥ x ′ ∥ 2 ) -r .

Although the random features are unbiased for all s, c > 0 , the value of these parameters will impact the error. We show that if s, c are suitably tuned, then the relative error can be bounded:

Lemma 4.6. Let x (1) , . . . , x ( n ) ∈ R d with min i ∥ x ( i ) ∥ 2 max i ∥ x ( i ) ∥ 2 ≥ ζ , and fix u ∈ [0 , 1] . Define the relative error

If c = 2 ζ 2 , s = rζ , then for some constant C independent of r this error satisfies,

Together, these lemmas suggest random features for the prefactor can be created by first estimating ζ (the minimum ratio of norms of input vectors), then using the random features from Lemma 4.5 with the values of s, c specified in Lemma 4.6.

4.2 A framework to produce random features for T DP

There are straightforward rules for producing random features for sums and products of kernels whose individual random features are known (Duvenaud, 2014, sec. 2.6.2). Random features for kernels k 1 , k 2 can be concatenated (denoted ⊕ ) to form random features for the sum kernel k 1 + k 2 , while their tensor product 2 (denoted ⊗ ) forms random features for the product kernel k 1 × k 2 . Our strategy to produce features for T DP is to combine random features for the 'prefactor' (presented in section 4.1) with random features for the polynomial kernel to produce random features for T DP 's power series (equation 7) truncated at R terms.

Fix R ∈ N , and for r = 1 , . . . , R , let ϕ r be a m r -dimensional random features map for the prefactor ( ∥ x ∥ 2 + ∥ x ′ ∥ 2 ) -r and let ψ r be a m ′ r -dimensional random features map for ( x · x ′ ) r . The function:

is therefore a random feature estimate for T DP 's power series, truncated at R terms. Unfortunately, these random features have dimension M = ∑ R r =1 m r m ′ r which depends on the product of the random features dimension of ϕ r and ψ r . Furthermore, the dimension m ′ r of the random features ψ r required to approximate the polynomial kernel ( x · x ′ ) r with good accuracy can scale poorly with r . For even modest values of m r , m ′ r the resulting value of M will likely be prohibitively large.

To remedy this, we turn to recent works which propose powerful linear maps to approximate tensor products with a lower-dimensional vector. Assuming x (1) , y (1) ∈ R d 1 and x (2) , y (2) ∈ R d 2 , these maps are effectively random matrices Π ∈ R m × ( d 1 d 2 ) , which exhibit a subspace embedding property whereby [Π( x (1) ⊗ x (2) )] · [Π( y (1) ⊗ y (2) )] concentrates sharply around ( x (1) ⊗ x (2) ) · ( y (1) ⊗ y (2) ) . Critically, the product Π( x (1) ⊗ x (2) ) can be computed without instantiating either matrix Π or the tensor product x (1) ⊗ x (2) . Examples of such methods include TENSORSKETCH and TENSORSRHT (Pagh, 2013; Pham and Pagh, 2013; Ahle et al., 2020), but for generality we will simply refer to these methods as SKETCH. Defining a series of sketches SKETCH r : R m r × R m ′ r ↦→ R ˜ m r we can modify ˜ Φ R from equation 12 into:

which has output dimension M = ∑ R r =1 ˜ m r , i.e. without any pathological dependencies on the dimensions of ϕ r ( x ) , ψ r ( x ) . However, because these features approximate a truncated power series, they will be biased downward due to the monotonicity of the power series (7). We propose two bias correction techniques to potentially improve empirical accuracy. One approach is to normalize the random features such that Φ( x ) · Φ( x ) = 1 = T DP ( x, x ) for all x ∈ R d , i.e., such that the diagonal entries of the kernel matrix K are estimated exactly. A second approach is based on sketching the residual of the power series. These approaches are presented in detail in Appendix E.

4.3 Implementing the random features

Instantiating the random features from the previous subsection (13) requires making concrete choices for R , SKETCH r , m r , m ′ r , ˜ m r , ϕ r , ψ r for all r , and choosing a bias correction technique. There are many reasonable choices for SKETCH r , such as TENSORSKETCH (Pham and Pagh, 2013) and TENSORSRHT (Ahle et al., 2020). These sketches generally allow m r , m ′ r , ˜ m r to be chosen freely (although naturally error will increase as ˜ m r decreases). Many of these sketches can also be used as random features for the polynomial kernel ψ r (Wacker et al., 2022), either directly or as part of more complex algorithms like TREESKETCH (Ahle et al., 2020) or complex-to-real sketches (Wacker et al., 2023). The QMC random features from section 4.1 can be used for the prefactor ϕ r , with the parameters s, c chosen based on the anticipated norms of the input vectors.

The only remaining inputs are R (the number of power series terms to approximate) and ˜ m 1 , . . . , ˜ m R (how many random features to use for each term). Assuming a fixed dimension M for the final random features, this choice involves a bias variance trade-off, as a higher value of R will reduce bias but require each term in the power series to have fewer features, thereby increasing variance. Intuitively, because the terms of the power series decrease monotonically the variance of terms for small r is likely to dominate the overall variance, and therefore we surmise that a decreasing sequence for { ˜ m r } R r =1 will be the best choice. Ultimately however we do not have theoretical results to dictate this choice in practice. We will evaluate these choices empirically in section 6.

4.4 Asymptotic error bound for T DP random features

Because many kernel methods use kernel matrices as linear operators, it is natural to examine the error of kernel approximations in the operator norm. Previous works have produced asymptotic error bounds of the random feature dimension m required to achieve a relative approximation error of ε in the operator norm. Defining ˜ sr ( K ) = Tr ( K ) / ∥ K ∥ op, Cohen et al. (2015) show that m = ˜ Ω(˜ sr ( K ) /ε 2 ) is essentially optimal for data-oblivious random features of linear kernels, even though it is possible to eliminate logarithmic factors. Our main theoretical result is that with the correct choices of base random features, the random features for T DP presented in section 4.2 achieve similar scaling. We now state this as a theorem.

Theorem 4.7. For any n ≥ 1 , let x (1) , . . . , x ( n ) ∈ R d be a set of inputs with min i ∥ x ( i ) ∥ 2 max i ∥ x ( i ) ∥ 2 ≥ ζ . Let K be the matrix with entries K i,j = T DP ( x ( i ) , x ( j ) ) . For all ε > 0 , there exists an oblivious sketch Φ : R d → R m with m = ˜ Ω(˜ sr ( K ) /ε 2 ) , such that

where ̂ K i,j = Φ( x ( i ) ) · Φ( x ( j ) ) . Furthermore, the sketch can be computed in time ˜ O ( ˜ sr ( K ) nε -2 + nnz ( X ) ε -2 + nζ -1 ε -3 ) .

The random features in the theorem follow equation 13, with specific choices for M , R , and { SKETCH r , ˜ m r , ϕ r , ψ r } R r =1 given in the proof in Appendix D.4. Theorem 4.7 essentially suggests that, with the correct settings, the error of the random features proposed in section 4.2 scales as well as one could reasonably expect for a kernel of this type. We would highlight that the computational cost of the sketch is sub-quadratic in n , and compares favourably with the cost of data-dependent low-rank approximation methods.

10

2

10

3

10

4

10

5

Number of random features

10

5

10

4

10

3

10

2

TMM

MSE

0.0

0.2

0.4

0.6

0.8

1.0

TMM

(

x

,

x

)

0.0

0.5

1.0

1.5

2.0

TMM

Gaussian, count FPs

Gaussian, binary FPs

5 Related work

Our work on random features fits into a large body of literature random features for kernels (Liu et al., 2021). The majority of work in this area focuses on stationary kernels (i.e. k ( x, x ′ ) = f ( x -x ′ ) ), because the Fourier transform can be applied to any stationary kernel to produce random features in a systematic way (Rahimi and Recht, 2007). There is however no analogous universal formula to produce random features for non-stationary kernels like T MM and T DP ; therefore each kernel requires a bespoke approach. Although our random features are novel, they build upon ideas present in prior works. Our random features for T MM critically rely on previously-proposed random hashes for T MM . Our approach to create random features for T DP via approximating its power series follows was inspired by the random features for the Gaussian kernel from Cotter et al. (2011), which were subsequently improved upon by Ahle et al. (2020). Similar techniques have also been used to create random features for the neural tangent kernel (Zandieh et al., 2021). However, to the best of our knowledge no prior works have proposed random features specifically for the Tanimoto kernel or its variants.

Other works have proposed other types of scalable approximations for Tanimoto coefficients which are not based on random features. Haque and Pande (2010) propose SCISSORS, an optimizationbased approach to estimate Tanimoto coefficients which is akin to a data-dependent sketch. A large number of works use hash-based techniques to find approximate nearest neighbours with the Tanimoto distance metric (Nasr et al., 2010; Kristensen et al., 2011; Tabei and Tsuda, 2011; Anastasiu and Karypis, 2017). Although these techniques are useful for information retrieval, unlike random features they cannot be used to directly scale kernel methods to larger datasets.

6 Experiments

In this section we apply the techniques in this paper to realistic datasets of molecular fingerprints. All experiments were performed in python using the numpy (Harris et al., 2020), pytorch (Paszke et al., 2019), gpytorch (Gardner et al., 2018), and rdkit (Landrum et al., 2023) packages. Molecules were represented with both binary (B) and count (C) Morgan fingerprints (Rogers and Hahn, 2010) of dimension 1024 (additional details in Appendix F.1). These vectors indicate the presence (B) or count (C) of different subgraphs in a molecule. Code to reproduce all experiments is available at: https://github.com/AustinT/tanimoto-random-features-neurips23.

6.1 Errors of random features on real datasets

Here we study the error of approximating matrices of Tanimoto coefficients using our random features, with the general goal of verifying the claims in sections 3-4 on a realistic dataset of molecules. We choose to study a sample of 1000 small organic molecules from the GuacaMol dataset (Brown et al., 2019; Mendez et al., 2019) which exemplify the types of molecules typically considered in drug discovery projects. We use both binary (B) and count (C) fingerprints of radius 2.

0.01

1

1e+02

c

/

c

*

r=1 (count FP)

r=3 (count FP)

0.01

1

1e+02

*

0.01

1

1e+02

c

/

c

*

r=1 (binary FP)

0.01

1

1e+02

*

r=3 (binary FP)

9.0

7.5

6.0

4.5

3.0

1.5

0.0

1.5

Median

log10MSE

10

2

10

3

10

4

10

5

Number of random features

10

9

10

7

10

5

10

3

Prefactor MSE

count FP, r=1

count FP, r=3

binary FP, r=1

binary FP, r=3

s

/

s

s

/

s

2

1

0

1

2

Feature allocation

p

0.0005

0.0010

0.0015

0.0020

TDP

MSE

10

2

10

3

10

4

10

5

Number of random features

10

4

10

3

10

2

TDP

MSE (p=-1)

no bias correction

normalize

sketch error

First, we investigate the random features for T MM proposed in section 3. We instantiate these features using the random hash from Ioffe (2010) (explained further in Appendix F.2) with Ξ both Gaussian and Rademacher distributed. The results are shown in Figure 1. The left subplot shows the median mean squared error (MSE), i.e. E i,j [ ( T MM ( x ( i ) , x ( j ) ) -ϕ ( x ( i ) ) · ϕ ( x ( j ) ) ) 2 ] as a function of the random feature dimension M . As expected for a Monte Carlo estimator, the square error decreases with O (1 /M ) (i.e. increasing the number of random features by 10 reduces the MSE by a factor of 10). As predicted by Theorem 3.1, the estimation error seems to depend only on the distribution of Ξ and not on the input vectors themselves; therefore the error curves for count and binary fingerprints overlap completely. The error is lowest when Ξ is Rademacher distributed, although the empirical difference in error seems small. The right subplot looks at the variance across across scalar random features, showing close matching with the predictions of Theorem 3.1. Overall these features behave exactly as expected.

Next, we investigate the random features for T DP from section 4. These features are more complex, so we start by studying the random features for the 'prefactor' from section 4.1. Recall that these features had free parameters s, c > 0 . Fixing the number of features M = 10 4 , Figure 2 (left) shows the MSE for the r = 1 and r = 3 terms for both the binary and count fingerprints (which have different norms) as a function of s and c . The values which minimize the relative error bound from lemma 4.6, denoted s ∗ , c ∗ , seem to lie in a broad plateau of low error in all settings, suggesting that these values of s, c are a prudent choice. Using these values of s, c , Figure 2 (right) shows the MSE with respect to the number of features M . As expected for a QMC method, the error dependence appears to be quadratic O (1 /M 2 ) (i.e. a ten-fold increase in M reduces MSE by 100-fold).

Because it is implemented in scikit-learn (Pedregosa et al., 2011), we use TENSORSKETCH (Pagh, 2013) both as the polynomial random feature map and to combine the polynomial and prefactor random features. We fix the number of random features for the prefactor to be 10 4 (recall it can be chosen freely without impacting the final random feature dimension). Figure F.1 shows the MSE of approximating both ( x · x ′ ) r and (( x · x ′ ) / ( ∥ x ∥ 2 + ∥ x ′ ∥ 2 )) r . In both cases, the MSE decreases approximately with O (1 /M ) . Finally, we empirically examine how to allocate M random features across R terms. Using R = 4 , Figure 3 (left) shows that allocating most of the features to the terms with small R results in lower error. We therefore heuristically suggest allocating features according to ˜ m r ∝ r -1 . Recall that truncating the power series biases the random features downward, and in section 4.2 two bias correction techniques were proposed. Figure 3 (right) studies the overall MSE for the plain features and both bias correction techniques. It appears that, in practice, neither technique is particularly helpful (normalization in fact appears harmful for large M ). All techniques show an error dependence of approximately O (1 /M ) .

6.2 Molecular property prediction and uncertainty quantification

To evaluate their efficacy in practice, we use our random features to approximate large-scale Gaussian processes (GPs) (Williams and Rasmussen, 2006) for molecular property prediction. Specifically, we study 5 tasks from the DOCKSTRING benchmark which entail predicting protein binding affinity from a molecular graph structure (García-Ortegón et al., 2022). Each task contains 250k molecules, making exact GP regression infeasible. We represent molecules with count fingerprints of radius 1.

We use M = 5000 random features for all methods. We compare to two approximate GP baselines. The first is an exact GP on a random subset of size M . Since this approach ignores most of the dataset, one should expect a reasonable approximate GP to perform better. The second is a sparse variational GP (SVGP) which approximates the dataset using M pseudo-data points Z (Titsias, 2009; Hensman et al., 2013). The locations of Z are typically chosen based on the input dataset (we use K-means clustering), making this method effectively a data-dependent sketch. Accordingly, one might expect the performance of this approximation to be better than data-oblivious random features. Details of Gaussian process training are given in appendix F.4.

Table 1 shows the average log probability of test set labels for all types of GP with T MM and T DP kernels. Several trends are evident. First, for each kernel random feature GPs (RFGPs) consistently outperform random subset GPs, but underperform SVGP. Second, for each kernel the difference between the RFGP varieties is small (generally less than the standard deviation). Third, on most targets T DP seems to perform better than the T MM kernel. The reason for this is unclear. Similar trends can be see in the R 2 metric (Table F.1). This suggests that the RFGPs in this paper can be used in large-scale regression, although it seems in practice that data-dependent approximations are more accurate.

6.3 Bayesian optimization in molecule space via Thompson sampling

Bayesian optimization (BO) uses a probabilistic surrogate model to guide optimization and is generally considered one of the most promising techniques for sample-efficient optimization (Shahriari et al., 2015). Because wet-lab experiments are expensive and time-consuming, there is considerable interest in using BO for experiment design. Chemistry experiments are often done in large batches, and

10

3

10

4

Number of data points

10

0

10

2

Time (s)

10

3

10

4

Number of data points

8.0

7.8

Average F2 score

TMM

(exact)

TMM

(RFs)

TDP

(exact)

TDP

(RFs)

therefore algorithms which use functions sampled from a probabilistic model are of particular interest (Hernández-Lobato et al., 2017). To sample from a normal distribution N ( µ, K ) one typically transforms i.i.d. samples Z ∼ N (0 , 1) via µ + K 1 / 2 Z . This requires computing K 1 / 2 , and thereby causes exact GP sampling to scale cubically in the number of evaluation points. By approximating K ≈ Φ T Φ , our random features allow for approximate sampling in linear time. In this section we apply this to a real-world dataset.

As a demonstration, we consider a single round of selecting 100 molecules from a random sub-sample of n molecules using Thompson sampling , a procedure for Bayesian optimization which selects molecules that maximize a function sampled from a GP prior using the T MM and T DP kernels. Similar to the setup from section 6.2, we use molecules and labels from the DOCKSTRING dataset. Molecules are represented as count fingerprints. M = 5000 random features used, with Rademacher Ξ for T MM and no bias correction for T DP . All other implementation details are the same as in the previous subsection. Figure 4 (left) shows that, as expected, exact Thompson sampling scales worse than approximate Thompson sampling with random features. Figure 4 (right) shows that using approximate instead of exact Thompson sampling does not seem to change the average F2 docking scores of the molecules chosen. This suggests that approximate Thompson sampling could fruitfully be applied to large datasets of molecules in Bayesian optimization tasks.

7 Discussion and conclusion

In this paper we presented two kinds of random features to estimate Tanimoto kernel matrices: one based on random hashes and another based on a power series expansion. To our knowledge, this is the first investigation into random features for the Tanimoto kernel. We theoretically analyze their approximation quality and demonstrate that they can effectively approximate the Tanimoto kernel matrices on realistic molecular fingerprint data. In the process we discovered a new Tanimoto-like kernel over all of R d which is a promising substitute for the more established T MM on regression and optimization tasks.

Despite promising theoretical and experimental results, our random features do have some limitations. We found that it was difficult to efficiently vectorize the computation of the random features for T MM , making them undesirably slow to compute. For T DP , we were able to exhibit an error bound on the spectral norm which depends on certain choices for the base sketch and sketch dimensions; however, it is unclear whether these choices are optimal in practice. Nonetheless, in Appendix D.5 we prove that exact low-rank factorizations of T MM and T DP kernel matrices are not possible; this means that follow-up works could reduce but never eliminate the approximation error.

We are most optimistic about the potential of our random features to be applied in Bayesian optimization, in particular by enabling scalable approximate sampling from GP posteriors (Wilson et al., 2020). Although we briefly explored this technique in section 6.3, in the future it could allow for sample-efficient Bayesian algorithms for complex tasks like Pareto frontier exploration and diverse optimization using the Bayesian algorithm execution framework (Neiswanger et al., 2021). These tasks are highly relevant to real-world drug discovery and there are scant new methods poised to solve them in a sample-efficient way. We hope that the methods presented in this paper enable impactful, large-scale applications of the Tanimoto kernel and its two extensions in chemoinformatics.

Acknowledgments and Disclosure of Funding

We thank Isaac Reid and Zhen Ning David Liu for helpful discussions. Austin Tripp acknowledges funding via a C T Taylor Cambridge International Scholarship and the Canadian Centennial Scholarship Fund. Sukriti Singh acknowledges funding from the UK Engineering and Physical Sciences Research Council. José Miguel Hernández-Lobato acknowledges support from a Turing AI Fellowship under grant EP/V023756/1.

Author contributions: the initial idea of using random hashes to produce random features for the Tanimoto kernel came from Sergio. Austin came across Ioffe (2010) and realized his proposed hash could be used to extend the scheme to T MM for general non-negative vectors. Austin derived and proved the variance statement from Theorem 3.1. Sergio proposed and proved Theorem 4.1 and Corollary 4.2 for the T DP kernel. Austin proposed and proved Proposition 4.3 after reading Charikar (2002). Sergio developed the random features for the prefactor (section 4.1) and the overall schema for the random features (section 4.2). Sergio proposed and proved Theorem 4.7. All experiments were designed and performed by Austin. Sukriti helped with some experiments which ultimately did not appear in this version of the manuscript. Sergio and José Miguel provided advising throughout the project. Writing was done jointly, but mostly by Austin and Sergio.

References

Ahle, T. D., Kapralov, M., Knudsen, J. B., Pagh, R., Velingker, A., Woodruff, D. P., and Zandieh, A. (2020). Oblivious sketching of high-degree polynomial kernels. In Proceedings of the Fourteenth Annual ACM-SIAM Symposium on Discrete Algorithms , pages 141-160. SIAM.

Anastasiu, D. C. and Karypis, G. (2017). Efficient identification of tanimoto nearest neighbors: All-pairs similarity search using the extended jaccard coefficient. International Journal of Data Science and Analytics , 4:153-172.

Ando, T., Horn, R. A., and Johnson, C. R. (1987). The singular values of a hadamard product: A basic inequality. Linear and Multilinear Algebra , 21(4):345-365.

Bajusz, D., Rácz, A., and Héberger, K. (2015). Why is tanimoto index an appropriate choice for fingerprint-based similarity calculations? Journal of cheminformatics , 7(1):1-13.

Bishop, C. M. and Nasrabadi, N. M. (2006). Pattern recognition and machine learning , volume 4. Springer.

Broder, A. Z. (1997). On the resemblance and containment of documents. In Proceedings. Compression and Complexity of SEQUENCES 1997 (Cat. No. 97TB100171) , pages 21-29. IEEE.

Broder, A. Z., Charikar, M., Frieze, A. M., and Mitzenmacher, M. (1998). Min-wise independent permutations. In Proceedings of the thirtieth annual ACM symposium on Theory of computing , pages 327-336.

Brown, N., Fiscato, M., Segler, M. H., and Vaucher, A. C. (2019). Guacamol: benchmarking models for de novo molecular design. Journal of chemical information and modeling , 59(3):1096-1108.

Cereto-Massagué, A., Ojeda, M. J., Valls, C., Mulero, M., Garcia-Vallvé, S., and Pujadas, G. (2015). Molecular fingerprint similarity search in virtual screening. Methods , 71:58-63.

Charikar, M. S. (2002). Similarity estimation techniques from rounding algorithms. In Proceedings of the thiry-fourth annual ACM symposium on Theory of computing , pages 380-388.

Cohen, M. B., Nelson, J., and Woodruff, D. P. (2015). Optimal approximate matrix product in terms of stable rank. arXiv preprint arXiv:1507.02268 .

Cortes, C. and Vapnik, V. (1995). Support-vector networks. Machine learning , 20:273-297.

Costa, L. d. F. (2021). Further generalizations of the jaccard index. arXiv preprint arXiv:2110.09619 .

Cotter, A., Keshet, J., and Srebro, N. (2011). Explicit approximations of the gaussian kernel. arXiv preprint arXiv:1109.4603 .

Dara, S., Dhamercherla, S., Jadav, S. S., Babu, C. M., and Ahsan, M. J. (2022). Machine learning in drug discovery: a review. Artificial Intelligence Review , 55(3):1947-1999.

David, L., Thakkar, A., Mercado, R., and Engkvist, O. (2020). Molecular representations in ai-driven drug discovery: a review and practical guide. Journal of Cheminformatics , 12(1):1-22.

Drineas, P., Magdon-Ismail, M., Mahoney, M. W., and Woodruff, D. P. (2012). Fast approximation of matrix coherence and statistical leverage. The Journal of Machine Learning Research , 13(1):34753506.

Drineas, P., Mahoney, M. W., and Cristianini, N. (2005). On the nyström method for approximating a gram matrix for improved kernel-based learning. journal of machine learning research , 6(12).

Duvenaud, D. (2014). Automatic model construction with gaussian processes.

García-Ortegón, M., Simm, G. N., Tripp, A. J., Hernández-Lobato, J. M., Bender, A., and Bacallado, S. (2022). Dockstring: easy molecular docking yields better benchmarks for ligand design. Journal of chemical information and modeling , 62(15):3486-3502.

Gardner, J., Pleiss, G., Weinberger, K. Q., Bindel, D., and Wilson, A. G. (2018). Gpytorch: Blackbox matrix-matrix gaussian process inference with gpu acceleration. Advances in neural information processing systems , 31.

Gilmer, J., Schoenholz, S. S., Riley, P. F., Vinyals, O., and Dahl, G. E. (2017). Neural message passing for quantum chemistry. In International conference on machine learning , pages 1263-1272. PMLR.

Gower, J. C. (1971). A general coefficient of similarity and some of its properties. Biometrics , pages 857-871.

Granda, J. M., Donina, L., Dragone, V., Long, D.-L., and Cronin, L. (2018). Controlling an organic synthesis robot with machine learning to search for new reactivity. Nature , 559(7714):377-381.

Haque, I. S. and Pande, V. S. (2010). Scissors: a linear-algebraical technique to rapidly approximate chemical similarities. Journal of chemical information and modeling , 50(6):1075-1088.

Harris, C. R., Millman, K. J., Van Der Walt, S. J., Gommers, R., Virtanen, P., Cournapeau, D., Wieser, E., Taylor, J., Berg, S., Smith, N. J., et al. (2020). Array programming with numpy. Nature , 585(7825):357-362.

Hensman, J., Fusi, N., and Lawrence, N. D. (2013). Gaussian processes for big data. In Proceedings of the Twenty-Ninth Conference on Uncertainty in Artificial Intelligence , pages 282-290.

Hernández-Lobato, J. M., Requeima, J., Pyzer-Knapp, E. O., and Aspuru-Guzik, A. (2017). Parallel and distributed thompson sampling for large-scale accelerated exploration of chemical space. In International conference on machine learning , pages 1470-1479. PMLR.

Ioffe, S. (2010). Improved consistent sampling, weighted minhash and l1 sketching. In 2010 IEEE international conference on data mining , pages 246-255. IEEE.

Jaccard, P. (1912). The distribution of the flora in the alpine zone. 1. New phytologist , 11(2):37-50.

Kosub, S. (2019). A note on the triangle inequality for the jaccard distance. Pattern Recognition Letters , 120:36-38.

Kristensen, T. G., Nielsen, J., and Pedersen, C. N. (2011). Using inverted indices for accelerating lingo calculations. Journal of chemical information and modeling , 51(3):597-600.

Landrum, G., Tosco, P., Kelley, B., Ric, Cosgrove, D., sriniker, gedeck, Vianello, R., NadineSchneider, Kawashima, E., N, D., Jones, G., Dalke, A., Cole, B., Swain, M., Turk, S., AlexanderSavelyev, Vaucher, A., Wójcikowski, M., Take, I., Probst, D., Ujihara, K., Scalfani, V. F., guillaume godin, Lehtivarjo, J., Pahl, A., Walker, R., Berenger, F., jasondbiggs, and strets123 (2023). rdkit/rdkit: 2023_09_1 (q3 2023) release beta.

Levandowsky, M. and Winter, D. (1971). Distance between sets. Nature , 234(5323):34-35.

Liu, F., Huang, X., Chen, Y., and Suykens, J. A. (2021). Random features for kernel approximation: A survey on algorithms, theory, and beyond. IEEE Transactions on Pattern Analysis and Machine Intelligence , 44(10):7128-7148.

Manasse, M., McSherry, F., and Talwar, K. (2010). Consistent weighted sampling. Unpublished technical report) http://research. microsoft. com/en-us/people/manasse , 2.

Marczewski, E. and Steinhaus, H. (1958). On a certain distance of sets and the corresponding distance of functions. In Colloquium Mathematicum , volume 6, pages 319-327. Instytut Matematyczny Polskiej Akademii Nauk.

Mendez, D., Gaulton, A., Bento, A. P., Chambers, J., De Veij, M., Félix, E., Magariños, M. P., Mosquera, J. F., Mutowo, P., Nowotka, M., et al. (2019). Chembl: towards direct deposition of bioassay data. Nucleic acids research , 47(D1):D930-D940.

Miranda-Quintana, R. A., Bajusz, D., Rácz, A., and Héberger, K. (2021). Differential consistency analysis: which similarity measures can be applied in drug discovery? Molecular Informatics , 40(7):2060017.

Nasr, R., Hirschberg, D. S., and Baldi, P. (2010). Hashing algorithms and data structures for rapid searches of fingerprint vectors. Journal of chemical information and modeling , 50(8):1358-1368.

Neiswanger, W., Wang, K. A., and Ermon, S. (2021). Bayesian algorithm execution: Estimating computable properties of black-box functions using mutual information. In International Conference on Machine Learning , pages 8005-8015. PMLR.

Nelson, J. and Nguyên, H. L. (2013). Osnap: Faster numerical linear algebra algorithms via sparser subspace embeddings. In 2013 ieee 54th annual symposium on foundations of computer science , pages 117-126. IEEE.

O'Boyle, N. M. and Sayle, R. A. (2016). Comparing structural fingerprints using a literature-based similarity benchmark. Journal of cheminformatics , 8(1):1-14.

Pagh, R. (2013). Compressed matrix multiplication. ACM Transactions on Computation Theory (TOCT) , 5(3):1-17.

Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019). Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems , 32.

Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., and Duchesnay, E. (2011). Scikit-learn: Machine learning in Python. Journal of Machine Learning Research , 12:2825-2830.

Pham, N. and Pagh, R. (2013). Fast and scalable polynomial kernels via explicit feature maps. In Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining , pages 239-247.

Rahimi, A. and Recht, B. (2007). Random features for large-scale kernel machines. Advances in neural information processing systems , 20.

Ralaivola, L., Swamidass, S. J., Saigo, H., and Baldi, P. (2005). Graph kernels for chemical informatics. Neural networks , 18(8):1093-1110.

Rogers, D. and Hahn, M. (2010). Extended-connectivity fingerprints. Journal of chemical information and modeling , 50(5):742-754.

Schölkopf, B., Smola, A. J., Bach, F., et al. (2002). Learning with kernels: support vector machines, regularization, optimization, and beyond . MIT press.

Shahriari, B., Swersky, K., Wang, Z., Adams, R. P., and De Freitas, N. (2015). Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE , 104(1):148-175.

Shrivastava, A. (2016). Simple and efficient weighted minwise hashing. Advances in Neural Information Processing Systems , 29.

Stanley, M., Bronskill, J. F., Maziarz, K., Misztela, H., Lanini, J., Segler, M., Schneider, N., and Brockschmidt, M. (2021). Fs-mol: A few-shot learning dataset of molecules. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2) .

Tabei, Y. and Tsuda, K. (2011). Sketchsort: Fast all pairs similarity search for large databases of molecular fingerprints. Molecular informatics , 30(9):801-807.

Tan, P.-N., Steinbach, M., and Kumar, V. (2016). Introduction to data mining . Pearson Education India.

Tanimoto, T. T. (1958). Elementary mathematical theory of classification and prediction.

Titsias, M. (2009). Variational learning of inducing variables in sparse gaussian processes. In Artificial intelligence and statistics , pages 567-574. PMLR.

Wacker, J., Kanagawa, M., and Filippone, M. (2022). Improved random features for dot product kernels. arXiv preprint arXiv:2201.08712 .

Wacker, J., Ohana, R., and Filippone, M. (2023). Complex-to-real sketches for tensor products with applications to the polynomial kernel. In Ruiz, F., Dy, J., and van de Meent, J.-W., editors, Proceedings of The 26th International Conference on Artificial Intelligence and Statistics , volume 206 of Proceedings of Machine Learning Research , pages 5181-5212. PMLR.

Walters, W. P. and Barzilay, R. (2020). Applications of deep learning in molecule generation and molecular property prediction. Accounts of chemical research , 54(2):263-270.

Williams, C. K. and Rasmussen, C. E. (2006). Gaussian processes for machine learning , volume 2. MIT press Cambridge, MA.

Wilson, J., Borovitskiy, V., Terenin, A., Mostowsky, P., and Deisenroth, M. (2020). Efficiently sampling functions from gaussian process posteriors. In International Conference on Machine Learning , pages 10292-10302. PMLR.

Xiong, Z., Wang, D., Liu, X., Zhong, F., Wan, X., Li, X., Li, Z., Luo, X., Chen, K., Jiang, H., et al. (2019). Pushing the boundaries of molecular representation for drug discovery with the graph attention mechanism. Journal of medicinal chemistry , 63(16):8749-8760.

Zandieh, A., Han, I., Avron, H., Shoham, N., Kim, C., and Shin, J. (2021). Scaling neural tangent kernels via sketching and random features. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W., editors, Advances in Neural Information Processing Systems , volume 34, pages 1062-1073. Curran Associates, Inc.

Outline of appendices

Appendix A comments on the items in the NeurIPS Paper Checklist.

Appendix B summarizes the notation used in this paper.

Appendix C explains TREESKETCH from Ahle et al. (2020), which is used in the proof of Theorem 4.7.

Appendix D provides proofs for all theoretical results in the paper.

Appendix E explains possible techniques to correct the bias of the T DP kernel which are mentioned briefly in section 4.

Appendix F gives additional details about the experiments from section 6 and presents some supplementary results.

A Paper Checklist

Here we explicitly comment on all areas of the NeurIPS paper checklist.

Claims The key claims in this paper are the creation of random features for the Tanimoto kernel and the creation of a new continuous kernel. The claims are mainly supported theoretically with proofs, but we also test our random features experimentally and show that they are usable with real-world data.

Code of Ethics + broader impacts Read and acknowledged. Our work is fairly abstract and theoretical and we do not work with human-related data, so we do not foresee significant direct ethical impacts (positive or negative).

Limitations Our work is primarily theoretical, so the main strength (random features with low approximation error) is also the main weakness (the approximation error could plausibly be lower). Random features are well-studied as a general approach so the strengths and weaknesses are generally well-understood (Liu et al., 2021). For this reason we did not include an explicit limitations section. However, we did mention some other limitations in section 7.

Theory Our key theorems have minimal assumptions which are clearly stated. All theorems are proved in Appendix D.

Experiments Code to reproduce the experiments is available at: https://github.com/AustinT/tanimoto-random-features-neurips23.

Training details The experiments in this paper have minimal details and the important details are specified in Appendix F. All unspecified details should be easy to find in the code (https://github.com/AustinT/tanimoto-random-features-neurips23).

Error bars Our tables and figures include error bars.

Compute The compute costs of the experiments in this paper were quite modest and were run on a single machine with no GPU usage. The approximate computational times were:

Section 6.1: a script to produce all the plots took approximately 24 h to run. The long runtime was mostly due to computing errors for many settings of R and p .

Section 6.2: for T MM , SVGP and each RFGP inference took approximately 1 h (total 3h per target). For T DP , SVGP took 0.5 h while RFGP took 0.3 h (total 1.5 h per target). Multiplying this by 5 targets and 5 trials gives a total compute time of ≈ 112 h.

Section 6.3: this experiment was quite fast: total runtime was perhaps 5 h.

Reproducibility Our contribution is reproducible via the statements of our random features and our code. We aim for a high standard of reproducibility: the exact commands to reproduce the results are stated explicitly in our code's README file, and the raw data behind all plots in this paper is included alongside the code.

Safeguards We believe there is nothing high-risk which we need to safeguard.

Licenses We cite the assets which we use in the paper.

Assets We are not releasing assets.

Humans Subjects / IRB Approvals Not applicable to our paper.

B Notation

We use ∥ x ∥ to denote the Euclidean norm of a vector x , ∥ A ∥ op for the spectral norm and ∥ A ∥ F for the Frobenius norm of a matrix A . x i denotes the i th element of a vector x , while x ( i ) denotes the i th vector in a list of vectors. We denote nnz ( · ) the number of non-zero entries in a vector or matrix. For two functions f, g we say f ( z ) = O ( g ( z )) if there is a constant C , such that 0 ≤ f ( z ) ≤ Cg ( z ) for z large enough. Similarly, f ( z ) = Ω( g ( z )) if there is a constant C , such that f ( z ) ≥ Cg ( z ) for z large enough. ˜ O and ˜ Ω omit poly-logarithmic factors.

Throughout the paper, we use d for the dimension of input vectors, n for the number of samples in the dataset, and M for the dimension of the sketch or random features.

C Definition of TREESKETCH

For simplicity, we shall define TREESKETCH (Ahle et al., 2020) for inputs which are r -fold tensor products x (1) ⊗··· ⊗ x ( r ) where r is a power of two, and each x ( i ) ∈ R d is of the same dimension d .

The main ingredients will be two base sketches, one for the leaves of the tree, and one for internal nodes. The leaf sketch, T base is a random matrix in R m × d . The internal sketch, S base, is a random matrix in R m × m 2 , which can be rapidly applied to the tensor product of two vectors in R m . It is possible to instantiate TreeSketch using simple sketches such as COUNTSKETCH, for the leaves, and TENSORSKETCH for internal nodes. However, our theory we assumes that T base is OSNAP (Nelson and Nguyên, 2013), and S base is TENSORSHRT, both of which enjoy a useful spectral property.

Having picked the base sketches, we can define for any power of two q ≥ 2 , a random map Q q : R m q → R m whose action on a tensor product v (1) ⊗··· ⊗ v ( q ) with v ( i ) ∈ R m is given by the following recursion:

Here, the matrices ( S i j ) are independent copies of S base. The action of TREESKETCH Π r on x (1) ⊗ · · · ⊗ x ( r ) is then defined by

where the matrices ( T i ) are independent copies of T base. Figure C.1 gives a schematic view of the computational tree for a tensor product with r = 4 .

Whilst the action of Π r was only defined for tensor products, this can be extended to arbitrary vectors in R d q . Ahle et al. (2020) prove that the resulting linear sketch has many desirable subspace embedding properties, some of which are used in the proof of Theorem 4.7.

S

4

1

S

2

1

T

2

⊗

T

1

x

(1)

x

(2)

S

4

2

T

4

⊗

T

3

x

(3)

x

(4)

⊗

D Proofs

D.1 Proof of Theorem 3.1

For simplicity, we drop the subscripts h, Ξ from probabilities and expectations. To show that the random features are unbiased, we re-write the expectation:

̸

̸

̸

Because in general V [ X ] = E [ X 2 ] -E [ X ] 2 , and we have E [ ϕ Ξ ,h ( x ) · ϕ Ξ ,h ( x ′ )] = T MM ( x, x ′ ) , we only need to compute E [ ( ϕ Ξ ,h ( x ) · ϕ Ξ ,h ( x ′ )) 2 ] . This can be done using a similar decomposition:

̸

̸

Here, the last step follows from the moments of Ξ which were fixed by assumption. Subtracting T MM ( x, x ′ ) 2 from the above yields the expression for the variance.

By Jensen's inequality, E [ ξ 4 ] ≥ E [ ξ 2 ] 2 , and E [ ξ 2 ] = 1 by assumption, so E [ ξ 4 ] ≥ 1 . Substituting E [ ξ 4 ] = 1 into the equation for V ( ϕ Ξ ,h ( x ) · ϕ Ξ ,h ( x ′ )) gives the lower bound and completes the proof.

D.2 Proof of Theorem 4.1 and Corollary 4.2

̸

Take any pair of inputs x, y = 0 in R d . By the Cauchy-Schwarz inequality, we have

Now, we can write the function T DP as

where in the identity (18) we use the bound in (15) to assert that the series is bounded by 2 -r and thus absolutely convergent.

̸

Furthermore, because ( x, y ) ↦→ x · y is a positive definite kernel, and so is

each summand in the power series is positive definite because it is a product of positive definite kernels. Hence, as sums and limits of positive definite kernels are positive definite, and the power series is convergent, T DP is positive definite in the space R d \ { 0 } . The extension to include the vector 0 ∈ R d is straightforward, as any Gram matrix including this vector is block diagonal. We conclude that T DP is a positive definite kernel in R d .

To prove Corollary 4.2, note that by the Moore-Aronszajn theorem, there exists an RKHS of functions ( H , ⟨·⟩ H ) on R d with reproducing kernel T DP , such that for any x, y ∈ R d , ⟨ T DP ( x, · ) , T DP ( y, · ) ⟩ H = T DP ( x, y ) . Then, observe that

where in the final identity, we use the fact that T DP ( x, x ) = 1 for all x ∈ R d . Dividing by 2 and taking square roots on both sides, we obtain

where the RKHS norm on the left is clearly a distance metric.

D.3 Proofs and derivations for prefactor random features from section 4.1

D.3.1 Scalar random features (Lemma 4.4)

First, we present a derivation for the scalar random features for the prefactor ( ∥ x ∥ 2 + ∥ x ′ ∥ 2 ) -r from Lemma 4.4. As the kernel is a function of ∥ x ∥ 2 and ∥ x ′ ∥ 2 , we can deal, without loss of generality, with the one-dimensional case. We begin by observing that for any a, b > 0 ,

Defining the function

and letting Z ∼ Gamma ( s, c ) , we recognise the right hand side of (20) as the expectation of φ Z ( a ) φ Z ( b ) . Hence,

D.3.2 QMCrandom features (Lemma 4.5)

Next we motivate and derive our QMC random features for the prefactor, proving Lemma 4.5. It is possible to sketch a with a vector φ ( a ) = 1 √ M ( φ Z 1 ( a ) , . . . , φ Z M ( a )) where Z 1 , . . . , Z M are independent copies of Z . This makes the kernel approximation φ ( a ) · φ ( b ) a Monte Carlo estimator

of the expectation, with error decreasing with the dimension M of the sketch at the standard rate O ( M -1 / 2 ) .

However, we shall instead consider a Quasi-Monte Carlo (QMC) estimator with an error decreasing at the faster rate O ( M -1 ) . Let γ s,c be the inverse cumulative distribution function of a Gamma ( s, c ) random variable. With a change of variables, we can write the integral (20) as

Fix n > 0 , u ∈ [0 , 1] , and let u i = ( u + i/M ) -⌊ u + i/M ⌋ for i = 1 , . . . , M . We can now define features ϕ u,r ( a ) ∈ R M , as in Section 4.2:

A QMC estimator for ( a + b ) -r is given by the dot product ϕ u ( a ) T ϕ u ( b ) . This estimator is unbiased: Lemma. If u is random and U (0 , 1) , then for all a, b > 0 ,

Proof. By linearity of expectation,

Observing that each u i ∼ U (0 , 1) , the result follows by Eq. (21).

This lemma is equivalent to Lemma 4.5, thereby proving it.

D.3.3 Relative error bound (Lemma 4.6)

In the following lemma, we establish a basic Quasi-Monte Carlo error bound.

Lemma D.1. For any u ∈ [0 , 1] , c ≤ ( a + b ) , 0 < s ≤ r ,

Proof. Consider the QMC approximation of an integral ∫ 1 0 f ( x ) dx for a function f of bounded variation, using a regular net of n points. It is well known that the error of this approximation is bounded above by V ( f ) /n where V is the total variation norm. Hence,

for f ( u ) = e -( a + b -c ) γ s,c ( u ) ( γ s,c ( u )) r -s on 0 ≤ u ≤ 1 . As γ s,c is monotone increasing, f is unimodal tending to 0 as u → 0 and u → 1 . Thus V ( f ) = 2 max 0 ≤ u ≤ 1 f ( u ) . The maximum of f is attained where γ s,c ( u ) = ( r -s ) / ( a + b -c ) , hence

We are interested in sketching ( ∥ x ( i ) ∥ 2 + ∥ x ( j ) ∥ 2 ) -r for every pair of inputs x ( i ) , x ( j ) in a dataset { x (1) , . . . , x ( n ) } . The previous lemma can be used to choose values c and s ≤ r which minimise the relative error

The kernel k ( x, y ) is invariant to scaling x and y , i.e. , k ( x, y ) = k ( ℓx, ℓy ) for all ℓ > 0 . Thus, if we are interested in approximating the Gram matrix for { x (1) , . . . , x ( n ) } , we may assume without loss of generality that max i ∥ x ( i ) ∥ 2 = 1 and let ζ = min i ∥ x ( i ) ∥ 2 . Here, ζ will act as a condition number for the dataset, which will determine the error of the random feature approximation.

The last two factors are log-convex in c/ ( ∥ x ( i ) ∥ 2 + ∥ x ( j ) ∥ 2 ) > 0 . Hence, for a fixed c , the product

is maximised at either ∥ x ( i ) ∥ 2 + ∥ x ( j ) ∥ 2 = 2 or ∥ x ( i ) ∥ 2 + ∥ x ( j ) ∥ 2 = 2 ζ . Therefore,

Using again the log-convexity of y ↦→ (1 -y ) s -r y -s , we find that

is minimised as a function of c at c 1 = 2 s/r . And

is minimised at c 2 = 2 ζs/r . Recall that Lemma D.1 requires 0 ≤ c ≤ ∥ x ( i ) ∥ 2 + ∥ x ( j ) ∥ 2 for all i, j or 0 ≤ c ≤ 2 ζ , which is satisfied by c 2 when s ≤ r , but not necessarily by c 1 . Thus, it is reasonable to set c = c 2 , which leads to

When ζ = 1 (all inputs x ( i ) have the same norm), setting s = r leads to an error bound of 4 /n . However, small values of ζ can make the upper bound blow up unless we tune s suitably. For a given value of ζ , this upper bound could be maximised numerically over 0 < s ≤ r . However, setting s = rζ , we obtain

where ρ ≈ 1 . 2 is defined by

This proves the first inequality in Lemma 4.6.

To conclude the proof of Lemma 4.6, we study the behaviour of the upper bound as it diverges when ζ → 0 . Using the asymptotic Γ( x ) ∼ 1 /x as x → 0 , we have

where 2( r/e ) -r ρ r r Γ( r ) is bounded for r ≥ 1 . Therefore, the upper bound is O ( M -1 ζ -1 ) . This completes the proof.

D.4 Proof of Theorem 4.7

We begin by defining the stable rank, a common notion of effective dimension for a data matrix A ∈ R d × n or its associated Gram matrix A T A .

Definition D.2. The stable rank sr ( A ) of a data matrix A ∈ R d × n is given by

For a positive semi-definite matrix K ∈ R n × n , define ˜ sr ( K ) = Tr ( K ) / ∥ K ∥ op.

The stable rank of A is upper bounded by the rank, but it is insensitive to small singular values, so it can be much smaller than min( n, d ) . Note that sr ( A ) = ˜ sr ( A T A ) . The function ˜ sr ( K ) extends the concept of stable rank to a general kernel matrix K .

We now re-state theorem 4.7 for convenience, and provide a proof:

Theorem. For any n ≥ 1 , let x (1) , . . . , x ( n ) ∈ R d be a set of inputs with min i ∥ x ( i ) ∥ 2 max i ∥ x ( i ) ∥ 2 ≥ ζ . Let K be the matrix with entries K i,j = T DP ( x ( i ) , x ( j ) ) . For all ε > 0 , there exists an oblivious sketch Φ : R d → R m with m = ˜ Ω(˜ sr ( K ) /ε 2 ) , such that

where ̂ K i,j = Φ( x ( i ) ) · Φ( x ( j ) ) . Furthermore, the sketch can be computed in time ˜ O ( ˜ sr ( K ) nε -2 + nnz ( X ) ε -2 + nζ -1 ε -3 ) .

We begin by specifying the sketch Φ . Define

where Φ r ( x ) = Π r +1 ( ϕ u,r ( x ) ⊗ x ⊗ r ) , Π r is a TREESKETCH (see appendix C) and ϕ u,r ( x ) is the random feature expansion for the prefactor defined in Lemma 4.5. TREESKETCH (Ahle et al., 2020), must be instantiated with the base sketches OSNAP (Nelson and Nguyên, 2013) at the leaves ( T base), and TENSORSRHT (Ahle et al., 2020) at internal nodes ( S base ) . The choices for R , and the dimensions of Π r and ϕ u,r will be made explicit in the proof.

For each r ∈ [ R ] , define matrices K r , ˜ K r , and ̂ K r in R n × n by

Then, by the triangle inequality, we have

We shall find high-probability bounds for each of the terms on the right in turn. For the last term, we can choose R = C log(˜ sr ( K ) ε -1 ) with C large enough, and apply the following upper bound

where the final inequality follows from the fact that ˜ sr ( K ) = n/ ∥ K ∥ op, as the kernel matrix K has ones on the diagonal.

The terms in the second sum of (23) may be written as

where ⊙ denotes the Hadamard product and E is the error matrix defined in (10). Noting that E is symmetric, we can apply Corollary 11 in Ando et al. (1987) to assert that

By the triangle inequality, for each r ≥ 1 , we have ∥ K r ∥ op ≤ ∥ K ∥ op. Hence, with the choice M = Ω( Rζ -1 ε -1 ) for the dimension of ϕ u,r , and applying Lemma 4.6 we obtain

Finally, we turn to the first term in (23). Let A r be a matrix with columns x ( i ) ⊗ r ⊗ ϕ u,r ( x ( i ) ) for i = 1 , . . . , n . We can rewrite the error in question

In Lemma D.3, we establish that ∥ A r ∥ 2 F ≤ (1 + ε ) Tr ( K ) and ∥ A r ∥ 2 op ≤ (1 + ε ) ∥ K ∥ op. Let δ = 1 / poly ( n ) denote the error tolerance. Choose the dimension of the sketch Π r +1 to be m r = Ω ( R 2 r 4 ˜ sr ( K ) ε 2 log 3 ( n ( d ∨ M ) εδ )) and the sparsity parameter of OSNAP as s = R 2 r 4 ε 2 log 3 ( n ( d ∨ M ) εδ ) . Then, Lemmata 32-34 in Ahle et al. (2020) ensure that,

Taking a union bound, we obtain

Finally, combining the bounds (26), (25), and (24), we obtain

Dimension of the sketch Φ : The total dimension of the sketch is m = m 1 + · · · + m R , where m r = Ω ( R 2 r 4 ˜ sr ( K ) ε 2 log 3 ( n ( d ∨ M ) εδ )) with R = C log(˜ sr ( K ) ε -1 ) . Hence, ignoring polylogarithmic factors, the sketch has dimension m = ˜ Ω(˜ sr ( K ) /ε 2 ) .

Runtime: We first estimate the runtime of applying Φ r . Applying OSNAP with sparsity parameter s to a vector w ∈ R d takes O ( s nnz ( w )) time. Thus, applying r independent OSNAP sketches to each x ( i ) for i ∈ [ n ] takes O ( sr nnz ( X )) time. Applying OSNAP to ϕ u,r ( x ( i ) ) for i ∈ [ n ] takes O ( nsM ) time. Then, applying one TENSORSRHT sketch takes O ( m r log m r ) time, and therefore applying all the necessary copies of S base requires O ( rm r log m r ) . In total, the sketch Φ a tior may be computed in O ( nrm r log m r + s nnz ( X ) + nsM ) . Adding these runtimes over r ∈ [ R ] gives us a total runtime which is ˜ O (˜ sr ( K ) nε -2 + nnz ( X ) ε -2 + nζ -1 ε -3 ) .

Proof. Note that A T r A r = ˜ K r . Hence, we aim to show that ∥ ˜ K r ∥ op ≤ (1 + ε ) ∥ K ∥ op, and that Tr ( ˜ K r ) ≤ (1 + ε ) Tr ( K ) . By the triangle inequality,

where the penultimate inequality was shown in the proof of Theorem 4.7, and the final inequality is due to the fact that K r ⪯ K .

Then, by linearity of the trace,

And applying once more Corollary 11 of Ando et al. (1987), we have

where σ i ( · ) denotes the i th singular value. Using once more that K r ⪯ K and hence σ i ( K r ) ≤ σ i ( K ) for all i ∈ [ n ] , we obtain,

Plugging this into (27) yields Tr ( ˜ K r ) ≤ (1 + ε ) Tr ( K ) .

D.5 Rank of Tanimoto Kernels

Lemma D.4. There does not exist an exact finite-dimensional feature map for either T MM or T DP .

Proof. We will use a proof by contradiction, first focusing on T MM . The setup for the contradiction is as follows: suppose there existed an exact feature map f : R d ↦→ R M such that T MM ( x, y ) = f ( x ) · f ( y ) ∀ x, y ∈ R d . This would imply that any kernel matrix between n points X could be written as an inner product

Because f outputs M dimensional vectors this would imply the resulting matrix has rank at most M . Therefore, under this hypothesis it should not be possible to form a kernel matrix of more than M inputs which is full-rank (invertible). Our contradiction will be to construct such a matrix.

We now present a way to construct a full-rank T MM kernel matrix with any number of points n . Consider the set of points { a ( i ) } n i =1 where a ( i ) = (2 n ) i ∈ R . For any i , we have that T MM ( a ( i ) , a ( i ) ) = 1 and if n ≥ 2 then

̸

̸

̸

meaning the kernel matrix is strictly diagonally dominant and therefore non-singular. Since such a construction exists for any n , setting n = M +1 contradicts the assumption of an M -dimensional

feature map, and repeating this argument for all finite M proves that no finite-dimensional feature map can exists for T MM .

The proof for T DP is almost identical. Consider the same sequence of points as in the preceding paragraph. For any i , we have that T DP ( a ( i ) , a ( i ) ) = 1 and

̸

̸

̸

meaning the kernel matrix for T DP is also strictly diagonally dominant, thereby also precluding the existence of an M -dimensional feature map for finite M .

This result suggests that the best we can hope for is an approximate finite-dimensional feature map for both kernels, which is exactly what is provided in this paper.

E Methods to correct the bias of Tanimoto dot product random features

Here, we describe the bias correction strategies mentioned in Section 4.2 and tested experimentally in section 6. We begin by noting that when x, x ′ ∈ R d ≥ 0 , x · x ′ ≥ 0 , and therefore, the power series

is monotone. Therefore, if we use an unbiased sketch for the truncated series

as the one constructed in Section 4.2, the final sketch will be biased downward :

for all x, x ′ ∈ R d ≥ 0 . Below we introduce two strategies to remedy this issue.

E.1 Bias correction strategy 1: normalize the features

Empirically, we observe that the highest bias occurs in the diagonal elements of the kernel matrix. As the kernel satisfies T DP ( x, x ) = 1 for all x ∈ R d , one possible correction is to normalize the sketch to obtain

This remains an oblivious sketch, and ensures that the diagonal of the kernel matrix is estimated exactly, at the expense of possibly introducing some bias in off-diagonal elements of K .

E.2 Bias correction strategy 2: sketch the residual

To simplify the algebra, let t x,y = x · y ∥ x ∥ 2 + ∥ y ∥ 2 . The power series for T DP can then be re-written as:

Critically, the truncation error can be written in terms of the kernel value itself, without any infinite sums. This enables a simple procedure to generate random features for both the truncated power series and the remainder:

Compute and store Φ( x ) as random features for the truncated kernel k R .

Concatenate a single 1 onto Φ( x ) to produce features Φ +1 ( x ) = (1 , Φ( x )) which approximate the kernel 1 + k R using only a single extra dimension. Treat this as approximate random features for the kernel 1 + k .

Compute random features Φ r +1 ( x ) for t R +1 x,y .

Apply SKETCH to the tensor product Φ r +1 ( x ) ⊗ Φ +1 ( x ) to obtain ∆( x ) , which approximates random features of the truncation error t R +1 x,y (1 + T DP ( x, y )) .

Concatenate the random features Φ( x ) and ∆( x ) , to obtain bias corrected random features Φ bc ( x ) = Φ ⊕ ∆( x ) .

Overall, these random features are essentially a concatenation of the random features for the truncated power series with an additional random feature estimate of the truncation error, which is formed using both the random features for the first R terms and the random features for the ( R +1) th term. A nice property of the procedure above is that it re-uses the random features Φ( x ) to estimate the error.

F Experimental details and Additional Results

F.1 Datasets and featurization

The 1000 molecules from GuacaMol are included in our code. The molecules from the DOCKSTRING dataset are available online ( https://github.com/dockstring/dataset ).

Note that when using count fingerprints for T DP we used the square root of the counts as the fingerprint. This was done for two reasons:

To reduce the norms of the vectors (and thereby increase ζ ).

To roughly make their interpretation consistent with T MM : i.e. the 'weight' of a fragment in which occurs n times in the numerator/denominator of T DP is n times the weight of a fragment which occurs once.

F.2 Details of T MM random features

We use the random hash from Ioffe (2010) in our implementation. To hash vectors in R D random variables r i , c i ∼ Γ(2 , 1) , β i ∼ U (0 , 1) are drawn i.i.d. for i = 1 , . . . , D , and then the following 3 are computed for all i :

Note that equation 29 uses the convention that ln 0 = -∞ . These variables do not have a clear interpretation in isolation so we refer the reader to Ioffe (2010) for an explanation of why this hashing procedure produces a random hash for T MM . The hash itself is formed of 2 integers: i ∗ ∈ { 1 , . . . , D } and t i ∗ ∈ Z . This unfortunately means that the number of possible outputs is potentially infinite, which would require us to potentially sample an arbitrarily large vector Ξ . To avoid this, we first use python's built-in hash function to map this pair of integers to a single integer, then take the result module 2 12 = 4096 . This allows us to sample a small (finite) vector Ξ , and although it introduces a small amount of bias this does not appear to be an issue in practice.

Elements of Ξ are always sampled i.i.d. from either a Rademacher or Gaussian distribution (the distribution should always be clear from context).

F.3 Additional results for T DP random features

Figure F.1 shows that the error for both polynomial random features from TENSORSKETCH and random features for individual terms in T ′ DP s power series decreases with O (1 /M ) . Figure F.2 shows the overall error for count fingerprints.

F.4 Gaussian process training details from section 6.2

Our GP models use a constant mean and Gaussian noise. Specifically, the model of the observed labels y is:

Therefore, the GP hyperparameters are three scalars:

The constant mean, µ

10

2

10

3

10

4

10

5

Number of random features

10

6

10

5

10

4

10

3

Polynomial MSE

10

2

10

3

10

4

10

5

Number of random features

10

6

10

5

10

4

10

3

10

2

TDP

term MSE

count FP, r=1

count FP, r=3

binary FP, r=1

binary FP, r=3

2

1

0

1

2

Feature allocation

p

0.0005

0.0010

0.0015

0.0020

0.0025

TDP

MSE

10

2

10

3

10

4

10

5

Number of random features

10

4

10

3

10

2

TDP

MSE (p=-1)

no bias correction

normalize

sketch error

The kernel amplitude/outputscale a

The observation noise σ 2

GP performance will be greatly affected by the choice of kernel hyperparameters. To ensure that the difference in performance is not due to differences in kernel hyperparameter settings we fit them in a consistent way for all methods. Specifically, for all methods, we start by fitting an exact GP to a random subset of M = 5000 data points by maximizing the marginal likelihood with L-BFGS. The different approximations are as follows:

Random subset: gpytorch 's exact GP inference is applied on a random subset of M data points (a different subset then was used to fit the hyperparameters).

SVGP: scikit-learn 's K-means clustering is run with M clusters to produce an initialization of the inducing points. These inducing points are used to initialize a sparse variational GP (Hensman et al., 2013), implemented in gpytorch . The variational parameters are optimized via natural gradient descent with a learning rate of 10 -1 and a batch size of 2 M = 10000 for one pass through the dataset. Although the inducing point locations themselves could be further optimized with gradient descent, we chose not to do so for this experiment.

RFGP: First, the training and test sets are converted into M dimensional random features. Then, the posterior equations for inference in Bayesian linear models are used to make predictions on the test set given the training set (Bishop and Nasrabadi, 2006, equations 3.49-3.51). The computation is done in a specific order to avoid forming any n × n matrices (the Woodbury matrix identity is used extensively for this). This is fairly clearly documented in the code.

Note that for T MM it was vital to implement the kernel as

instead of a naive implementation which follows equation 1. This is because such an implementation requires forming a tensor of shape N × M × d when calculating the kernel between N and M points in R d (for example T ijk = min( x ik , y jk ) ) which can exceed memory limits for modest N,M,d . This identity allows us to use the relatively efficient torch.cdist function. Note that we did not invent this identity ourselves; we discovered it in Ioffe (2010).

Even with this implementation, M = 5000 inducing points did not fit into GPU memory for T MM , so all experiments were run on CPU.

F.5 Metrics and additional results from section 6.2

Metrics Table 1 reports average log probability , which calculated by first calculating the log probability of each test point individually (i.e. marginally, not jointly), then averaging these values. We also report the coefficient of determination ( R 2 score), calculated using the function sklearn.metrics.r2_score . This measures only the error of the GP mean. A value of 1 indicates perfect prediction, while a value of 0 can be achieved by predicting the sample mean for every data point.

Additional results Table F.1 reports the average R 2 . Trends are similar to Table 1. We also include baselines for two types of graph neural network: Attentive FP (Xiong et al., 2019) and MPNN (Gilmer et al., 2017). Although the performance of GP methods does not match that of Attentive FP, it is often close, suggesting there is potential for approximate GPs to be competitive with graph neural networks for molecular property prediction.

Peer reviews:
comments: SUMMARY:
Random feature map can overcome the limitation of kernel methods, and reduces its computational cost to linear in the number of training data points. Recent advances in random feature map broaden its application to not only to stationary kernels, but also to non-stationary kernels such as product kernels. The authors propose two kinds of random features for Tanimoto coefficient such that kernel methods based on Tanimoto coefficient can be applied to large datasets.

STRENGTHS:
1. Proposal of two random features
2. Theoretical analysis on the variance and approximation error for the proposed random features.
3. Demonstrated efficiency through computational experiments.
4. Proposal of positive semidefinite Tanimoto product kernel for real-valued vectors.

WEAKNESSES:
As mentioned by the authors in Discussion,
1. Unexpectedly slow computation of T_MM 
2. Uncertainty on the correctness of the error bound

QUESTIONS:
“2. Uncertainty on the correctness of the error bound”
This reviewer finds this a serious point. What is the reason behind this ? Bound not tight enough, or bug in experiments ?
Please describe what the authors can do to resolve it.

LIMITATIONS:
The authors clearly mentioned the limitations of their work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-26

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates two low-rank approximations for the tanimoto similarity in the context of molecules. The two approximations are both random features newly proposed in the paper using random hashes and sketches. As theory, the paper contributes a proof of optimality for a certain way of constructing one of the approximations and an error bound for the operator norm of the other. They perform four experiments answering the immediate questions about the random features and investigating the claimed advantages of the methods.\n\nThank you for any time you take to answer my questions!\n\nSTRENGTHS:\n1. They propose a scheme to use random hashes to index an alphabet of random vectors to construct random features as a low rank approximation of the tanimoto similarity. It has several desirable properties in terms of their variance and they prove what the optimal choice is for the alphabet of random vectors.\n2. They prove that a previously known kernel is for binary inputs is also a kernel for real vectors. For real vectors their first method of constructing random features does not work and they propose another novel method for constructing random features thus arriving at random features with desirable properties like differentiability which allows for using them in e.g., bayesian optimization. For this they hold significant promise since they are fast to compute as well.\n3. The paper provides an error bound on the operator norm of the constructed T_DP random features which apparently is important for sketches to work well according to a citation that I did not check.\n4. All claimed advantages are investigated in 3 experiments and the authors do not shy away from showing better performing methods for some tasks and the weaknesses of the random features/kernel methods in general. The experiments are insightful and well chosen.\n\nWEAKNESSES:\n1. Main concern: How relevant are hand-crafted kernels, and what are the advantages compared to deep learning embeddings? It seems that also in the DOCKSTRING experiments, deep-learned features fare better. Are there any efficiency and scalability advantages? If the answer is that there are no real advantages, I would see it as a large drawback even if we can say that, in practice hand crafted kernels are still often commonly used. If the answer is otherwise, I think it would be important to discuss this in detail.\n2. A more extensive investigation of deep learned fingerprints would be valuable (maybe I am wrong, and you can explain why not): There already is an investigation of deep learned embeddings in terms of Attentive FP in Table 1, but how do deep learning FPs compare in the other experiments, could the f(lambda) computation be extended to deep learned fingerprints? \n3. A more extensive investigation of deep learned fingerprints would be valuable (maybe I am wrong, and you can explain why not): If other embeddings from SSL methods, such as those from “Self-supervised Graph-level Representation Learning with Local and Global Structure” would be tried as well that would be interesting or the pretrained features from “Uni-Mol: A Universal 3D Molecular Representation Learning Framework”.\n4. Are the baselines in the PMO benchmark actually strong, or are they just easily implemented baselines, and that is why they already were in the benchmark, and one should instead, e.g., compare against other kernels?\n5. How do we know that the DOCKSTRING scores reflect/have the same ranking as actual binding affinities, and is the DockString task actually relevant as a real-world benchmark? I think you are making a mistake in saying they are binding affinity tasks - am I wrong in that?\n6. Why can we assume that T_MM without random features is the best and should be a baseline to compare to in, e.g., Figure 1? Should a comparison with other kernels not also be considered?\n7. Is it fair to say that we now have justification for using T_DP for nonbinary vectors through your Theorem 4.1, but no new capability is unlocked in the sense of the kernel being known and used previously?\n\nQUESTIONS:\n1. Has the trick with random hashes for constructing random features based on an alphabet of random vectors and indexing that with the hashes really never been done before?\n2. (Only if you can find the time to answer) Regarding data oblivious sketches: What does it mean for a sketch to be data dependent? Just that it for instance makes assumptions on the sparsity of the kernel matrix? What would be an example of a data-dependent sketch?\n3. What is Attentive FP? A half-sentence explanation in the paper might be useful as well.\n4. Corollary 4.2 and Theorem 4.1 imply that TDP is an extension of set-valued Tanimoto similarity to real vectors. Is there any inherent value to that or why is the Tanimoto similarity put on such a high pedestal?\n\nLIMITATIONS:\nThe paper seems very transparent with weaknesses and does not shy away from e.g. showing results of better-performing methods on certain tasks. They point out limitations in a dedicated section. These are not just conjured up little limitations or remarks about possible small extensions as we see them all too often in ML papers. Instead, they are significant such as computation difficulties that result in speed impairments. The authors could have easily hidden this to make their method seem better but they do not. Thank you!\n\nMaybe the paper should discuss the limitations compared to deep learned fingerprints more and e.g. the weakness compared to attentive FP instead of just showing the results.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work develops random feature maps to approximate the Tanimoto coefficient among molecular datapoints. Approximating the Tanimoto coefficients allows this measure to be used as a kernel in classical machine learning approaches on larger datasets such as clustering and Bayesian optimization.\n\nSTRENGTHS:\nThe presentation and technical precision of this work is excellent. The authors provide sufficient background to their approach and logic such that even a reader with a rudimentary understanding of analysis and algebra can comprehend their work as well as understand the chain of logical events constituting the work's contribution to research. While much of my research focuses on deep learning, I am deeply appreciative of the authors' work.\n\nIt is my opinion that the primary contribution of this work is to develop approximations to the Tanimoto kernel. This enables its usage in classical non-parameteric machine learning algorithms such as bayesian optimization and clustering. The authors present two different approaches---one based on hashing and another on power-series expansion---that each convey disparate benefits and drawbacks. While outside my primary research focus, the solution is quite technically interesting, and I believe it would be relevant for non-deep learning application of molecular science. \n\nThe empirical evaluation on binding-affinity prediction is also well-received as it illusatrates how approximating the Tanimoto kernel can be translated into a real-world application. \n\nA final point of appreciation relates to the clarity and honesty of the writing style and evaluation --- the authors do not seek to claim sota or surpass the performance of the most recent deep learning methods, but rather make a fundamental contribution that would be relevant to scaling non-parametric algorithms operating on molecular data to larger datasets. It is also refreshing to read a paper that is very precise with mathematical jargon, using accurate terminology to leverage existing structure in analysis and not proving theorems that are only tangentially related to the thrust of the work.\n\nWEAKNESSES:\nWith the continued march of progress in deep learning algorithms---and their application to domains of molecular science---it is likely this approach will not see significant stand-alone use in applications. The counterpoint to this estimation is that there's exciting work integrating deep learning with classical non-parametric algorithms, and such approaches may leverage non-parametric approaches as inductive biases to assist representational learning.\n\nQUESTIONS:\nPlease explain L156 more: $x,x' \\in \\mathbb{R}^d = \\vec{0} \\Rightarrow T_{DP}(\\vec{0}, \\vec{0}) = 1$? This does not appear to be a continuous function and therefore not differentiable at 0 (which is at odds with the prior paragraph L149 motivating the need for differentiable random feature maps for Bayesian optimization). Perhaps reformulate (6) as a sequence of functions (i.e. \\begin{cases}..)  to include 0 and then argue for continuity at 0 ($T_{DP} \\rightarrow 1$ as $x,x' \\rightarrow 0$).\n\nLIMITATIONS:\nThe authors have adequately described the limitations of their approach.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

comments: SUMMARY:
This paper presents two methods for approximating a Tanimoto kernel using random features, shows that the approximations are good, and validates the utility of the proposed approach by applying it to three different settings using molecular fingerprint data.  The writing is clear, and the contributions are clearly stated.  Scaling the Tanimoto kernel to larger datasets is of practical importance.

STRENGTHS:
Clarity of exposition.

A nice combination of theoretical and empirical results.

The experiments are extensive and show convincingly that the method works well on a variety of real datasets.

I particularly liked the background in Sections 2 and 3, which set up the contribution of this work very nicely.

The construction of the data-oblivious sketch (Section 4.1) is elegant.

WEAKNESSES:
One of the main theoretical result is really just to show that Equation (6) is positive definite not just on binary inputs (which was shown in 2005) but for real vectors.  This seems like a relatively straightforward result.

QUESTIONS:
I was confused by the status of Theorem 4.7.  Is this a novel contribution, or just a restatement of something previously shown by (Cohen et al. 2015)?

What is the intuition for why T_{DP} achieves better  accuracy than T_{MM} in Section 5.3?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-14

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper derives two random feature maps for Tanimoto kernels of fingerprint data. The feature maps are applied in GP and BO settings to allow accelerated primal learning.\n\nSTRENGTHS:\nThe idea of featurising Tanimoto is clever and novel. The two feature maps are seemingly well derived, and theoretical analysis is provided. The paper shows how these ideas open the door for feature-based learning in GP setting, which is potentially significant.  \n\nThe results demonstrate the efficiency of the feature maps, and show good results when applied to BO/GP problems.\n\nWEAKNESSES:\nThe paper clarity is frustratingly low, and it lacks a lot of rigor and its claims or contributions are not very clear.\n\nThe paper is vague about the cost or running times of the new feature maps, and how they relate to exact kernel computation. \n\nComputation of Tanimoto is just few matrix products, and should be fast. I don’t think the paper has demonstrated sufficiently what was the bottleneck earlier in using Tanimoto, or why do we need these new feature maps in the first place. Furthermore, if we have relatively small fingerprints and linear kernels, why do we even need GPs at all: wouldn’t a Bayesian linear regression do the same thing and avoid any need to construct feature maps? Why don’t we just use the datapoints as feature vectors directly? It seems that the paper is motivated by plugging Tanimoto’s into existing (and complex) GP frameworks, without first dismissing simpler direct approaches.\n\nQUESTIONS:\n- “In the early stage..”. Not sure I agree with this statement. Big datasets are used in all stages of drug development. It’s also not clear why deep learning methods struggle, and there is no citation. Can you explain?\n- \\cdot is undefined\n- What does random function mean? Does it change every time? Is it stochastic? Is it random? Random in what sense? What is a random feature map? The presentation lacks rigor.\n- What does eq 2 mean? What does E_f mean? What is the measure or probability of f? It seems that we take infinitely many feature maps, and require that they together average to a kernel’s feature map. This feels strange, where is this coming from? It does not seem to be RFFs, since in RFFs the expectation is over frequencies (ie taking m to infinity), not functions. Eq 2 doesn’t require that any particular f would approximate the correct kernel value, and thus the statement hatK = f.f is not true based on eq 2.\n- “There is no general formula to define random features for non-stationary kernels”. Not sure this is correct. Please check seminal papers Kom-Samo et al “Generalised spectral kernel”, and Shen et al “Harmonizable mixture kernels with variational Fourier features”; and followup works.\n- P_h is undefined\n- What is “discrete range”?\n- What is an “independent copy”? So we copy the number independently, resulting in just identical copies (surely all copies are independent)? Or is the idea that each is an iid sample of the random variable? Is \\xi scalar?\n- What does \\Xi_h(x) mean? I don’t understand where the h(x) goes. I think h(x) is a selector function that gives one choice out of K choices, and then we pick that \\xi from the list of \\xi’s? But that would be just \\xi_{h(x)}, so \\Xi_h(x) should then be something else. Maybe it’s a set of K \\xi’s that we somehow index? It would help a lot if you would describe the domain and size of all symbols and variables in the paper (eg. is \\phi(x) a scalar or vector?). It’s also annoying that \\cdot’s are used to what are seemingly scalar products (line 136, or maybe its not scalar? Ok, line 144 declares them as scalars.)\n- What is the E_\\Xi,h expectation over? Are there measures over \\Xi and h? What are they?\n- What is a “tight” lower bound?\n- Theorem 3.1. states that \\xi needs to have variance 1. Yet, the optimal \\xi is declared to be Unif(-1,1), which has variance of 1/3. This seems to nullify the entire Rademacher result. Can you explain?\n- Why is the variance at most 1/m?\n- I’m confused what the 1/m analysis actually says. Let’s assume we have million-dimensional input vectors. Now we seemingly only need 1000 features to accurately represent them. Surely this can’t be true. What if we have trillion input dimensions? I think this analysis has some caveats that are not made transparent. Can you explain? Furthermore, the eq 4 requires us to average both \\Xi and h’s. It seems that the text is forgetting the \\Xi part. I’m also now realising that I don’t understand what the “m” random features refer to. Is this m hashes, or m \\xi’s, or m \\Xis, or something else?\n- I don’t understand what the 1-bit remark means. What is an “entry” and how come it needs only one bit? Why do we get 100kB?\n- I don’t understand what is the contribution in sec 3. It seems that a new feature map is introduced. Ok, but how is it novel? What problem does it solve? What is its significance? How does it improve over earlier stuff? The text mentions low-variance, but there has not been any discussion of variance of earlier methods. How do we know the variance is then lower?\n- Sec 3 also feels vague: the hash has not been described. Surely the hash affects things and can’t be ignored? What if we have a really crappy hash that outputs constant value: surely then the line 136 would not be true. Please make all assumptions explicit.\n- Why eq 7 doesn’t support x=x’=0? The equation seems to work for it as well.\n- Are m_r related to each other?\n- I don’t understand line 194. So phi is the feature map of kernel (x+x’)^-r, but why does the feature inner product add squares into the kernel? This feels surprising. Why do we even care about the non-square kernel, line 190 doesn’t have that either.\n- It’s not clear what u or i mean conceptually. What is their role?\n- The story seems to restart at 4.2. Suddenly we are talking about data matrix, it’s linear kernel and its rank. I can’t follow what’s happening. Is a paragraph missing between 4.2. and end of 4.1.? Can you explain? It’s also strange that we redefine data as A, but later keep using datapoints as x, and also X appears (whatever it is, maybe data again?).\n- What does min_i |x_i|^2 mean? I can’t follow the notation. So it seems that we take the norm of the data vector, which gives a scalar, and then its minimum is just the scalar itself. Or is this the smallest element in x_i? Is x_i a vector or an element of a vector? It would be helpful to use boldfacing on vectors.\n- What does zeta mean? In usual fingerprints we have always some zero values, which would indicate that zeta is zero. But then the eq 204 seems to get screwed up. Does this mean that the theory of sec 4 requires the fingerprints to be strictly non-zero or strictly positive? That would be a significant limitation.\n- What is _op?  What is poly(n)? What is tildeO? What is tilde\\Omega?\n- It seems that the paper hides the actual method description to the appendix, or borrows the TreeSketch/SRHT/OSNAP methods to do the heavy lifting. This makes it a bit difficult to follow what’s going on. Could you summarise the main ideas of these, and how does your method extend them?\n- In 5.2. I wonder what is the computational cost between T_DP exact and approx? The exact is just few matrix products, while the approx is something more involved. Why is there such a big running time difference?\n- In table 1 are the methods approx or exact?\n\nLIMITATIONS:\nNo issues", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-20"}