SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Austin Tripp
ajt212@cam.ac.uk
Sergio Bacallado
sb2116@cam.ac.uk
Sukriti Singh
ss2971@cam.ac.uk
The Tanimoto coefficient is commonly used to measure the similarity between molecules represented as discrete fingerprints, either as a distance metric or a positive definite kernel. While many kernel methods can be accelerated using random feature approximations, at present there is a lack of such approximations for the Tanimoto kernel. In this paper we propose two kinds of novel random features to allow this kernel to scale to large datasets, and in the process discover a novel extension of the kernel to real-valued vectors. We theoretically characterize these random features, and provide error bounds on the spectral norm of the Gram matrix. Experimentally, we show that these random features are effective at approximating the Tanimoto coefficient of real-world datasets and are useful for molecular property prediction and optimization tasks. Future updates to this work will be available at http://arxiv.org/abs/2306.14809.
In recent years there have been notable advances in the use of machine learning (ML) for drug discovery, including molecule generation and property prediction (Dara et al., 2022). Despite ceaseless progress in deep learning, conventional methods such as support vector machines or random forest trained on molecular fingerprints are still competitive in the low-data regime (Walters and Barzilay, 2020; Stanley et al., 2021). These fingerprints essentially encode fragments from a molecule into a sparse vector, thereby compactly representing a large number of molecular substructures (David et al., 2020). They are extensively used in virtual screening for substructure and similarity searches as well as an input for ML models (Cereto-Massague et al., 2015; Granda et al., 2018).
The Tanimoto coefficient (also known as the Jaccard index) stands out as a natural way to compare such fingerprints. This coefficient is most commonly expressed as a function on sets  $T_{S}$  or as a function of non-negative vectors  $T_{MM}$  (Jaccard, 1912; Tanimoto, 1958; Ralaivola et al., 2005; Costa, 2021; Tan et al., 2016):
$$
T _ {S} \left(X, X ^ {\prime}\right) = \frac {\left| X \cap X ^ {\prime} \right|}{\left| X \cup X ^ {\prime} \right|}, \quad X, X ^ {\prime} \subseteq \Omega , \quad T _ {M M} \left(x, x ^ {\prime}\right) = \frac {\sum_ {i} \min  \left(x _ {i} , x _ {i} ^ {\prime}\right)}{\sum_ {i} \max  \left(x _ {i} , x _ {i} ^ {\prime}\right)}, \quad x, x ^ {\prime} \in \mathbb {R} _ {\geq 0} ^ {d}. \tag {1}
$$
If  $a$  and  $b$  are binary indicator vectors representing sets  $A$  and  $B$  respectively, then  $T_{S}(A,B) = T_{MM}(a,b)$ . Therefore  $T_{MM}$  can be viewed as a generalization of  $T_{S}$ ; for this reason it is sometimes called the "weighted Jaccard coefficient" or min-max coefficient.
The Tanimoto coefficient is widely used in machine learning and cheminformatics to compute similarities between molecular fingerprints (Bajusz et al., 2015; O'Boyle and Sayle, 2016; Miranda-Quintana et al., 2021), chiefly because of the following properties:
1. Clear Interpretation: The value of  $T_{MM}(x, x')$  represents the degree of overlap between  $x$  and  $x'$  and is always between 0 and 1.  $T(x, x') = 1$  only when  $x = x'$ .
2. Kernel:  $T_{MM}(\cdot, \cdot)$  is positive definite (Gower, 1971; Ralaivola et al., 2005), meaning it can be used as the kernel for algorithms like support vector machines or Gaussian processes.
3. Metric:  $1 - T_{MM}(x, x')$  is a valid distance metric (typically called Jaccard/Soergel distance) and can therefore be used in nearest-neighbour and clustering algorithms (Marczewski and Steinhaus, 1958; Levandowsky and Winter, 1971).
In this paper, we present and characterize two efficient low-rank approximations for large matrices of Tanimoto coefficients. The first method, presented in section 3, uses a random hash function to index a random tensor and enjoys exceptionally low variance. The second method, presented in section 4, uses a power series expansion of the Tanimoto similarity for binary vectors. This line of research also unexpectedly led to the discovery of a new generalization of the Tanimoto coefficient to arbitrary vectors in  $\mathbb{R}^d$ ,  $T_{DP}$ , which is also a kernel and can be used to form a distance metric. In section 6 we demonstrate experimentally that our random features are effective at approximating Tanimoto matrices of real-world fingerprint data and demonstrate its application to molecular property prediction and optimization problems.
2 Background: kernel methods and random features
Kernel methods are a broad class of machine learning algorithms which make predictions using a positive definite kernel function  $k: \mathcal{X} \times \mathcal{X} \mapsto \mathbb{R}$  (Schölkopf et al., 2002). Common methods in this class are support vector machines (Cortes and Vapnik, 1995) and Gaussian processes (Williams and Rasmussen, 2006). Given a dataset of  $n$  data points, training most kernel methods requires computing the  $n \times n$  kernel matrix $^1$ $K_{i,j} = k(x^{(i)}, x^{(j)})$  (with  $O(n^2)$  time complexity) and possibly inverting it (with  $O(n^3)$  time complexity). Because of this, applying kernel methods to large datasets generally requires approximations.
Given a kernel  $k$ , a random features map is a random function  $f: \mathcal{X} \mapsto \mathbb{R}^M$ , with the property that  $f(x) \cdot f(x')$  approximates  $k(x, x')$  for every pair  $x, x' \in \mathcal{X}$ . The approximation is often exact in expectation:
$$
\mathbb {E} _ {f} \left[ f (x) \cdot f \left(x ^ {\prime}\right) \right] = k \left(x, x ^ {\prime}\right) \quad \text {f o r a l l} x, x ^ {\prime} \in \mathcal {X}. \tag {2}
$$
Random features allow the kernel matrix to be approximated as  $\widehat{K}_{i,j} = f(x^{(i)})\cdot f(x^{(j)})$ . Because this matrix has rank at most  $M$ , this approximation generally reduces the cost of  $O(n^3) / O(n^2)$  computations to  $O(M^3) / O(nM^2)$ , i.e. at most linear in  $n$ .
The seminal work of Rahimi and Recht (2007), which coined the term random features, gave a general method based on Fourier analysis to construct random features for any Bochner or stationary kernel, for which  $k(x,x')$  is a function of  $x - x'$ . This class includes many common kernels including the RBF and Matérn kernels, but excludes  $T_{MM}$ . Subsequent works have proposed random features for other kernels including the polynomial kernel and the arc-cosine kernel (Liu et al., 2021). However, there is no general formula to define random features for non-stationary kernels, such as  $T_{MM}$ .
A random features map is sometimes called a data-oblivious sketch, to distinguish it from other data-dependent low-rank approximation methods which depend on a given dataset  $x^{(1)}, \ldots, x^{(n)}$ . Examples of data-dependent low rank sketches are the Nyström approximation and leverage-score sampling (Drineas et al., 2005, 2012). Although data-dependent methods may result in lower approximation errors for a given dataset, data-oblivious sketches are naturally parallelizable and useful in cases where the dataset changes over time (e.g. streaming or optimization) or for ultra-large datasets which may not fit in memory.
<sup>1</sup>We write  $x_{i}$  to refer to the  $i$ th element of a vector and  $x^{(i)}$  to denote the  $i$ th vector in a list.
3 Low-variance random features for Tanimoto and MinMax kernels
Outside of chemistry, the Tanimoto coefficient has been widely used to measure the similarity between text documents and rank results in search engines. To quickly find documents with high similarity to a user's query, many prior works have studied random hashes for the Tanimoto coefficient, i.e. a family of random functions  $h: \mathcal{X} \mapsto \{1, \dots, K\}$  such that
$$
\mathbb {P} _ {h} \left(h (x) = h \left(x ^ {\prime}\right)\right) = T _ {M M} \left(x, x ^ {\prime}\right). \tag {3}
$$
Although initially these hashes were only applicable to binary inputs (Broder, 1997; Broder et al., 1998; Charikar, 2002), more recent work has produced efficient random hashes for arbitrary non-negative vectors (Manasse et al., 2010; Ioffe, 2010; Shrivastava, 2016). In this section we propose a novel family of low-variance random features for  $T_{MM}$  (and by extension  $T_{S}$ ) which is based on random hashes.
It is important to clarify that although the definition of random hashes in equation 3 resembles the definition of random features in equation 2, they are actually distinct. Random hash functions output discrete objects (typically an integer or tuple of integers) whose probability of equality is  $T_{MM}$ , while random features must output vectors in  $\mathbb{R}^M$  whose expected inner product is  $T_{MM}$ . If a random hash maps to  $\{1,\dots ,K\}$ , a naive approach may be to use a  $K$ -dimensional indicator vector as a random feature. Because hash equality is a binary outcome, the variance of such random features would be  $T_{MM}\left(1 - T_{MM}\right)$ . Realistic hash functions like that of Ioffe (2010) use  $K \geq 10^{3}$ , implying a "variance per feature" of  $\approx 10^{2}$ , which is undesirably high.
Our main insight is that low-variance scalar random features can be created by using a random hash to index a suitably distributed random vector. In the following theorem, we show that a vector of i.i.d. samples from any distribution with the correct first and second moments can be combined with random hashes to produce random features for  $T_{MM}$ .
Theorem 3.1. Let  $h: \mathcal{X} \to \mathcal{Y}$  be a random hash for  $T_{MM}$  satisfying equation 3, with  $|\mathcal{Y}| = K$ . Furthermore, let  $\xi$  be a random variable such that  $\mathbb{E}[\xi] = 0$  and  $\mathbb{E}[\xi^2] = 1$ , and let  $\Xi = [\xi_1, \dots, \xi_K]$  be a vector of independent copies of  $\xi$ . Then the 1D random features
$$
\phi_ {\Xi , h} (x) = \Xi_ {h (x)} \tag {4}
$$
estimate  $T_{MM}$  without bias:  $\mathbb{E}_{\Xi,h}(\phi_{\Xi,h}(x)\cdot \phi_{\Xi,h}(x')) = T_{MM}(x,x')$ , and with variance
$$
\mathbb {V} _ {\Xi , h} \left[ \phi_ {\Xi , h} (x) \cdot \phi_ {\Xi , h} \left(x ^ {\prime}\right) \right] = 1 + T _ {M M} \left(x, x ^ {\prime}\right) \left(E \left[ \xi^ {4} \right] - 1 - T _ {M M} \left(x, x ^ {\prime}\right)\right) \geq 1 - T _ {M M} \left(x, x ^ {\prime}\right) ^ {2}. \tag {5}
$$
Furthermore, the lower bound is tight and achieved when  $\xi$  is Rademacher distributed (i.e. uniform in  $\{-1,1\}$ ).
The proof is given in Appendix D.1. This theorem shows that Rademacher  $\xi$  yields the smallest possible variance in the class of random features defined in eq. (4).
These random features have many desirable properties. First, unlike random features for many other kernels such as the Gaussian kernel (Liu et al., 2021), the variance does not depend on the dimension of the input data or norms of the input vectors. Second, because these random features are 1-dimensional scalars,  $M$  independent random feature functions can be concatenated to produce  $M$ -dimensional random feature vectors with variance at most  $1 / M$ . This suggests that as few as  $\approx 10^3$  random features could be used in practical problems. Third, although each instance of  $\Xi$  requires storing a  $K$  dimensional random vector, if  $\xi$  is chosen to be Rademacher distributed, then each entry can be stored with a single bit, requiring just  $\approx 100\mathrm{~kB}$  of memory when  $K = 10^{6}$ .
One disadvantage of these random features is that they are not continuous or differentiable with respect to their inputs. For applications such as Bayesian optimization which require optimizing over model inputs this would create difficulties as gradient-based optimization could no longer be done. It was this disadvantage which motivated us to search for other random features, leading to the discoveries in the following section.
4 Tanimoto dot product kernel and its random features
Ralaivola et al. (2005) gave a definition for the Tanimoto coefficient involving dot products:
$$
T _ {D P} \left(x, x ^ {\prime}\right) = \frac {x \cdot x ^ {\prime}}{\| x \| ^ {2} + \| x ^ {\prime} \| ^ {2} - x \cdot x ^ {\prime}}, \tag {6}
$$
with  $T_{DP}(x,x^{\prime}) = 1$  when  $x,x^{\prime} = 0$ . It is easy to check that  $T_{DP}(x,x^{\prime}) = T_{MM}(x,x^{\prime})$  on binary vectors, which was used by Ralaivola et al. (2005) to prove that  $T_{DP}$  is a kernel on the space  $\{0,1\}^{d}$ , referencing prior work by Gower (1971). However,  $T_{DP}$  is not identical to  $T_{MM}$  for general inputs  $x,x^{\prime}\in \mathbb{R}_{\geq 0}^{d}$ . Here, we give the first proof that  $T_{DP}$  is a positive definite function in  $\mathbb{R}^d$  and thus, also a valid kernel in this space.
Theorem 4.1. For  $x, x' \neq 0$  in  $\mathbb{R}^d$ , we have
$$
T _ {D P} \left(x, x ^ {\prime}\right) = \sum_ {r = 1} ^ {\infty} \left(x \cdot x ^ {\prime}\right) ^ {r} \left(\| x \| ^ {2} + \| x ^ {\prime} \| ^ {2}\right) ^ {- r}, \tag {7}
$$
where the series is absolutely convergent. The function  $T_{DP}$  is a positive definite kernel in  $\mathbb{R}^d$ .
It has been noticed previously that, unlike  $1 - T_{MM}$ , the function  $1 - T_{DP}$  is not a distance metric on non-binary inputs (Kosub, 2019). Indeed, when  $d = 1$ , the inputs  $\{1,2,4\}$  violate the triangle inequality. However, we can easily derive a distance metric from  $T_{DP}$ .
Corollary 4.2.  $d_{DP}(x,x^{\prime}) = \sqrt{1 - T_{DP}(x,x^{\prime})}$  corresponds to the RKHS norm of the function  $\frac{1}{2} [T_{DP}(x,\cdot) - T_{DP}(x^{\prime},\cdot)]$  and is therefore a valid distance metric on  $\mathbb{R}^d$ .
Proofs are given in Appendix D.2. These results imply that  $T_{DP}$ , like  $T_{MM}$ , is an extension of the set-valued Tanimoto coefficient (equation 1) to real vectors and can be used as a substitute for  $T_{MM}$  in machine learning algorithms that require a kernel or distance metric. Unlike  $T_{MM}$ , the kernel  $T_{DP}$  is differentiable everywhere with respect to its inputs. It can also be computed in batches using matrix-matrix multiplication, allowing for efficient vectorized computation.
We now consider producing a random features approximation to  $T_{DP}$  for large-scale applications. Motivated by the close relationship between  $T_{DP}$  and  $T_{MM}$ , one may be tempted to find a random hash for  $T_{DP}$  and apply the techniques developed in section 3. Unfortunately, we are able to prove that this is not possible.
Proposition 4.3. There exists no random hash function for  $T_{DP}$  over non-binary vectors.
Proof. Charikar (2002) proved that if  $s(x, x')$  is a similarity function for which there exists a random hash, then  $1 - s(x, x')$  must satisfy the triangle inequality (see their Lemma 1). Because  $1 - T_{DP}(x, x')$  does not satisfy the triangle inequality it follows by contradiction that there does not exist a random hash for  $T_{DP}$ .
Therefore producing random features for  $T_{DP}$  will require another approach. In the remainder of this section we present a framework to produce random features for  $T_{DP}$  by directly approximating its power series (equation 7). We first describe a method to produce random features for  $(\| x\| ^2 +\| x'\| ^2)^{-r}$  (4.1). Then we describe how these features can be combined with existing random features for the polynomial kernel to approximate  $T_{DP}$ 's truncated power series (4.2-4.3). Lastly, we present an error bound for the kernel matrix of a dataset in the spectral norm, showing that the required dimension for the sketch scales optimally with the stable rank of the kernel matrix (4.4).
4.1 Random features for the "prefactor"  $\left(\| x\| ^2 +\| x'\| ^2\right)^{-r}$
In this section we present a random feature map for the positive definite kernel  $(x, x') \mapsto (\|x\|^2 + \|x'\|^2)^{-r}$ , which we will refer to as the the prefactor. We defer all proofs to Appendix D.3. We begin with the following lemma, which defines scalar random features for the prefactor:
Lemma 4.4. If  $Z \sim \mathrm{Gamma}(s, c)$  (where  $c$  is a rate parameter), then
$$
\varphi_ {r, Z} (x) = e ^ {(1 / 2 - \| x \| ^ {2}) Z} Z ^ {(r - s) / 2} \sqrt {c ^ {- s} e ^ {(c - 1) Z} \Gamma (s) / \Gamma (r)} \tag {8}
$$
is an unbiased scalar random feature for the prefactor  $(\| x\| ^2 +\| x'\| ^2)^{-r}$  for all  $s,c > 0$
Although independent copies of  $Z$  could be combined to form an  $M$ -dimensional sketch, we instead propose to use a dependent point set  $Z_{1},\ldots ,Z_{M}$  where each element  $Z_{i}$  has a Gamma(s,c) distribution whilst maximally covering the real line. This is a well-established Quasi-Monte Carlo (QMC) technique which generally attains lower variance. We define our  $M$ -dimensional QMC features in the following lemma:
Lemma 4.5. Let  $\gamma_{s,c}$  be the inverse cumulative distribution function of a Gamma(s,c) random variable. Fix  $M,r\in \mathbb{N}$ $u\in (0,1)$ $c,s > 0$  and let  $u_{i} = u + i / M - \lfloor u + i / M\rfloor$  for  $i = 1,\ldots ,M$  Define  $\phi_{u,r}(x) = (\phi_{u,r,1}(x),\dots ,\phi_{u,r,M}(x))$  , where:
$$
\phi_ {u, r, i} (x) = \frac {1}{\sqrt {M}} \sqrt {\frac {c ^ {- s} \Gamma (s)}{\Gamma (r)}} e ^ {- (\| x \| ^ {2} - c / 2) \gamma_ {s, c} (u _ {i})} (\gamma_ {s, c} (u _ {i})) ^ {(r - s) / 2}. \tag {9}
$$
If  $u\sim \mathcal{U}(0,1)$  then  $\phi_{u,r}(x)$  forms unbiased random features of the prefactor  $(\| x\| ^2 +\| x'\| ^2)^{-r}$
Although the random features are unbiased for all  $s, c > 0$ , the value of these parameters will impact the error. We show that if  $s, c$  are suitably tuned, then the relative error can be bounded:
Lemma 4.6. Let  $x^{(1)}, \ldots, x^{(n)} \in \mathbb{R}^d$  with  $\frac{\min_i \|x^{(i)}\|^2}{\max_i \|x^{(i)}\|^2} \geq \zeta$ , and fix  $u \in [0,1]$ . Define the relative error
$$
E _ {i, j} = \frac {\phi_ {u , r} \left(x ^ {(i)}\right) \cdot \phi_ {u , r} \left(x ^ {(j)}\right) - \left(\left\| x ^ {(i)} \right\| ^ {2} + \left\| x ^ {(j)} \right\| ^ {2}\right) ^ {- r}}{\left(\left\| x ^ {(i)} \right\| ^ {2} + \left\| x ^ {(j)} \right\| ^ {2}\right) ^ {- r}}. \tag {10}
$$
If  $c = 2\zeta^2$ ,  $s = r\zeta$ , then for some constant  $C$  independent of  $r$  this error satisfies,
$$
\max  _ {1 \leq i, j \leq n} | E _ {i, j} | \leq \frac {2}{M} \frac {\Gamma (r \zeta) \zeta^ {- r \zeta}}{\Gamma (r)} (r / e) ^ {r (\zeta - 1)} (1. 3) ^ {r} \leq C (M \zeta) ^ {- 1}. \tag {11}
$$
Together, these lemmas suggest random features for the prefactor can be created by first estimating  $\zeta$  (the minimum ratio of norms of input vectors), then using the random features from Lemma 4.5 with the values of  $s, c$  specified in Lemma 4.6.
4.2 A framework to produce random features for  $T_{DP}$
There are straightforward rules for producing random features for sums and products of kernels whose individual random features are known (Duvenaud, 2014, sec. 2.6.2). Random features for kernels  $k_{1}, k_{2}$  can be concatenated (denoted  $\oplus$ ) to form random features for the sum kernel  $k_{1} + k_{2}$ , while their tensor product $^{2}$  (denoted  $\otimes$ ) forms random features for the product kernel  $k_{1} \times k_{2}$ . Our strategy to produce features for  $T_{DP}$  is to combine random features for the "prefactor" (presented in section 4.1) with random features for the polynomial kernel to produce random features for  $T_{DP}$ 's power series (equation 7) truncated at  $R$  terms.
Fix  $R \in \mathbb{N}$ , and for  $r = 1, \dots, R$ , let  $\phi_r$  be a  $m_r$ -dimensional random features map for the prefactor  $(\|x\|^2 + \|x'\|^2)^{-r}$  and let  $\psi_r$  be a  $m_r'$ -dimensional random features map for  $(x \cdot x')^r$ . The function:
$$
\tilde {\Phi} _ {R} (x) = \oplus_ {r = 1} ^ {R} \left[ \phi_ {r} (x) \otimes \psi_ {r} (x) \right] \tag {12}
$$
is therefore a random feature estimate for  $T_{DP}$ 's power series, truncated at  $R$  terms. Unfortunately, these random features have dimension  $M = \sum_{r=1}^{R} m_r m_r'$  which depends on the product of the random features dimension of  $\phi_r$  and  $\psi_r$ . Furthermore, the dimension  $m_r'$  of the random features  $\psi_r$  required to approximate the polynomial kernel  $(x \cdot x')^r$  with good accuracy can scale poorly with  $r$ . For even modest values of  $m_r, m_r'$  the resulting value of  $M$  will likely be prohibitively large.
To remedy this, we turn to recent works which propose powerful linear maps to approximate tensor products with a lower-dimensional vector. Assuming  $x^{(1)},y^{(1)}\in \mathbb{R}^{d_1}$  and  $x^{(2)},y^{(2)}\in \mathbb{R}^{d_2}$ , these maps are effectively random matrices  $\Pi \in \mathbb{R}^{m\times (d_1d_2)}$ , which exhibit a subspace embedding property whereby  $[\Pi (x^{(1)}\otimes x^{(2)})]\cdot [\Pi (y^{(1)}\otimes y^{(2)})]$  concentrates sharply around  $(x^{(1)}\otimes x^{(2)})\cdot (y^{(1)}\otimes y^{(2)})$ . Critically, the product  $\Pi (x^{(1)}\otimes x^{(2)})$  can be computed without instantiating either matrix  $\Pi$  or the tensor product  $x^{(1)}\otimes x^{(2)}$ . Examples of such methods include TENSORSKETCH and TENSORSRHT (Pagh, 2013; Pham and Pagh, 2013; Ahle et al., 2020), but for generality we will simply refer to these methods as SKETCH. Defining a series of sketches  $\mathrm{SKETCH}_r:\mathbb{R}^{m_r}\times \mathbb{R}^{m_r'}\mapsto \mathbb{R}^{\tilde{m}_r}$  we can modify  $\tilde{\Phi}_R$  from equation 12 into:
$$
\Phi_ {R} (x) = \oplus_ {r = 1} ^ {R} \operatorname {S K E T C H} _ {r} \left[ \phi_ {r} (x), \psi_ {r} (x) \right] \tag {13}
$$
2For two vectors  $x\in \mathbb{R}^{d_1}$  and  $y\in \mathbb{R}^{d_2}$  , define the tensor product  $x\otimes y = \operatorname {vec}(xy^T)\in \mathbb{R}^{d_1d_2}$
which has output dimension  $M = \sum_{r=1}^{R} \tilde{m}_r$ , i.e. without any pathological dependencies on the dimensions of  $\phi_r(x), \psi_r(x)$ . However, because these features approximate a truncated power series, they will be biased downward due to the monotonicity of the power series (7). We propose two bias correction techniques to potentially improve empirical accuracy. One approach is to normalize the random features such that  $\Phi(x) \cdot \Phi(x) = 1 = T_{DP}(x,x)$  for all  $x \in \mathbb{R}^d$ , i.e., such that the diagonal entries of the kernel matrix  $K$  are estimated exactly. A second approach is based on sketching the residual of the power series. These approaches are presented in detail in Appendix E.
4.3 Implementing the random features
Instantiating the random features from the previous subsection (13) requires making concrete choices for  $R$ ,  $\mathrm{SKETCH}_r, m_r, m_r', \tilde{m}_r, \phi_r, \psi_r$  for all  $r$ , and choosing a bias correction technique. There are many reasonable choices for  $\mathrm{SKETCH}_r$ , such as TENSORSKETCH (Pham and Pagh, 2013) and TENSORSRHT (Ahle et al., 2020). These sketches generally allow  $m_r, m_r', \tilde{m}_r$  to be chosen freely (although naturally error will increase as  $\tilde{m}_r$  decreases). Many of these sketches can also be used as random features for the polynomial kernel  $\psi_r$  (Wacker et al., 2022), either directly or as part of more complex algorithms like TREESKETCH (Ahle et al., 2020) or complex-to-real sketches (Wacker et al., 2023). The QMC random features from section 4.1 can be used for the prefactor  $\phi_r$ , with the parameters  $s, c$  chosen based on the anticipated norms of the input vectors.
The only remaining inputs are  $R$  (the number of power series terms to approximate) and  $\tilde{m}_1, \ldots, \tilde{m}_R$  (how many random features to use for each term). Assuming a fixed dimension  $M$  for the final random features, this choice involves a bias variance trade-off, as a higher value of  $R$  will reduce bias but require each term in the power series to have fewer features, thereby increasing variance. Intuitively, because the terms of the power series decrease monotonically the variance of terms for small  $r$  is likely to dominate the overall variance, and therefore we surmise that a decreasing sequence for  $\{\tilde{m}_r\}_{r=1}^R$  will be the best choice. Ultimately however we do not have theoretical results to dictate this choice in practice. We will evaluate these choices empirically in section 6.