SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Code of Ethics + broader impacts Read and acknowledged. Our work is fairly abstract and theoretical and we do not work with human-related data, so we do not foresee significant direct ethical impacts (positive or negative).
Limitations Our work is primarily theoretical, so the main strength (random features with low approximation error) is also the main weakness (the approximation error could plausibly be lower). Random features are well-studied as a general approach so the strengths and weaknesses are generally well-understood (Liu et al., 2021). For this reason we did not include an explicit limitations section. However, we did mention some other limitations in section 7.
Theory Our key theorems have minimal assumptions which are clearly stated. All theorems are proved in Appendix D.
Experiments Code to reproduce the experiments is available at: https://github.com/AustinT/tanimoto-random-features-neurips23.
Training details The experiments in this paper have minimal details and the important details are specified in Appendix F. All unspecified details should be easy to find in the code (https://github.com/AustinT/tanimoto-random-features-neurips23).
Error bars Our tables and figures include error bars.
Compute The compute costs of the experiments in this paper were quite modest and were run on a single machine with no GPU usage. The approximate computational times were:
- Section 6.1: a script to produce all the plots took approximately  $24\mathrm{h}$  to run. The long runtime was mostly due to computing errors for many settings of  $R$  and  $p$ .
- Section 6.2: for  $T_{MM}$ , SVGP and each RFGP inference took approximately  $1\mathrm{h}$  (total  $3\mathrm{h}$  per target). For  $T_{DP}$ , SVGP took  $0.5\mathrm{h}$  while RFGP took  $0.3\mathrm{h}$  (total  $1.5\mathrm{h}$  per target). Multiplying this by 5 targets and 5 trials gives a total compute time of  $\approx 112\mathrm{h}$ .
- Section 6.3: this experiment was quite fast: total runtime was perhaps  $5\mathrm{h}$ .
Reproducibility Our contribution is reproducible via the statements of our random features and our code. We aim for a high standard of reproducibility: the exact commands to reproduce the results are stated explicitly in our code's README file, and the raw data behind all plots in this paper is included alongside the code.
Safeguards We believe there is nothing high-risk which we need to safeguard.
Licenses We cite the assets which we use in the paper.
Assets We are not releasing assets.
Humans Subjects / IRB Approvals Not applicable to our paper.
We use  $\| x\|$  to denote the Euclidean norm of a vector  $x$ ,  $\| A\|_{\mathrm{op}}$  for the spectral norm and  $\| A\| _F$  for the Frobenius norm of a matrix  $A$ .  $x_{i}$  denotes the  $i$ th element of a vector  $x$ , while  $x^{(i)}$  denotes the  $i$ th vector in a list of vectors. We denote  $\mathrm{nnz}(\cdot)$  the number of non-zero entries in a vector or matrix. For two functions  $f, g$  we say  $f(z) = O(g(z))$  if there is a constant  $C$ , such that  $0\leq f(z)\leq Cg(z)$  for  $z$  large enough. Similarly,  $f(z) = \Omega (g(z))$  if there is a constant  $C$ , such that  $f(z)\geq Cg(z)$  for  $z$  large enough.  $\tilde{O}$  and  $\tilde{\Omega}$  omit poly-logarithmic factors.
Throughout the paper, we use  $d$  for the dimension of input vectors,  $n$  for the number of samples in the dataset, and  $M$  for the dimension of the sketch or random features.
C Definition of TREESKETCH
For simplicity, we shall define TREESKETCH (Ahle et al., 2020) for inputs which are  $r$ -fold tensor products  $x^{(1)} \otimes \dots \otimes x^{(r)}$  where  $r$  is a power of two, and each  $x^{(i)} \in \mathbb{R}^d$  is of the same dimension  $d$ .
The main ingredients will be two base sketches, one for the leaves of the tree, and one for internal nodes. The leaf sketch,  $T_{\mathrm{base}}$  is a random matrix in  $\mathbb{R}^{m\times d}$ . The internal sketch,  $S_{\mathrm{base}}$ , is a random matrix in  $\mathbb{R}^{m\times m^2}$ , which can be rapidly applied to the tensor product of two vectors in  $\mathbb{R}^m$ . It is possible to instantiate TreeSketch using simple sketches such as COUNTSKETCH, for the leaves, and TENSORSKETCH for internal nodes. However, our theory we assumes that  $T_{\mathrm{base}}$  is OSNAP (Nelson and Nguyen, 2013), and  $S_{\mathrm{base}}$  is TENSORSHRT, both of which enjoy a useful spectral property.
Having picked the base sketches, we can define for any power of two  $q \geq 2$ , a random map  $Q^q: \mathbb{R}^{m^q} \to \mathbb{R}^m$  whose action on a tensor product  $v^{(1)} \otimes \dots \otimes v^{(q)}$  with  $v^{(i)} \in \mathbb{R}^m$  is given by the following recursion:
$$
Q ^ {q} = Q ^ {q / 2} (S _ {1} ^ {q} (v ^ {(1)} \otimes v ^ {(2)}) \otimes S _ {2} ^ {q} (v ^ {(3)} \otimes v ^ {(4)}) \otimes \dots \otimes S _ {q / 2} ^ {q} (v ^ {(q - 1)} \otimes v ^ {(q)})).
$$
Here, the matrices  $(S_j^i)$  are independent copies of  $S_{\mathrm{base}}$ . The action of TREESKETCH  $\Pi^r$  on  $x^{(1)} \otimes \dots \otimes x^{(r)}$  is then defined by
$$
\Pi^ {r} (x ^ {(1)} \otimes \dots \otimes x ^ {(r)}) = Q ^ {r} (T _ {1} (x ^ {(1)}) \otimes \dots \otimes T _ {r} (x ^ {(r)}))
$$
where the matrices  $(T_{i})$  are independent copies of  $T_{\mathrm{base}}$ . Figure C.1 gives a schematic view of the computational tree for a tensor product with  $r = 4$ .
Whilst the action of  $\Pi^r$  was only defined for tensor products, this can be extended to arbitrary vectors in  $\mathbb{R}^{d^q}$ . Ahle et al. (2020) prove that the resulting linear sketch has many desirable subspace embedding properties, some of which are used in the proof of Theorem 4.7.
Figure C.1: Schematic view of the computational tree for TREESKETCH for a tensor product of four vectors  $x^{(1)} \otimes \dots \otimes x^{(4)}$ .
For simplicity, we drop the subscripts  $h, \Xi$  from probabilities and expectations. To show that the random features are unbiased, we re-write the expectation:
$$
\begin{array}{l} \mathbb {E} \left[ \phi_ {\Xi , h} (x) \cdot \phi_ {\Xi , h} \left(x ^ {\prime}\right) \right] \\ = \mathbb {P} (h (x) = h \left(x ^ {\prime}\right)) \mathbb {E} \left[ \Xi_ {h (x)} \Xi_ {h \left(x ^ {\prime}\right)} | h (x) = h \left(x ^ {\prime}\right) \right] \\ + \mathbb {P} (h (x) \neq h \left(x ^ {\prime}\right)) \mathbb {E} \left[ \Xi_ {h (x)} \Xi_ {h \left(x ^ {\prime}\right)} | h (x) \neq h \left(x ^ {\prime}\right) \right] \\ = \mathbb {P} (h (x) = h \left(x ^ {\prime}\right)) \mathbb {E} \left[ \Xi_ {h (x)} ^ {2} \right] \\ + \mathbb {P} (h (x) \neq h \left(x ^ {\prime}\right)) \mathbb {E} \left[ \Xi_ {h (x)} \right] \mathbb {E} \left[ \Xi_ {h \left(x ^ {\prime}\right)} \right] \\ = 1 \cdot \mathbb {P} (h (x) = h \left(x ^ {\prime}\right)) + 0 \cdot \mathbb {P} (h (x) \neq h \left(x ^ {\prime}\right)) \\ = T _ {M M} \left(x, x ^ {\prime}\right) \\ \end{array}
$$
$$
\left(\Xi_ {h (x)} = \Xi_ {h \left(x ^ {\prime}\right)}\right)
$$
$$
\left(\Xi_ {h (x)}, \Xi_ {h \left(x ^ {\prime}\right)} \text {i n d e p e n d e n t}\right)
$$
$$
(\text {a s s u m e d} \xi)
$$
$$
(h (x) \text {i s a n u n b i a s e d h a s f o r} T _ {M M})
$$
Because in general  $\mathbb{V}[X] = \mathbb{E}[X^2] - \mathbb{E}[X]^2$ , and we have  $\mathbb{E}\left[\phi_{\Xi,h}(x) \cdot \phi_{\Xi,h}(x')\right] = T_{MM}(x,x')$ , we only need to compute  $\mathbb{E}\left[\left(\phi_{\Xi,h}(x) \cdot \phi_{\Xi,h}(x')\right)^2\right]$ . This can be done using a similar decomposition:
$$
\begin{array}{l} \mathbb {E} \left[ \left(\phi_ {\Xi , h} (x) \cdot \phi_ {\Xi , h} \left(x ^ {\prime}\right)\right) ^ {2} \right] = \mathbb {P} (h (x) = h \left(x ^ {\prime}\right)) \mathbb {E} \left[ \left(\phi_ {\Xi , h} (x) \cdot \phi_ {\Xi , h} \left(x ^ {\prime}\right)\right) ^ {2} | h (x) = h \left(x ^ {\prime}\right) \right] \\ + \mathbb {P} (h (x) \neq h \left(x ^ {\prime}\right)) \mathbb {E} \left[ \left(\phi_ {\Xi , h} (x) \cdot \phi_ {\Xi , h} \left(x ^ {\prime}\right)\right) ^ {2} | h (x) \neq h \left(x ^ {\prime}\right) \right] \\ = T _ {M M} (x, x ^ {\prime}) \mathbb {E} \left[ \Xi_ {h (x)} ^ {4} \right] \\ + \left(1 - T _ {M M} (x, x ^ {\prime})\right) \mathbb {E} \left[ \Xi_ {h (x)} ^ {2} \right] \mathbb {E} \left[ \Xi_ {h (x ^ {\prime})} ^ {2} \right] \\ = T _ {M M} (x, x ^ {\prime}) \mathbb {E} \left[ \Xi_ {h (x)} ^ {4} \right] + \left(1 - T _ {M M} (x, x ^ {\prime})\right) \\ \end{array}
$$
Here, the last step follows from the moments of  $\Xi$  which were fixed by assumption. Subtracting  $T_{MM}(x,x')^2$  from the above yields the expression for the variance.
By Jensen's inequality,  $\mathbb{E}[\xi^4] \geq \mathbb{E}[\xi^2]^2$ , and  $\mathbb{E}[\xi^2] = 1$  by assumption, so  $\mathbb{E}[\xi^4] \geq 1$ . Substituting  $\mathbb{E}[\xi^4] = 1$  into the equation for  $\mathbb{V}(\phi_{\Xi,h}(x) \cdot \phi_{\Xi,h}(x'))$  gives the lower bound and completes the proof.
D.2 Proof of Theorem 4.1 and Corollary 4.2
Take any pair of inputs  $x, y \neq 0$  in  $\mathbb{R}^d$ . By the Cauchy-Schwarz inequality, we have
$$
\left| \frac {x \cdot y}{\| x \| ^ {2} + \| y \| ^ {2}} \right| \leq \frac {\| x \| \| y \|}{\| x \| ^ {2} + \| y \| ^ {2}} \leq \left(\| x \| / \| y \| + \frac {1}{\| x \| / \| y \|}\right) ^ {- 1} \leq \frac {1}{2}. \tag {15}
$$
Now, we can write the function  $T_{DP}$  as
$$
\begin{array}{l} T _ {D P} (x, y) = \frac {x \cdot y}{\| x \| ^ {2} + \| y \| ^ {2} - x \cdot y} = \frac {\frac {x \cdot y}{\| x \| ^ {2} + \| y \| ^ {2}}}{1 - \frac {x \cdot y}{\| x \| ^ {2} + \| y \| ^ {2}}} (16) \\ = \frac {x \cdot y}{\| x \| ^ {2} + \| y \| ^ {2}} \left(\frac {1}{1 - \frac {x \cdot y}{\| x \| ^ {2} + \| y \| ^ {2}}}\right) (17) \\ = \frac {x \cdot y}{\| x \| ^ {2} + \| y \| ^ {2}} \sum_ {r = 0} ^ {\infty} \left(\frac {x \cdot y}{\| x \| ^ {2} + \| y \| ^ {2}}\right) ^ {r} (18) \\ = \sum_ {r = 1} ^ {\infty} \left(\frac {x \cdot y}{\| x \| ^ {2} + \| y \| ^ {2}}\right) ^ {r}, (19) \\ \end{array}
$$
where in the identity (18) we use the bound in (15) to assert that the series is bounded by  $2^{-r}$  and thus absolutely convergent.
Furthermore, because  $(x,y)\mapsto x\cdot y$  is a positive definite kernel, and so is
$$
(x, y) \mapsto \frac {1}{\| x \| ^ {2} + \| y \| ^ {2}} = \int_ {0} ^ {\infty} e ^ {- (\| x \| ^ {2} + \| y \| ^ {2}) t} d t,
$$
each summand in the power series is positive definite because it is a product of positive definite kernels. Hence, as sums and limits of positive definite kernels are positive definite, and the power series is convergent,  $T_{DP}$  is positive definite in the space  $\mathbb{R}^d \setminus \{0\}$ . The extension to include the vector  $0 \in \mathbb{R}^d$  is straightforward, as any Gram matrix including this vector is block diagonal. We conclude that  $T_{DP}$  is a positive definite kernel in  $\mathbb{R}^d$ .
To prove Corollary 4.2, note that by the Moore-Aronszajn theorem, there exists an RKHS of functions  $(\mathcal{H},\langle \cdot \rangle_{\mathcal{H}})$  on  $\mathbb{R}^d$  with reproducing kernel  $T_{DP}$ , such that for any  $x,y\in \mathbb{R}^d$ ,  $\langle T_{DP}(x,\cdot),T_{DP}(y,\cdot)\rangle_{\mathcal{H}} = T_{DP}(x,y)$ . Then, observe that
$$
\begin{array}{l} \left\| T _ {D P} (x, \cdot) - T _ {D P} (y, \cdot) \right\| _ {\mathcal {H}} ^ {2} = \left\langle T _ {D P} (x, \cdot) - T _ {D P} (y, \cdot), T _ {D P} (x, \cdot) - T _ {D P} (y, \cdot) \right\rangle_ {\mathcal {H}} \\ = \left\langle T _ {D P} (x, \cdot), T _ {D P} (x, \cdot) \right\rangle_ {\mathcal {H}} + \left\langle T _ {D P} (y, \cdot), T _ {D P} (y, \cdot) \right\rangle_ {\mathcal {H}} \\ - 2 \langle T _ {D P} (x, \cdot), T _ {D P} (y, \cdot) \rangle_ {\mathcal {H}} \\ = 2 - 2 T _ {D P} (x, y), \\ \end{array}
$$
where in the final identity, we use the fact that  $T_{DP}(x,x) = 1$  for all  $x\in \mathbb{R}^d$ . Dividing by 2 and taking square roots on both sides, we obtain
$$
\left. \left\| \frac {1}{2} T _ {D P} (x, \cdot) - \frac {1}{2} T _ {D P} (y, \cdot) \right\| _ {\mathcal {H}} = \sqrt {1 - T _ {D P} (x , y)}, \right.
$$
where the RKHS norm on the left is clearly a distance metric.
D.3 Proofs and derivations for prefactor random features from section 4.1
D.3.1 Scalar random features (Lemma 4.4)
First, we present a derivation for the scalar random features for the prefactor  $(\| x\| ^2 +\| x'\| ^2)^{-r}$  from Lemma 4.4. As the kernel is a function of  $\| x\| ^2$  and  $\| x^{\prime}\|^{2}$ , we can deal, without loss of generality, with the one-dimensional case. We begin by observing that for any  $a,b > 0$
$$
\begin{array}{l} \left(\frac {1}{a + b}\right) ^ {r} = \int_ {0} ^ {\infty} e ^ {(1 / 2 - a) t} e ^ {(1 / 2 - b) t} \frac {t ^ {r - 1} e ^ {- t}}{\Gamma (r)} d t \\ = \int_ {0} ^ {\infty} e ^ {(1 / 2 - a) t} e ^ {(1 / 2 - b) t} c ^ {- r} e ^ {(c - 1) t} \frac {c ^ {r} t ^ {r - 1} e ^ {- c t}}{\Gamma (r)} d t \\ = \int_ {0} ^ {\infty} e ^ {(1 / 2 - a) t} e ^ {(1 / 2 - b) t} \frac {c ^ {- s} e ^ {(c - 1) t} t ^ {r - s} \Gamma (s)}{\Gamma (r)} \frac {c ^ {s} t ^ {s - 1} e ^ {- c t}}{\Gamma (s)} d t. \tag {20} \\ \end{array}
$$
$$
\varphi_ {Z} (a) = e ^ {(1 / 2 - a) Z} Z ^ {(r - s) / 2} \left(\frac {c ^ {- s} e ^ {(c - 1) Z} \Gamma (s)}{\Gamma (r)}\right) ^ {1 / 2}
$$
and letting  $Z \sim \operatorname{Gamma}(s, c)$ , we recognise the right hand side of (20) as the expectation of  $\varphi_Z(a)\varphi_Z(b)$ . Hence,
$$
\left(\frac {1}{a + b}\right) ^ {r} = \mathbb {E} (\varphi_ {Z} (a) \varphi_ {Z} (b)).
$$
This proves Lemma 4.4.
D.3.2 QMC random features (Lemma 4.5)
Next we motivate and derive our QMC random features for the prefactor, proving Lemma 4.5. It is possible to sketch  $a$  with a vector  $\varphi(a) = \frac{1}{\sqrt{M}} (\varphi_{Z_1}(a), \ldots, \varphi_{Z_M}(a))$  where  $Z_1, \ldots, Z_M$  are independent copies of  $Z$ . This makes the kernel approximation  $\varphi(a) \cdot \varphi(b)$  a Monte Carlo estimator
of the expectation, with error decreasing with the dimension  $M$  of the sketch at the standard rate  $O(M^{-1/2})$ .
However, we shall instead consider a Quasi-Monte Carlo (QMC) estimator with an error decreasing at the faster rate  $O(M^{-1})$ . Let  $\gamma_{s,c}$  be the inverse cumulative distribution function of a Gamma(s,c) random variable. With a change of variables, we can write the integral (20) as
$$
\left(\frac {1}{a + b}\right) ^ {r} = \frac {c ^ {- s} \Gamma (s)}{\Gamma (r)} \int_ {0} ^ {1} e ^ {- (a + b - c) \gamma_ {s, c} (u)} \left(\gamma_ {s, c} (u)\right) ^ {r - s} d u. \tag {21}
$$
Fix  $n > 0$ ,  $u \in [0,1]$ , and let  $u_i = (u + i / M) - \lfloor u + i / M \rfloor$  for  $i = 1, \ldots, M$ . We can now define features  $\phi_{u,r}(a) \in \mathbb{R}^M$ , as in Section 4.2:
$$
\phi_ {u, r, i} (a) = \frac {1}{\sqrt {M}} \sqrt {\frac {c ^ {- s} \Gamma (s)}{\Gamma (r)}} e ^ {- (a - c / 2) \gamma_ {s, c} (u _ {i})} (\gamma_ {s, c} (u _ {i})) ^ {(r - s) / 2}.
$$
A QMC estimator for  $(a + b)^{-r}$  is given by the dot product  $\phi_u(a)^T\phi_u(b)$ . This estimator is unbiased:
Lemma. If  $u$  is random and  $\mathcal{U}(0,1)$ , then for all  $a,b > 0$ ,
$$
\left(\frac {1}{a + b}\right) ^ {r} = \mathbb {E} \left(\phi_ {u, r} (a) \cdot \phi_ {u, r} (b)\right).
$$
Proof. By linearity of expectation,
$$
\mathbb {E} (\phi_ {u, r} (a) \cdot \phi_ {u, r} (b)) = \frac {1}{M} \sum_ {i = 1} ^ {M} \mathbb {E} \left[ \frac {c ^ {- s} \Gamma (s)}{\Gamma (r)} e ^ {- (a + b - c) \gamma_ {s, c} (u _ {i})} (\gamma_ {s, c} (u _ {i})) ^ {r - s} \right].
$$
Observing that each  $u_{i}\sim \mathcal{U}(0,1)$  , the result follows by Eq. (21).
This lemma is equivalent to Lemma 4.5, thereby proving it.
D.3.3 Relative error bound (Lemma 4.6)
In the following lemma, we establish a basic Quasi-Monte Carlo error bound.
Lemma D.1. For any  $u\in [0,1],c\leq (a + b),0 <   s\leq r,$
$$
\left| \frac {1}{(a + b) ^ {r}} - \phi_ {u, r} (a) \cdot \phi_ {u, r} (b) \right| \leq \frac {2}{M} \frac {c ^ {- s} \Gamma (s)}{\Gamma (r)} \left(\frac {r - s}{e (a + b - c)}\right) ^ {r - s}.
$$
Proof. Consider the QMC approximation of an integral  $\int_0^1 f(x)dx$  for a function  $f$  of bounded variation, using a regular net of  $n$  points. It is well known that the error of this approximation is bounded above by  $V(f) / n$  where  $V$  is the total variation norm. Hence,
$$
\left| \frac {1}{(a + b) ^ {r}} - \phi_ {u, r} (a) \cdot \phi_ {u, r} (b) \right| \leq \frac {1}{n} \frac {c ^ {- s} \Gamma (s)}{\Gamma (r)} V (f)
$$
for  $f(u) = e^{-(a + b - c)\gamma_{s,c}(u)}(\gamma_{s,c}(u))^{r - s}$  on  $0 \leq u \leq 1$ . As  $\gamma_{s,c}$  is monotone increasing,  $f$  is unimodal tending to 0 as  $u \to 0$  and  $u \to 1$ . Thus  $V(f) = 2\max_{0 \leq u \leq 1} f(u)$ . The maximum of  $f$  is attained where  $\gamma_{s,c}(u) = (r - s) / (a + b - c)$ , hence
$$
V (f) = 2 \left(\frac {r - s}{e (a + b - c)}\right) ^ {r - s}.
$$
We are interested in sketching  $(\| x^{(i)}\| ^2 +\| x^{(j)}\| ^2)^{-r}$  for every pair of inputs  $x^{(i)},x^{(j)}$  in a dataset  $\{x^{(1)},\ldots ,x^{(n)}\}$ . The previous lemma can be used to choose values  $c$  and  $s\leq r$  which minimise the relative error
$$
E _ {i, j} = \frac {\phi_ {u , r} (x ^ {(i)}) \cdot \phi_ {u , r} (x ^ {(j)}) - (\| x ^ {(i)} \| ^ {2} + \| x ^ {(j)} \| ^ {2}) ^ {- r}}{(\| x ^ {(i)} \| ^ {2} + \| x ^ {(j)} \| ^ {2}) ^ {- r}}.
$$
The kernel  $k(x,y)$  is invariant to scaling  $x$  and  $y$ , i.e.,  $k(x,y) = k(\ell x,\ell y)$  for all  $\ell > 0$ . Thus, if we are interested in approximating the Gram matrix for  $\{x^{(1)},\ldots ,x^{(n)}\}$ , we may assume without loss of generality that  $\max_i\| x^{(i)}\| ^2 = 1$  and let  $\zeta = \min_{i}\| x^{(i)}\|^{2}$ . Here,  $\zeta$  will act as a condition number for the dataset, which will determine the error of the random feature approximation.
By Lemma D.1, we have
$$
\begin{array}{l} \left| E _ {i, j} \right| \leq \left(\left\| x ^ {(i)} \right\| ^ {2} + \left\| x ^ {(j)} \right\| ^ {2}\right) ^ {r} \frac {2}{M} \frac {c ^ {- s} \Gamma (s)}{\Gamma (r)} \left(\frac {r - s}{e \left(\left\| x ^ {(i)} \right\| ^ {2} + \left\| x ^ {(j)} \right\| ^ {2} - c\right)}\right) ^ {r - s} \\ = (\| x ^ {(i)} \| ^ {2} + \| x ^ {(j)} \| ^ {2}) ^ {r} \frac {2}{M} \frac {c ^ {- s} \Gamma (s) (r - s) ^ {r - s}}{\Gamma (r) e ^ {(r - s)}} \left(\frac {1}{\| x ^ {(i)} \| ^ {2} + \| x ^ {(j)} \| ^ {2} - c}\right) ^ {r - s} \\ = \frac {2}{M} \frac {\Gamma (s) (r - s) ^ {r - s}}{\Gamma (r) e ^ {(r - s)}} \left(1 - \frac {c}{\| x ^ {(i)} \| ^ {2} + \| x ^ {(j)} \| ^ {2}}\right) ^ {s - r} \left(\frac {c}{\| x ^ {(i)} \| ^ {2} + \| x ^ {(j)} \| ^ {2}}\right) ^ {- s}. \\ \end{array}
$$
The last two factors are log-convex in  $c / (\| x^{(i)}\| ^2 +\| x^{(j)}\| ^2) > 0$  . Hence, for a fixed  $c$  , the product
$$
\left(1 - \frac {c}{\| x ^ {(i)} \| ^ {2} + \| x ^ {(j)} \| ^ {2}}\right) ^ {s - r} \left(\frac {c}{\| x ^ {(i)} \| ^ {2} + \| x ^ {(j)} \| ^ {2}}\right) ^ {- s}
$$
is maximised at either  $\| x^{(i)}\| ^2 +\| x^{(j)}\| ^2 = 2$  or  $\| x^{(i)}\| ^2 +\| x^{(j)}\| ^2 = 2\zeta$  . Therefore,
$$
| E _ {i, j} | \leq \frac {2}{M} \frac {\Gamma (s) (r - s) ^ {r - s}}{\Gamma (r) e ^ {(r - s)}} \left[ \left(1 - \frac {c}{2}\right) ^ {s - r} \left(\frac {c}{2}\right) ^ {- s} \vee \left(1 - \frac {c}{2 \zeta}\right) ^ {s - r} \left(\frac {c}{2 \zeta}\right) ^ {- s} \right].
$$
Using again the log-convexity of  $y \mapsto (1 - y)^{s - r}y^{-s}$ , we find that
$$
\left(1 - \frac {c}{2}\right) ^ {s - r} \left(\frac {c}{2}\right) ^ {- s}
$$
is minimised as a function of  $c$  at  $c_{1} = 2s / r$ . And
$$
\left(1 - \frac {c}{2 \zeta}\right) ^ {s - r} \left(\frac {c}{2 \zeta}\right) ^ {- s}
$$
is minimised at  $c_2 = 2\zeta s / r$ . Recall that Lemma D.1 requires  $0 \leq c \leq \| x^{(i)}\|^2 + \| x^{(j)}\|^2$  for all  $i, j$  or  $0 \leq c \leq 2\zeta$ , which is satisfied by  $c_2$  when  $s \leq r$ , but not necessarily by  $c_1$ . Thus, it is reasonable to set  $c = c_2$ , which leads to
$$
\begin{array}{l} \left| E _ {i, j} \right| \leq \frac {2}{M} \frac {\Gamma (s) (r - s) ^ {r - s}}{\Gamma (r) e ^ {(r - s)}} \left[ \left(1 - \frac {\zeta s}{r}\right) ^ {s - r} \left(\frac {\zeta s}{r}\right) ^ {- s} \vee \left(1 - \frac {s}{r}\right) ^ {s - r} \left(\frac {s}{r}\right) ^ {- s} \right] \\ \leq \frac {2}{M} \frac {\Gamma (s) (r - s) ^ {r - s} r ^ {r} s ^ {- s}}{\Gamma (r) e ^ {(r - s)}} \left[ (r - \zeta s) ^ {s - r} \zeta^ {- s} \vee (r - s) ^ {s - r} \right] \\ \leq \frac {2}{M} \frac {\Gamma (s) (s / e) ^ {- s}}{\Gamma (r) (r / e) ^ {- r}} \left[ \left(\frac {r - \zeta s}{r - s}\right) ^ {s - r} \zeta^ {- s} \vee 1 \right]. \\ \end{array}
$$
When  $\zeta = 1$  (all inputs  $x^{(i)}$  have the same norm), setting  $s = r$  leads to an error bound of  $4 / n$ . However, small values of  $\zeta$  can make the upper bound blow up unless we tune  $s$  suitably. For a given value of  $\zeta$ , this upper bound could be maximised numerically over  $0 < s \leq r$ . However, setting  $s = r\zeta$ , we obtain
$$
\begin{array}{l} | E _ {i, j} | \leq \frac {2}{M} \frac {\Gamma (r \zeta) (r \zeta / e) ^ {- r \zeta}}{\Gamma (r) (r / e) ^ {- r}} \left[ \left(\frac {1 - \zeta^ {2}}{1 - \zeta}\right) ^ {r (\zeta - 1)} \zeta^ {- r \zeta} \vee 1 \right] \\ \leq \frac {2}{M} \frac {\Gamma (r \zeta) \zeta^ {- r \zeta}}{\Gamma (r)} (r / e) ^ {r (\zeta - 1)} \left[ \left(\left(\frac {1 - \zeta^ {2}}{1 - \zeta}\right) ^ {\zeta - 1} \zeta^ {- \zeta}\right) ^ {r} \vee 1 \right] \\ \leq \frac {2}{M} \frac {\Gamma (r \zeta) \zeta^ {- r \zeta}}{\Gamma (r)} (r / e) ^ {r (\zeta - 1)} \rho^ {r} \\ \end{array}
$$
where  $\rho \approx 1.2$  is defined by
$$
\rho := \max  _ {0 \leq \zeta \leq 1} \left(\frac {1 - \zeta^ {2}}{1 - \zeta}\right) ^ {\zeta - 1} \zeta^ {- \zeta}.
$$
This proves the first inequality in Lemma 4.6.
To conclude the proof of Lemma 4.6, we study the behaviour of the upper bound as it diverges when  $\zeta \to 0$ . Using the asymptotic  $\Gamma(x) \sim 1/x$  as  $x \to 0$ , we have
$$
\frac {2}{M} \frac {\Gamma (r \zeta) \zeta^ {- r \zeta}}{\Gamma (r)} (r / e) ^ {r (\zeta - 1)} \rho^ {r} \sim (M \zeta) ^ {- 1} \frac {2 (r / e) ^ {- r} \rho^ {r}}{r \Gamma (r)}
$$
where  $\frac{2(r / e)^{-r}\rho^r}{r\Gamma(r)}$  is bounded for  $r\geq 1$ . Therefore, the upper bound is  $O(M^{-1}\zeta^{-1})$ . This completes the proof.
D.4 Proof of Theorem 4.7
We begin by defining the stable rank, a common notion of effective dimension for a data matrix  $A \in \mathbb{R}^{d \times n}$  or its associated Gram matrix  $A^T A$ .
Definition D.2. The stable rank  $\operatorname{sr}(A)$  of a data matrix  $A \in \mathbb{R}^{d \times n}$  is given by
$$
\operatorname {s r} (A) = \frac {\| A \| _ {F} ^ {2}}{\| A \| _ {\mathrm {o p}} ^ {2}}.
$$
For a positive semi-definite matrix  $K\in \mathbb{R}^{n\times n}$ , define  $\tilde{\mathrm{sr}} (K) = \mathrm{Tr}(K) / \| K\|_{\mathrm{op}}$
The stable rank of  $A$  is upper bounded by the rank, but it is insensitive to small singular values, so it can be much smaller than  $\min(n, d)$ . Note that  $\operatorname{sr}(A) = \tilde{\operatorname{sr}}(A^T A)$ . The function  $\tilde{\operatorname{sr}}(K)$  extends the concept of stable rank to a general kernel matrix  $K$ .
We now re-state theorem 4.7 for convenience, and provide a proof:
Theorem. For any  $n \geq 1$ , let  $x^{(1)}, \ldots, x^{(n)} \in \mathbb{R}^d$  be a set of inputs with  $\frac{\min_i \|x^{(i)}\|^2}{\max_i \|x^{(i)}\|^2} \geq \zeta$ . Let  $K$  be the matrix with entries  $K_{i,j} = T_{DP}(x^{(i)}, x^{(j)})$ . For all  $\varepsilon > 0$ , there exists an oblivious sketch  $\Phi: \mathbb{R}^d \to \mathbb{R}^m$  with  $m = \tilde{\Omega}(\tilde{s}r(K) / \varepsilon^2)$ , such that
$$
\mathbb {P} _ {\Phi} \left(\| \widehat {K} - K \| _ {o p} \geq \varepsilon \| K \| _ {o p}\right) \leq \frac {1}{p o l y (n)} \tag {22}
$$
We begin by specifying the sketch  $\Phi$ . Define
$$
\Phi (x) = \oplus_ {r = 1} ^ {R} \Phi^ {r} (x)
$$