SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} f _ {\mathrm {D E}} ^ {(k)} (\boldsymbol {\omega}, \mathbf {x}) = D \exp \left(\boldsymbol {\omega} ^ {\top} \mathbf {Q} \widetilde {\mathbf {A}} \mathbf {Q} ^ {\top} \boldsymbol {\omega} + \boldsymbol {\omega} ^ {\top} \mathbf {B} ^ {(k)} \mathbf {x} + \mathbf {x} ^ {\top} \mathbf {C} ^ {(k)} \mathbf {x}\right) = \\ = D \exp (\widetilde {\boldsymbol {\omega}} ^ {\top} \widetilde {\mathbf {A}} \widetilde {\boldsymbol {\omega}} + \widetilde {\boldsymbol {\omega}} ^ {\top} \widetilde {\mathbf {B}} ^ {(k)} \mathbf {x} + \mathbf {x} ^ {\top} \mathbf {C} ^ {(k)} \mathbf {x}) = \widetilde {f} _ {\mathrm {D E}} ^ {(k)} (\widetilde {\boldsymbol {\omega}}, \mathbf {x}), \\ \end{array}
$$
where  $\widetilde{\mathbf{B}}^{(k)} = \mathbf{Q}^{\top}\mathbf{B}^{(k)},\widetilde{\omega} = \mathbf{Q}^{\top}\omega \sim \mathcal{N}(\mathbf{0}_d,\mathbf{I}_d)$  since the distribution  $\omega \sim \mathcal{N}(\mathbf{0}_d,\mathbf{I}_d)$  is isometric, i.e. rotation-invariant and  $\widetilde{f}_{\mathrm{DE}}^{(k)}$  are DERFs with parameters  $\widetilde{\mathbf{A}},\widetilde{\mathbf{B}}^{(k)},\mathbf{C}^{(k)},D$ . We conclude that with any  $\mathbf{A}$ ,  $f_{\mathrm{DE}}^{(k)}(\boldsymbol {\omega},\mathbf{x})$  can be expressed as DERFs  $\widetilde{f}_{\mathrm{DE}}^{(k)}$  with  $\widetilde{\mathbf{A}}\in \mathbb{D}_d$ . Hence, hereafter we only consider  $\mathbf{A}\in \mathbb{D}_d$  without loss of generality.
Since  $\mathbf{B}^{(k)},\mathbf{C}^{(k)}$  are dense matrices in general, evaluation of  $f_{\mathrm{DE}}^{(k)}(\pmb {\omega},\mathbf{x})$  takes  $O(d^{2})$  time which is bigger than the  $O(d)$  complexity for TrigRFs, PosRFs and GERFs. However,  $\mathbf{P}$  and  $\mathbf{S}$  matrices (2) can be still computed in a time subquadratic in  $L$ . For that, precompute  $(\mathbf{B}^{(k)})^{\top}\pmb{\omega}^{(m)},\mathbf{C}^{(1)}\mathbf{x}^{(i)},$ $\mathbf{C}^{(2)}\mathbf{y}^{(j)}$  for all  $k\in \{1,2\}$ ,  $1\leq m\leq M$ ,  $1\leq i,j\leq L$  in  $O((M + L)d^{2})$  time. Then, computing  $f_{\mathrm{DE}}^{(1)}(\pmb{\omega}^{(m)},\mathbf{x}^{(i)})$ ,  $f_{\mathrm{DE}}^{(2)}(\pmb{\omega}^{(m)},\mathbf{y}^{(j)})$  for all  $1\leq i,j\leq L$ ,  $1\leq m\leq M$  takes  $O(LMd)$  operations. The complexity of constructing (2) then is  $O(L(Md + d^{2}) + Md^{2})$  which is still subquadratic in  $L$ .
Our goal is to minimize  $\overline{\mathcal{L}} (\theta_{\mathrm{DE}};\mathcal{X},\mathcal{Y},\mathcal{T}_{\mathrm{DE}})$  for  $\theta_{\mathrm{DE}} = \{\mathbf{A},\mathbf{B}^{(1)},\mathbf{B}^{(2)},\mathbf{C}^{(1)},\mathbf{C}^{(2)},D\}$ . However, we find that even for a single pair of  $\mathbf{x},\mathbf{y}$  it's unclear how to minimize the variance (9) in closed form. Hence, below we consider special cases where an analytic solution is feasible.
Figure 2: Log of the relative variance of new and existing RF mechanisms, mean value over multiple samples.  $0.1 \leq \sigma \leq 1$ .
4.1 Asymmetric dense-exponential random features
Define RFs  $\mathcal{T}_{\mathrm{ADE}} = \langle \nu_{\mathrm{ADE}},f^{(1)}_{\mathrm{ADE}},f^{(2)}_{\mathrm{ADE}}\rangle$  in the same way as  $\mathcal{T}_{\mathrm{DE}}$  with the only difference that  $\mathbf{A} = A\mathbf{I}_d$  where  $A\in \mathbb{R}$ . We refer to these RFs as asymmetric dense-exponential RFs (ADERFs) since  $f_{\mathrm{ADE}}^{(1)}\neq f_{\mathrm{ADE}}^{(2)}$  in general. The only additional restriction of ADERFs compared to DERFs is that all diagonal entries of  $\mathbf{A}\in \mathbb{D}_d$  are the same. The parameters of  $\mathcal{T}_{\mathrm{ADE}}$  are  $\theta_{\mathrm{ADE}} = \{A,\mathbf{B}^{(1)},\mathbf{B}^{(2)},\mathbf{C}^{(1)},\mathbf{C}^{(2)},D\}$ . By  $\Theta_{\mathrm{ADE}}$  denote a set of all possible  $\theta_{\mathrm{ADE}}$  's resulting in correct RFs for the kernel  $K^{(0)}$ , i.e. satisfying Theorem 4.1 with  $\mathbf{A} = A\mathbf{I}_d$ . The following result gives an analytic formula for a minimum of  $\overline{\mathcal{L}} (\theta_{\mathrm{ADE}};\mathcal{X},\mathcal{Y},\mathcal{T}_{\mathrm{ADE}})$ . In the theorem, we use notions of SVD and eigendecomposition of a symmetric matrix [45] (all proofs are in the Appendix).
Theorem 4.2. Let  $\mathcal{X} = \{\mathbf{x}^{(i)}\in \mathbb{R}^d\}_{i = 1}^L$ $\mathcal{V} = \{\mathbf{y}^{(j)}\in \mathbb{R}^d\}_{j = 1}^L$  . Let  $\mathbf{M}^{(1)} = \frac{1}{L}\sum_{i = 1}^{L}\mathbf{x}^{(i)}(\mathbf{x}^{(i)})^{\top}$ $\mathbf{M}^{(2)} = \frac{1}{L}\sum_{j = 1}^{L}\mathbf{y}^{(j)}(\mathbf{y}^{(j)})^{\top}$  . Suppose that  $\mathbf{M}^{(1)},\mathbf{M}^{(2)}\in \mathbb{S}_d$  are nonsingular (implying that  $L\geq d)$  . Define  $\mu^{(3)} = d^{-1}L^{-2}\left(\sum_{i = 1}^{L}\mathbf{x}^{(i)}\right)^{\top}\left(\sum_{j = 1}^{L}\mathbf{y}^{(j)}\right)\in \mathbb{R}$  . For  $k\in \{1,2\}$  , let  $\mathbf{M}^{(k)} =$ $\mathbf{Q}^{(k)}\boldsymbol{\Lambda}^{(k)}(\mathbf{Q}^{(k)})^{\top}$  be eigendecomposition of a symmetric  $\mathbf{M}^{(k)}$  where  $\mathbf{Q}^{(k)}\in \mathbb{O}_d$  .  $\pmb{\Lambda}^{(k)}\in \mathbb{D}_d$  has strictly positive diagonal values since  $\mathbf{M}^{(k)}\succeq 0$  by definition and  $\mathbf{M}^{(k)}$  is nonsingular. Let  $\mathbf{U\Sigma V}^{\top}$  be SVD decomposition of  $(\Lambda^{(1)})^{\frac{1}{2}}(\mathbf{Q}^{(1)})^{\top}\mathbf{Q}^{(2)}(\Lambda^{(2)})^{\frac{1}{2}}$  where  $\mathbf{U},\mathbf{V}\in \mathbb{O}^d$ $\pmb {\Sigma}\in \mathbb{D}_d$  has nonnegative diagonal entries.
One of the solutions  $\pmb{\theta}_{\mathrm{ADE}}^{*} = \{A, \mathbf{B}^{(1)}, \mathbf{B}^{(2)}, \mathbf{C}^{(1)}, \mathbf{C}^{(2)}, D\}$  of  $\min_{\pmb{\theta}_{\mathrm{ADE}} \in \Theta_{\mathrm{ADE}}} \overline{\mathcal{L}}(\pmb{\theta}_{\mathrm{ADE}}; \mathcal{X}, \mathcal{Y}, \mathcal{T}_{\mathrm{ADE}})$  is as follows. Set  $\phi = 2d^{-1} \sum_{l=1}^{d} \pmb{\Sigma}_{l,l} + 2\mu^{(3)}$  and, for  $k \in \{1,2\}$ ,
$$
A = \frac {1}{1 6} \left(1 - 2 \phi - \sqrt {\left(2 \phi + 1\right) ^ {2} + 8 \phi}\right), \quad \mathbf {B} ^ {(1)} = \sqrt {1 - 4 A} \boldsymbol {\Sigma} ^ {1 / 2} \mathbf {U} ^ {\top} (\boldsymbol {\Lambda} ^ {(1)}) ^ {- 1 / 2} (\mathbf {Q} ^ {(1)}) ^ {\top},
$$
$$
\mathbf {B} ^ {(2)} = \sqrt {1 - 4 A} \pmb {\Sigma} ^ {- 1 / 2} \mathbf {U} ^ {\top} (\pmb {\Lambda} ^ {(1)}) ^ {1 / 2} (\mathbf {Q} ^ {(1)}) ^ {\top}, \quad \mathbf {C} ^ {(k)} = - \frac {1}{2 (1 - 4 A)} (\mathbf {B} ^ {(k)}) ^ {\top} \mathbf {B} ^ {(k)}, \quad D = (1 - 4 A) ^ {d / 4}.
$$
Further, we have:  $\overline{\mathcal{L}} (\pmb{\theta}_{\mathrm{ADE}}^{*};\mathcal{X},\mathcal{Y},\mathcal{T}_{\mathrm{ADE}}) =$
$$
d \left(\log (1 - 4 A) - \frac {1}{2} \log (1 - 8 A) + 2 (1 - 8 A) ^ {- 1} \left(d ^ {- 1} \sum_ {l = 1} ^ {d} \boldsymbol {\Sigma} _ {l, l} + \mu^ {(3)}\right) + 2 \mu^ {(3)}\right). \tag {10}
$$
Theorem 4.2 implies an algorithm for finding  $\theta_{\mathrm{ADE}}^*$  efficiently. Namely, compute  $\mathbf{M}^{(k)}, k \in \{1,2\}$  ( $O(Ld^2)$  time) and  $\mu^{(3)}$  ( $O(Ld)$  time). Then, perform matrix decompositions to obtain  $\mathbf{Q}^{(k)}, \boldsymbol{\Lambda}^{(k)}, k \in \{1,2\}$ , and  $\mathbf{U}, \boldsymbol{\Sigma}, \mathbf{V}$  in  $O(d^3)$  time. After that,  $A, \mathbf{B}^{(1)}, \mathbf{B}^{(2)}, \mathbf{C}^{(1)}, \mathbf{C}^{(2)}, D$  can be all evaluated in  $O(d^3)$  time using formulae from Theorem 4.2. The total time complexity of the approximation scheme is therefore  $O(L(Md + d^2) + Md^2 + d^3)$  which is subquadratic in  $L$  as required.
4.2 Symmetric dense-exponential random features
Define  $\mathcal{T}_{\mathrm{SDE}} = \langle \nu_{\mathrm{SDE}},f_{\mathrm{SDE}}^{(1)},f_{\mathrm{SDE}}^{(2)}\rangle$  in the same way as  $\mathcal{T}_{\mathrm{DE}}$  with the only difference that  $\mathbf{B}^{(1)} = \mathbf{B}^{(2)} = \mathbf{B}$ . From the conditions in Theorem 4.1 it follows immediately that also  $\mathbf{C}^{(1)} = \mathbf{C}^{(2)} = \mathbf{C}$ . Hence,  $f_{\mathrm{SDE}}^{(1)} = f_{\mathrm{SDE}}^{(2)}$  and we refer to these RFs as symmetric dense-exponential RFs (SDERFs). The parameters of  $\mathcal{T}_{\mathrm{SDE}}$  are  $\pmb{\theta}_{\mathrm{SDE}} = \{\mathbf{A},\mathbf{B},\mathbf{C},D\}$ . By  $\Theta_{\mathrm{SDE}}$  denote a set of all possible  $\pmb{\theta}_{\mathrm{SDE}}$ 's resulting in correct RFs for the kernel  $K^{(0)}$ , i.e. satisfying conditions from Theorem 4.1 with  $\mathbf{B}^{(k)} = \mathbf{B}$ ,  $\mathbf{C}^{(k)} = \mathbf{C}$ ,  $k\in \{1,2\}$ . The following theorem gives an analytic solution for a global minimum of  $\overline{\mathcal{L}} (\pmb{\theta}_{\mathrm{SDE}};\mathcal{X},\mathcal{Y},\mathcal{T}_{\mathrm{SDE}})$ .
Theorem 4.3. Let  $\mathcal{X} = \{\mathbf{x}^{(i)}\in \mathbb{R}^d\}_{i = 1}^L$ $\mathcal{Y} = \{\mathbf{y}^{(j)}\in \mathbb{R}^d\}_{j = 1}^L$  and let  $\mathbf{M}^{(1)},\mathbf{M}^{(2)}$  be defined as in Theorem 4.2 (but without the restriction of nonsingularity) and define  $\pmb{\mu}^{(4)} = \frac{1}{L}\sum_{i = 1}^{L}\mathbf{x}^{(i)}\in \mathbb{R}^{d}$ ,  $\pmb{\mu}^{(5)} = \frac{1}{L}\sum_{j = 1}^{L}\mathbf{y}^{(j)}\in \mathbb{R}^{d}$ . Further, let  $\mathbf{Q}^{(3)}\pmb{\Lambda}^{(3)}(\mathbf{Q}^{(3)})^\top$  be eigendecomposition of a symmetric positive semidefinite matrix  $\mathbf{M}^{(1)} + \pmb{\mu}^{(4)}(\pmb{\mu}^{(5)})^\top +\pmb{\mu}^{(5)}(\pmb{\mu}^{(4)})^\top +\mathbf{M}^{(2)}$  where  $\mathbf{Q}^{(3)}\in \mathbb{O}_d$  and  $\pmb{\Lambda}^{(3)}\in \mathbb{D}_d$  with nonnegative diagonal entries. Let the entries on the diagonal of  $\pmb{\Lambda}^{(3)}$  be sorted in the non-ascending order.
One of the solutions  $\pmb{\theta}_{\mathrm{SDE}}^{*} = \{\mathbf{A},\mathbf{B},\mathbf{C},D\}$  of  $\min_{\pmb{\theta}_{\mathrm{SDE}}\in \Theta_{\mathrm{SDE}}}\overline{\mathcal{L}} (\pmb{\theta}_{\mathrm{SDE}};\mathcal{X},\mathcal{Y},\mathcal{T}_{\mathrm{SDE}})$  is as follows.  $\mathbf{A}\in \mathbb{D}_d$  , for all  $1\leq l\leq d$  ..  $\mathbf{A}_{l,l} = \frac{1}{16}\left(1 - 2\pmb{\Lambda}_{l,l}^{(3)} - \sqrt{\left(2\pmb{\Lambda}_{l,l}^{(3)} + 1\right)^2 + 8\pmb{\Lambda}_{l,l}^{(3)}}\right),$ $\mathbf{B} = (\mathbf{I}_d - 4\mathbf{A})^{1 / 2}(\mathbf{Q}^{(3)})^{\top}$ $\mathbf{C} = -\frac{1}{2}\mathbf{I}_d$ $D = \operatorname *{det}(\mathbf{I}_d - 4\mathbf{A})^{1 / 4}$  . Further, we have:  $\overline{\mathcal{L}} (\pmb{\theta}_{\mathrm{SDE}};\mathcal{X},\mathcal{Y},\mathcal{T}_{\mathrm{SDE}}) =$
$$
\sum_ {l = 1} ^ {d} \left(\log (1 - 4 \mathbf {A} _ {l, l}) - \frac {1}{2} \log (1 - 8 \mathbf {A} _ {l, l}) + \left(1 + (1 - 8 \mathbf {A} _ {l, l}) ^ {- 1}\right) \boldsymbol {\Lambda} _ {l, l} ^ {(3)}\right) - \frac {1}{L} \sum_ {i = 1} ^ {L} \| \mathbf {x} ^ {(i)} \| ^ {2} - \frac {1}{L} \sum_ {j = 1} ^ {L} \| \mathbf {y} ^ {(j)} \| ^ {2} \tag {11}
$$
Again, Theorem 4.3 implies an algorithm for finding  $\pmb{\theta}_{\mathrm{SDE}}^{*}$  in a time subquadratic in  $L$ . That is, we can compute  $\mathbf{M}^{(1)}, \mathbf{M}^{(2)}, \mu^{(3)}, \mu^{(4)}, \mu^{(5)}$  in  $O(Ld^2)$  total time. Then, perform an eigendecomposition to obtain  $\mathbf{Q}^{(3)}, \mathbf{\Lambda}^{(3)}$  in  $O(d^3)$  time. After that,  $\mathbf{A}, \mathbf{B}, \mathbf{C}, D$  can be computed in  $O(d^3)$  time using formulae from Theorem 4.3. The total time complexity of the approximation scheme is the same as for ADERFs:  $O(L(Md + d^2) + Md^2 + d^3)$  or  $O(L(Md + d^2) + Md^2)$  if we assume that  $L \geq d$ .
4.3 Simplified ADERFs
While having a compact and closed-form expression, both ADERFs and SDERFs rely on eigendecomposition and SVD decompositions: operations for which implementation has not yet matured in popular deep learning libraries with GPU and TPU support. For this reason, we propose simplified ADERFs (SADERFs)  $\mathcal{T}_{\mathrm{SADE}} = \langle \nu_{\mathrm{SADE}},f_{\mathrm{SADE}}^{(1)},f_{\mathrm{SADE}}^{(2)}\rangle$  which extend GERFs but require only basic unary operations. SADERFs are defined via GERFs as follows:  $\Omega_{\mathrm{SADE}} = \mathbb{R}^d$ $\nu_{\mathrm{SADE}} = \mathcal{N}(0,1)^d$ $f_{\mathrm{SADE}}^{(1)}(\pmb {\omega},\mathbf{x}) = f_{\mathrm{GE}}^{(1)}(\pmb {\omega},\pmb {\Psi}\mathbf{x}),f_{\mathrm{SADE}}^{(2)}(\pmb {\omega},\mathbf{y}) = f_{\mathrm{GE}}^{(2)}(\pmb {\omega},\pmb{\Psi}^{-1}\mathbf{y})$  where  $\pmb {\Psi}\in \mathbb{D}_d$  is a diagonal matrix with nonzero diagonal entries. First of all,  $\mathcal{T}_{\mathrm{SADE}}$  are valid random features for the softmax kernel  $K^{(0)}$  since  $\mathbb{E}_{\nu_{\mathrm{SADE}}}[f_{\mathrm{SADE}}^{(1)}(\pmb {\omega},\mathbf{x})f_{\mathrm{SADE}}^{(2)}(\pmb {\omega},\mathbf{y})] = \mathbb{E}_{\nu_{\mathrm{GE}}}[f_{\mathrm{GE}}^{(1)}(\pmb {\omega},\pmb {\Psi}\mathbf{x})f_{\mathrm{GE}}^{(2)}(\pmb {\omega},\pmb{\Psi}^{-1}\mathbf{y})] =$ $K^{(0)}(\Psi \mathbf{x},\Psi^{-1}\mathbf{y}) = K^{(0)}(\mathbf{x},\mathbf{y})$  , where we use  $K^{(0)}(\mathbf{x},\mathbf{y}) = \exp (\mathbf{x}^{\top}\mathbf{y})$  by the definition.
We find  $\Psi$  by optimizing the objective (8) for  $\mathcal{T}_{\mathrm{SADE}}$ , the form of which is easily deduced from  $\overline{\mathcal{L}} (\theta_{\mathrm{GE}};\mathcal{X},\mathcal{Y},\mathcal{T}_{\mathrm{GE}})$ :
$$
\begin{array}{l} \overline {{\mathcal {L}}} (\boldsymbol {\theta} _ {\mathrm {S A D E}}; \mathcal {X}, \mathcal {Y}, \mathcal {T} _ {\mathrm {S A D E}}) - d \log \left(\frac {1 - 4 A}{\sqrt {1 - 8 A}}\right) = \frac {2 - 8 A}{1 - 8 A} \frac {1}{L ^ {2}} \sum_ {i, j} \| \boldsymbol {\Psi} \mathbf {x} ^ {(i)} + \boldsymbol {\Psi} ^ {- 1} \mathbf {y} ^ {(j)} \| ^ {2} \\ - \frac {1}{L} \sum_ {i} \left(\| \Psi \mathbf {x} ^ {(i)} \| ^ {2} + \| \Psi^ {- 1} \mathbf {y} ^ {(j)} \| ^ {2}\right) = \frac {1}{L ^ {2} (1 - 8 A)} \sum_ {i, j} \| \Psi \mathbf {x} ^ {(i)} + \Psi^ {- 1} \mathbf {y} ^ {(j)} \| ^ {2} + \frac {2}{L ^ {2}} \sum_ {i, j} \left(\mathbf {x} ^ {(i)}\right) ^ {\top} \mathbf {y} ^ {(j)}, \tag {12} \\ \end{array}
$$
where we move a term not depending on  $\Psi$  to the left-hand side. Since  $1 - 8A > 0$ , we conclude that minimizing (12) is equivalent to minimizing  $\sum_{i,j} \| \Psi \mathbf{x}^{(i)} + \Psi^{-1} \mathbf{y}^{(j)} \|^2 =$
$$
\sum_ {l} \sum_ {i, j} \left(\Psi_ {l, l} \mathbf {x} _ {l} ^ {(i)} + \Psi_ {l, l} ^ {- 1} \mathbf {y} _ {l} ^ {(j)}\right) ^ {2} = \sum_ {l} \sum_ {i, j} \left(\Psi_ {l, l} ^ {2} \left(\mathbf {x} _ {l} ^ {(i)}\right) ^ {2} + 2 \mathbf {x} _ {l} ^ {(i)} \mathbf {y} _ {l} ^ {(j)} + \Psi_ {l, l} ^ {- 2} \left(\mathbf {y} _ {l} ^ {(j)}\right) ^ {2}\right). \tag {13}
$$
Figure 3: Kernel classification, test accuracy  $(\%)$ . The last plot shows average curves over 8 benchmarks. We observe that our proposed method SDERF, shows the best accuracy across most of the (benchmark,  $M$ ) pairs and also shows the best average performance.
Optimizing (13) reduces to independent optimization problems with respect to  $\Psi_{l,l}$ ,  $1\leq l\leq d$ . Each problem is convex and the solution is found trivially by setting the derivative to zero: for all  $1\leq l\leq d:\Psi_{l,l}^{*} = (\sum_{j}(\mathbf{y}_{l}^{(j)})^{2} / \sum_{i}(\mathbf{x}_{l}^{(i)})^{2})^{1 / 4}$ . The solution can be computed in  $O(dL)$  time, after which the parameters of  $f_{\mathrm{GE}}^{(1)},f_{\mathrm{GE}}^{(2)}$  can be found efficiently as described in Section 2.3.
It is easy to see that  $\mathcal{T}_{\mathrm{SADE}}$  are a special case of ADERFs (Section 4.1) which explains their name. Furthermore, the case  $\Psi = \mathbf{I}_d$  reduces  $\mathcal{T}_{\mathrm{SADE}}$  to  $\mathcal{T}_{\mathrm{GE}}$ , hence the latter is a special case of the former. Figure 1 (left) illustrates all the new types of random features in a Venn diagram.
In this section, we evaluate DERFs experimentally in various machine learning applications. More details about each experiment can be found in Appendix B.
5.1 Variance comparison
We follow the variance comparison setup from [31]: we sample pairs of vectors  $\mathbf{x},\mathbf{y}$  and compute relative variances of the approximation  $\mathrm{Var}\widehat{K}^{(0)}(\mathbf{x},\mathbf{y}) / K^{(0)}(\mathbf{x},\mathbf{y})$  where  $\widehat{K}^{(0)}$  denotes the RF approximation and  $\mathrm{Var}\widehat{K}^{(0)}(\mathbf{x},\mathbf{y})$  is evaluated via (9). We set  $d = 64$  as in [31] and take 6 different regimes for sampling  $\mathbf{x},\mathbf{y}$ : normal where  $\mathbf{x},\mathbf{y}$  are drawn from  $\mathcal{N}(\mathbf{0}_d,\sigma^2\mathbf{I}_d)$ , sphere where  $\mathbf{x},\mathbf{y}$  are drawn uniformly on a sphere  $\sigma S^{d - 1}$ , heterogen where  $\mathbf{x},\mathbf{y}$  are drawn from different distributions  $\mathcal{N}(\mathbf{0}_d,\sigma^2\mathbf{I}_d)$  and  $\mathcal{N}(\sigma \mathbf{1}_d,\sigma^2\mathbf{I}_d)$ . mnist and cifar10 are where  $\mathbf{x},\mathbf{y}$  are random images from MNIST [19] or CIFAR10 [27], resized to  $8\times 8$ , scaled by  $\sigma >0$  and flattened. Finally, mnist/cifar10 is a regime where  $\mathbf{x}$  is drawn as in mnist and  $\mathbf{y}$  is drawn as in cifar10.
We do not report SADERFs since they're a special case of ADERFs (Figure 2). SDERFs outperform or are on par with other methods in all setups – about  $e^5$  times better than GERFs in heterogen, mnist and mnist/cifar10 and about  $e^{10}$  times better in cifar10. Further, ADERFs outperform GERFs by around  $e^3$  times in mnist/cifar10 where  $\mathbf{x}$  and  $\mathbf{y}$  are drawn "asymmetrically".
Figure 4: Comparison of FAVOR# using SDRF with FAVOR++ Performer for regular Conformer-Transducer training with  $m$  random features (TRANS- $m$ ) as well as the Noisy Student Training variant with  $m$  random features (NST- $m$ ) on the LibriSpeech corpus. We report commonly used normalized word error rate (NWER) metric.
5.2 Kernel classification
In this experiment, we compare accuracy of different RF methods in kernel classification on 8 benchmarks from UCI [21], following the setup of [31]. Kernel regression [33, 47] is applied for predicting class probabilities. Training objects are denoted as  $\mathbf{u}^{(1)},\dots,\mathbf{u}^{(L)}\in \mathbb{R}^d$  and their one-hot labels as  $\mathbf{r}^{(1)},\ldots ,\mathbf{r}^{(L)}\in \mathbb{R}^n$ . During testing, the goal is to predict the class of a new object  $\mathbf{u}^*$  as  $\mathrm{argmax}_{1\leq l\leq n}\mathbf{r}^*$  where  $\mathbf{r}^{*} = \sum_{i = 1}^{L}K^{(-0.5)}(\sigma \mathbf{u}^{*},\sigma \mathbf{u}^{(i)})\mathbf{r}^{(i)}$  and  $\sigma >0$  is tuned on the validation set. With  $O(nLM)$  preprocessing, RFs are used to find an unbiased approximation of  $\mathbf{r}^*$  in  $O(nM)$  instead of  $O(nL)$  exact computation. For each benchmark, we range the values of  $M$  from  $2^{4}$  to  $2^{7}$  (Figure 3). We observe that SDERF, which is proposed in this paper, shows the best accuracy across most of the (benchmark,  $M$ ) pairs and also shows the best average performance.
5.3 DERFs for long-sequence Transformers
In this section, we evaluate DERFs for self-attention approximation in Performer-Transformer training setups [16]. We refer to the DERF-based self-attention approximation method as FAVOR#.
5.3.1 Speech modelling
In our first set of experiments, we focus on speech models. We train Performer-encoders and test them on the LibriSpeech corpus [35], commonly used for benchmarking speech models. We considered two Transformers architectures/training setups: (a) Conformer-Transducer [23] trained in a regular way (TRANS) as well as: (b) the Noisy Student Training (NST) variant introduced in [36]. We compare "performed" variants of these architectures, applying FAVOR# with SDERF (since it worked best in the previous setups) as well as FAVOR++ [31].
In the first setting, we see that FAVOR# consistently outperforms FAVOR++ for smaller  $m$  (where reduced variance of the softmax-kernel estimation is more critical) and both achieve similar scores for larger  $m$ . In the NST-experiment, we focused on the smaller  $m$  variant, where FAVOR# again beats FAVOR++. All results are presented in Fig. 4.
5.3.2 Natural language processing
The General Language Understanding Evaluation (GLUE) benchmark [46] consists of 8 different natural language understanding tasks with the sequence length ranging from 32 to 128. We use this to test the performance of different low rank attention methods on NLP tasks. We used the same training parameters as mentioned in [20] (see Appendix B.4.2 for details). We warm start all low-rank
Table 1: GLUE Dev results on base sized models. Number of training examples is reported below each task. MCC score is reported for CoLA, F1 score is reported for MRPC, Spearman correlation is reported for STS-B, and accuracy scores are reported for the other tasks. The best result, second best.
System	MNLI(m) 392k	QQP 363k	QNLI 108k	SST-2 67k	CoLA 8.5k	STS-B 5.7k	MRPC 3.5k	RTE 2.5k
Local Attention [37]	75.68	86.52	87.31	89.71	61.07	83.65	75.47	59.84
Global + Sliding Wndw [4, 53]	81.82	90.35	89.91	92.43	61.81	85.58	87.80	67.15
ELU [26]	82.58	90.05	89.81	92.43	58.63	87.91	87.50	67.15
RELU [16]	82.49	90.71	89.68	92.32	57.57	88.15	87.25	68.95
FAVOR+ [16]	77.69	86.69	89.41	91.80	54.87	83.78	80.73	66.19
FAVOR++ [31]	82.29	90.43	89.73	92.20	58.85	85.90	88.73	67.63
FAVOR#	82.69	90.68	90.01	92.53	59.33	85.48	87.99	69.68
Transformers with a pre-trained BERT-base model checkpoint [20], thus contrasting how well the low rank methods approximate the softmax kernel.
We compared FAVOR++ [31], FAVOR+ [16], ELU [26], ReLU [16] variants of the Performers [16] and sparse attention [37, 4] against the FAVOR# variant and report the results in Table 1. We couldn't use SDERF in this setup because eigendecomposition led to errors on TPUs due to a different implementation compared to the speech modelling experiment. For this reason, we used SADERF which doesn't require any matrix decompositions. On most tasks we find that FAVOR# is the best or second best performing variant showcasing its effectiveness in modelling the softmax kernel for Transformers.
We proposed an extension of generalized exponential random features (GERFs) for the Gaussian and softmax kernels: dense-exponential random features (DERFs). DERFs employ matrix parameters and are more flexible than GERFs. We evaluated DERFs in several applications such as kernel regression and two Transformers training setups, demonstrating downstream performance benefits.
Limitations & broader impact. Optimizing matrix parameters in the most general formulation of DERFs (Theorem 4.1) could lead to further variance reductions. It remains an open question how to find a closed form solution of the optimum which we leave to future work. Our work is primarily theoretical but has broad applications. A prominent application is efficient Transformers which should be used responsibly due to their potential for misuse and significant societal impact, and carbon footprint [48, 5, 10].
V. L. acknowledges support from the Cambridge Trust and DeepMind. V. L. was part-time employed by Google while a PhD student. A.W. acknowledges support from a Turing AI Fellowship under EPSRC grant EP/V025279/1, The Alan Turing Institute, and the Leverhulme Trust via CFI.
[1] Nir Ailon and Edo Liberty. An almost optimal unrestricted fast johnson-lindenstrauss transform. ACM Trans. Algorithms, 9(3):21:1-21:12, 2013.
[2] Haim Avron, Michael Kapralov, Cameron Musco, Christopher Musco, Ameya Velingker, and Amir Zandieh. Random Fourier features for kernel ridge regression: Approximation bounds and statistical guarantees. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pages 253-262. PMLR, 2017.
[3] Haim Avron, Vikas Sindhwani, Jiyan Yang, and Michael W. Mahoney. Quasi-monte carlo feature maps for shift-invariant kernels. J. Mach. Learn. Res., 17:120:1-120:38, 2016.