Title: Dense-Exponential Random Features: Sharp Positive Estimators of the Gaussian Kernel
Venue: NeurIPS 2023
Accepted: yes
Keywords: Gaussian kernel

softmax kernel
Abstract: The problem of efficient approximation of a linear operator induced by the Gaussian or softmax kernel is often addressed using random features (RFs) which yield an unbiased approximation of the operator's result. Such operators emerge in important applications ranging from kernel methods to efficient Transformers. We propose parameterized, positive, non-trigonometric RFs which approximate Gaussian and softmax-kernels. In contrast to traditional RF approximations, parameters of these new methods can be optimized to reduce the variance of the approximation, and the optimum can be expressed in closed form. We show that our methods lead to variance reduction in practice (e^{10}-times smaller variance and beyond) and outperform previous methods in a kernel regression task. Using our proposed mechanism, we also present FAVOR#, a method for self-attention approximation in Transformers. We show that FAVOR# outperforms other random feature methods in speech modelling and natural language processing.

Dense-Exponential Random Features: Sharp Positive Estimators of the Gaussian Kernel

Abstract

The problem of efficient approximation of a linear operator induced by the Gaussian or softmax kernel is often addressed using random features (RFs) which yield an unbiased approximation of the operator's result. Such operators emerge in important applications ranging from kernel methods to efficient Transformers. We propose parameterized, positive, non-trigonometric RFs which approximate Gaussian and softmax-kernels. In contrast to traditional RF approximations, parameters of these new methods can be optimized to reduce the variance of the approximation, and the optimum can be expressed in closed form. We show that our methods lead to variance reduction in practice ( e 10 -times smaller variance and beyond) and outperform previous methods in a kernel regression task. Using our proposed mechanism, we also present FAVOR#, a method for self-attention approximation in Transformers. We show that FAVOR# outperforms other random feature methods in speech modelling and natural language processing.

1 Introduction

Random feature decomposition is an important technique for the linearization of nonlinear kernel functions with theoretical guarantees such as unbiasedness and concentration around the true kernel value. Linearization allows a significant reduction in computations from quadratic to linear complexity in the size of the operator induced by the kernel. The technique emerged under the name of random kitchen sinks (RKS) introduced in [39, 40, 41] and was used in many applications such as kernel SVM [43, 29, 38, 3], dimensionality reduction [18, 1], neural networks [12, 22, 49, 24, 14], function-tofunction regression [34], kernel regression [28, 2], nonparametric adaptive control [6], differentiallyprivate ML algorithms [11], operator-valued kernels [32, 8] and semigroup kernels [51]. An in-depth theoretical analysis of random features was performed by [30, 52, 44, 42].

An exciting recent application of random features is in the area of scalable Transformer networks [16, 13, 17, 26], where the self-attention matrix is approximated as a low-rank matrix when the sequence is long. However, the RKS family of methods relies on the Fourier transform, resulting in sin and cos types of random features, which were shown to be unsuitable for application in Transformers due to negative values in the low-rank matrix. [16] proposed a solution in the form of positive-valued random features relying on the exponential function ( positive random features, PosRFs ), yielding

a method they called Fast Attention Via Orthogonal positive Random features (FAVOR+) for selfattention approximation. This solution was improved by [31] by means of a careful choice of the linear combination parameters under the exponent, and the so-called homogeneity heuristic , which allows a choice of one set of parameters for all approximated values. The resulting random features were called generalized exponential random features (GERFs) , and the corresponding self-attention approximation method was termed FAVOR++ .

Contributions: In this paper, we make a leap forward in the design of positive-valued random features by proposing dense exponential random features (DERFs) which contain both PosRFs and GERFs as special cases. Instead of scalar parameters as in GERFs, DERFs rely on matrix parameters and dense quadratic forms inside the exponent. We show how to select parameters of the new random features efficiently without harming the overall subquadratic complexity.

More technically, our contributions are as follows:

We show that the homogeneity heuristic of [31] may in fact be viewed not as a heuristic, but a closed-form optimum of the shifted log-variance objective .

We introduce DERFs and three special instantiations: asymmetric DERFs ( ADERFs ), symmetric DERFs ( SDERFs ), and simplified ADERFs ( SADERFs ). All these instantiations contain GERFs as a special case (Figure 1, left). For each instantiation we prove that a closed-form optimum of the shifted log-variance objective can be found efficiently.

We show that our new variants result in lower variance than GERFs and other previous methods in practice (e.g. up to e 10 times variance improvement as in Figure 1, right). Further, we show that DERFs outperform other random features in kernel regression and Transformer setups (speech modelling and natural language processing). We refer to the DERF-based self-attention approximation method as FAVOR# .

2 Prerequisites

2.1 Scaled softmax kernel and random features

By the scaled softmax kernel K ( α ) : R d × R d → (0 , + ∞ ) , where α ∈ R , we denote a mapping defined as K ( α ) ( x , y ) = exp( α ∥ x ∥ 2 + x ⊤ y + α ∥ y ∥ 2 ) for all x , y ∈ R d where ∥ · ∥ is an L 2 -norm. Two important special cases of the scaled softmax kernel are 1) the Gaussian kernel K ( -1 / 2) ( x , y ) = exp( -∥ x -y ∥ 2 / 2) and 2) the softmax kernel K (0) ( x , y ) = exp( x ⊤ y ) . For two sets of vectors X = { x ( i ) ∈ R d } L i =1 and Y = { y ( j ) ∈ R d } L j =1 , by K ( X , Y ) ∈ R L × L we denote a matrix where K ( α ) ( X , Y ) i,j = K ( α ) ( x ( i ) , y ( j ) ) for all 1 ≤ i, j ≤ L .

In this paper, we will be interested in the problem of computing K ( α ) ( X , Y ) C where X , Y and a matrix C ∈ R L × n are provided as an input. A naive solution requires O ( L 2 ( d + n )) computations for constructing K ( α ) ( X , Y ) ( O ( L 2 d ) ) and computing the matrix multiplication K ( α ) ( X , Y ) × C ( O ( L 2 n ) ). Instead, we will use an efficient Monte-Carlo approximation of K ( α ) ( X , Y ) × C using the following notion of random features (RFs) for the scaled softmax kernel :

cifar10

mnist/cifar10

DERF

20

GERF

20

var)

SDERF

log(relative var)

log(relative

15

15

SDERF

GERF

SADERF

ADERF

10

10

5

5

0

0

101

100

101

100

Definition 2.1. By random features for the scaled softmax kernel K ( α ) , α ∈ R , we denote a triple T = ⟨ ν, f (1) , f (2) ⟩ where ν is a probability distribution over random objects ω ∈ Ω and f ( i ) : Ω × R d → R , i ∈ { 1 , 2 } , are such mappings that, for all x , y ∈ R d ,

The decomposition of type (1) can be used for an efficient unbiased approximation of K ( α ) ( X , Y ) C . Let ω (1) , . . . , ω ( M ) ∈ Ω be i.i.d. samples from ν . Define matrices P , S ∈ R L × M where for all 1 ≤ i, j ≤ L ,

where P i, : , S j, : ∈ R d are column vectors corresponding to the rows of P , S . Then according to (1), ̂ K = PS ⊤ is an unbiased Monte Carlo (MC) approximation of K ( α ) ( X , Y ) on M samples. The variance Var ̂ K i,j = M -1 Var ν f (1) ( ω , x ( i ) ) f (2) ( ω , y ( j ) ) of this approximation is inversely proportional to M , hence M is a tradeoff parameter between computations and precision. Now, ̂ K C is an unbiased approximation of K ( α ) ( X , Y ) C but ̂ K = PS ⊤ is a rankM matrix, hence computing ̂ K C has O ( LMn ) complexity. Assuming that sampling each ω ( m ) and computing f ( · ) ( · , · ) are O ( d ) operations, which is usually the case, precomputing P and S takes O ( LMd ) computations, resulting in a total O ( LM ( d + n )) computational complexity. By choosing M ≪ L , we obtain a significant reduction in computations compared to the exact variant: O ( LM ( d + n )) ≪ O ( L 2 ( d + n )) .

Operations of type K ( α ) ( X , Y ) C , especially for the Gaussian kernel α = -1 / 2 , emerge in kernel SVM [40], kernel regression [33, 47] and in physics in the form of the Gauss transform [50]. Another important application is in the area of efficient Transformers and is discussed in the next section [16].

2.2 Random features for efficient Transformers

RFs found a prominent application in the area of efficient long-sequence Transformers [16]. Transformers rely on a self-attention block for propagating information between elements of the sequence. If the sequence length is L and input matrices are denoted as Q , K , V ∈ R L × d ( queries , keys and values ), then self-attention outputs the following matrix:

where 1 L ∈ R L is a vector of all ones, diag( · ) returns a diagonal ( L × L ) -sized matrix with the argument on the diagonal, X = { x ( i ) = d -1 / 4 Q i, : ∈ R d } , and Y = { y ( j ) = d -1 / 4 K j, : ∈ R d } . Hence, substitution of ̂ K instead of K (0) ( X , Y ) in (3) reduces computational complexity from O ( L 2 d ) to O ( LMd ) ( n = d +1 ). diag( K (0) ( X , Y ) 1 L ) -1 K (0) ( X , Y ) is the result of a softmax operation performed on rows of d -1 / 2 QK ⊤ .

2.3 Existing random features for the softmax kernel

Representation (1) is not unique and different RFs can be proposed for a single K ( α ) . Note that if ⟨ ν, f (1) , f (2) ⟩ are RFs for K (0) , then ⟨ ν, ̂ f (1) , ̂ f (2) ⟩ are RFs for K ( α ) for α ∈ R where ̂ f ( k ) ( ω , x ) = exp( α ∥ x ∥ 2 ) f ( k ) ( ω , x ) . Hereafter we focus on the softmax kernel K (0) without loss of generality.

[15] proposed to use trigonometric random features (TrigRFs) from [40] for efficient Transformers:

where ω = ( θ, ˜ ω ) , Unif( · ) denotes a uniform distribution on the argument set, N ( 0 d , I d ) is a multivariate Gaussian distribution with mean 0 d (vector of d zeros) and covariance matrix I d (identity matrix of size d × d ).

The next iteration of efficient attention approximators [16] observed a problem with TrigRFs (4-5). The attention matrix diag( K (0) ( X , Y ) 1 L ) -1 K (0) ( X , Y ) from (3) is right stochastic meaning that its

entries are nonnegative and each row sums to 1 due to the normalizing term diag( K (0) ( X , Y ) 1 L ) -1 . However, since f ( · ) trig can be arbitrary real numbers, P , S (2) and, therefore, ̂ K can take negative values. Hence, diag( ̂ K 1 L ) -1 ̂ K is not right stochastic in general and entries of ̂ K 1 L can take very small and/or negative values resulting in unstable behaviour when inverting diag( ̂ K 1 L ) -1 . [16] therefore proposed a new type of positive random features (PosRFs) which have the form:

It is clear that such f ( · ) pos only take strictly positive values resulting in the right stochastic diag( ̂ K 1 L ) -1 ̂ K and a stable Transformer training procedure.

[31] extended PosRFs, proposing generalized exponential random features (GERFs) 1 for K (0) :

where A,B,C,D are real numbers 2 satisfying: 1 -8 A > 0 , B = 1 -4 A , C = -0 . 5 , D = (1 -4 A ) d/ 4 . [31] express B,C,D through A and find a closed-form equation for the variance of (1):

The minimum variance corresponds to the minimum L ( A, x , y ) since K (0) ( x , y ) 2 does not depend on A . Since L ( A, x , y ) is defined for a single pair of x , y and not for sets X , Y , [31] propose a homogeneity heuristic when they replace ∥ x + y ∥ 2 , ∥ x ∥ 2 , ∥ y ∥ 2 in (7) with averages over X , Y : L -2 ∑ i,j ∥ x ( i ) + y ( j ) ∥ 2 , L -1 ∑ i ∥ x ( i ) ∥ 2 and L -1 ∑ j ∥ y ( j ) ∥ 2 respectively. This heuristic is based on the assumption that { x ( i ) } and { y ( j ) } are homogeneous and their statistics are tightly concentrated around the mean. After this, the minimum of (7) with respect to A can be found in closed form.

3 Dense-exponential random features (DERFs)

We prove that the homogeneity heuristic (Section 2.3) corresponds to a certain minimization problem. Then, we present DERFs which generalize GERFs and provide a tighter solution of that problem.

3.1 The objective minimized by GERFs

Our first contribution is showing that the homogeneity heuristic adopted in GERFs is actually an analytic solution of a certain optimization problem. Define

where T = ⟨ ν, f (1) , f (2) ⟩ are RFs for the kernel K (0) and θ are their parameters appearing implicitly in ν, f (1) , f (2) . (8) is a mean log-variance shifted by K (0) ( x ( i ) , y ( j ) ) 2 . The best possible value of (8) is log K (0) ( x ( i ) , y ( j ) ) which corresponds to all variances Var f (1) ( ω , x ( i ) ) f (2) ( ω , y ( j ) ) being zero, meaning that RFs provide exact kernel estimation. Hence, minimization of (8) leads to more precise estimators on X , Y . We call the loss function L ( θ ; X , Y , T ) the shifted log-variance objective. Since log is concave loss L is conceptually similar to relative standard deviation: the smaller K (0) ( x ( i ) , y ( j ) ) , the higher L is for the same amount of variance Var f (1) ( ω , x ( i ) ) f (2) ( ω , y ( j ) ) .

If T GE = ⟨ ν GE , f (1) GE , f (2) GE ⟩ are taken in (8), then θ GE = { A,B,C,D } and L ( θ GE ; X , Y , T GE ) = L -2 ∑ i,j L GE ( A ; x ( i ) , y ( j ) ) . Using (7), we get: L ( θ GE ; X , Y , T GE ) =

That is, L ( A ; X , Y ) coincides with (7) when ∥ x + y ∥ 2 , ∥ x ∥ 2 , ∥ y ∥ 2 are replaced by their average statistics computed on X , Y . Hence, the homogeneity heuristic (Section 2.3) is nothing but minimization of (8). While in general it's unclear how to find a closed-form optimum of Var ̂ K or Var( ̂ K C ) , the global minimum of (8) is feasible and can be computed in O (1) time. Further, in the case of GERF, the values minimizing (8) lead to very good results in large-scale applications of efficient Transformers as shown by [31] without knowing about (8) objective. In the next section we present extensions of GERFs which aim to minimize (8) in closed form.

4 Towards DERFs

Dense-exponential random features (DERFs) are an extension of GERFs where scalars A,B,C are replaced with dense matrices. DERFs may be viewed as a generalization that contain the previously introduced classes as special cases. We define DERFs as follows: Ω DE = R d , ν DE = N (0 , 1) d and for k ∈ { 1 , 2 } :

where B ( k ) , C ( k ) ∈ R d × d , D ∈ R , A ∈ S d (a set of d × d real symmetric matrices). Clearly, GERFs with parameters A,B,C,D can be expressed via DERFs with parameters A = A I d , B (1) = B (2) = B I d , C (1) = C (2) = C I d , D is unchanged. Our first theoretical result is giving the conditions when T DE = ⟨ ν DE , f (1) DE , f (2) DE ⟩ are valid RFs:

Theorem 4.1. Let the following conditions hold: 8 A ≺ I d , ( B (1) ) ⊤ ( I d -4 A ) -1 B (2) = I d , C ( k ) = -1 2 ( B ( k ) ) ⊤ ( I d -4 A ) -1 B ( k ) , D = det( I d -4 A ) 1 / 4 where k ∈ { 1 , 2 } . Then T DE are RFs for K (0) and for all x , y ∈ R d : Var ν DE f (1) DE ( ω , x ) f (2) DE ( ω , y ) =

Our ultimate goal is to find optimal parameters A , B ( k ) , C ( k ) and D minimizing the variance of the low-rank approximation of K (0) ( X , Y ) where sets X , Y are provided. Our first observation is that we can assume that A ∈ D d (a set of d × d real diagonal matrices). Indeed, any symmetric A can be expressed as Q ˜ AQ ⊤ where Q ∈ O d (a set of orthogonal matrices { Z ∈ R d × d | Z ⊤ Z = I d } ) and ˜ A ∈ D d . Let ω ∼ N ( 0 d , I d ) . Then, for any x ∈ R d , k ∈ { 1 , 2 } ,

where ˜ B ( k ) = Q ⊤ B ( k ) , ˜ ω = Q ⊤ ω ∼ N ( 0 d , I d ) since the distribution ω ∼ N ( 0 d , I d ) is isometric , i.e. rotation-invariant and ˜ f ( k ) DE are DERFs with parameters ˜ A , ˜ B ( k ) , C ( k ) , D . We conclude that with any A , f ( k ) DE ( ω , x ) can be expressed as DERFs ˜ f ( k ) DE with ˜ A ∈ D d . Hence, hereafter we only consider A ∈ D d without loss of generality.

Since B ( k ) , C ( k ) are dense matrices in general, evaluation of f ( k ) DE ( ω , x ) takes O ( d 2 ) time which is bigger than the O ( d ) complexity for TrigRFs, PosRFs and GERFs. However, P and S matrices (2) can be still computed in a time subquadratic in L . For that, precompute ( B ( k ) ) ⊤ ω ( m ) , C (1) x ( i ) , C (2) y ( j ) for all k ∈ { 1 , 2 } , 1 ≤ m ≤ M , 1 ≤ i, j ≤ L in O (( M + L ) d 2 ) time. Then, computing f (1) DE ( ω ( m ) , x ( i ) ) , f (2) DE ( ω ( m ) , y ( j ) ) for all 1 ≤ i, j ≤ L , 1 ≤ m ≤ M takes O ( LMd ) operations. The complexity of constructing (2) then is O ( L ( Md + d 2 ) + Md 2 ) which is still subquadratic in L .

Our goal is to minimize L ( θ DE ; X , Y , T DE ) for θ DE = { A , B (1) , B (2) , C (1) , C (2) , D } . However, we find that even for a single pair of x , y it's unclear how to minimize the variance (9) in closed form. Hence, below we consider special cases where an analytic solution is feasible.

normal

sphere

heterogen

20

TrigRF

0.5

25

log(relative var)

15

PosRF

ADERF

GERF

log(relative var)

0.0

log(relative var)

20

15

10

SDERF

0.5

10

-1.0

5

1.5

0

-2.0

101

100

101

100

101

100

mnist

cifar10

mnist/cifar10

20

20

20

log(relative var)

15

log(relative var)

15

log(relative var)

15

10

10

10

5

0

0

0

101

100

10-1

100

101

100

4.1 Asymmetric dense-exponential random features

̸

Define RFs T ADE = ⟨ ν ADE , f (1) ADE , f (2) ADE ⟩ in the same way as T DE with the only difference that A = A I d where A ∈ R . We refer to these RFs as asymmetric dense-exponential RFs (ADERFs) since f (1) ADE = f (2) ADE in general. The only additional restriction of ADERFs compared to DERFs is that all diagonal entries of A ∈ D d are the same. The parameters of T ADE are θ ADE = { A, B (1) , B (2) , C (1) , C (2) , D } . By Θ ADE denote a set of all possible θ ADE 's resulting in correct RFs for the kernel K (0) , i.e. satisfying Theorem 4.1 with A = A I d . The following result gives an analytic formula for a minimum of L ( θ ADE ; X , Y , T ADE ) . In the theorem, we use notions of SVD and eigendecomposition of a symmetric matrix [45] (all proofs are in the Appendix).

Theorem 4.2. Let X = { x ( i ) ∈ R d } L i =1 , Y = { y ( j ) ∈ R d } L j =1 . Let M (1) = 1 L ∑ L i =1 x ( i ) ( x ( i ) ) ⊤ , M (2) = 1 L ∑ L j =1 y ( j ) ( y ( j ) ) ⊤ . Suppose that M (1) , M (2) ∈ S d are nonsingular (implying that L ≥ d ). Define µ (3) = d -1 L -2 ( ∑ L i =1 x ( i ) ) ⊤ ( ∑ L j =1 y ( j ) ) ∈ R . For k ∈ { 1 , 2 } , let M ( k ) = Q ( k ) Λ ( k ) ( Q ( k ) ) ⊤ be eigendecomposition of a symmetric M ( k ) where Q ( k ) ∈ O d . Λ ( k ) ∈ D d has strictly positive diagonal values since M ( k ) ⪰ 0 by definition and M ( k ) is nonsingular. Let UΣV ⊤ be SVD decomposition of ( Λ (1) ) 1 2 ( Q (1) ) ⊤ Q (2) ( Λ (2) ) 1 2 where U , V ∈ O d , Σ ∈ D d has nonnegative diagonal entries.

One of the solutions θ ∗ ADE = { A, B (1) , B (2) , C (1) , C (2) , D } of min θ ADE ∈ Θ ADE L ( θ ADE ; X , Y , T ADE ) is as follows. Set ϕ = 2 d -1 ∑ d l =1 Σ l,l +2 µ (3) and, for k ∈ { 1 , 2 } ,

Further, we have: L ( θ ∗ ADE ; X , Y , T ADE ) =

Theorem 4.2 implies an algorithm for finding θ ∗ ADE efficiently. Namely, compute M ( k ) , k ∈ { 1 , 2 } ( O ( Ld 2 ) time) and µ (3) ( O ( Ld ) time). Then, perform matrix decompositions to obtain Q ( k ) , Λ ( k ) , k ∈ { 1 , 2 } , and U , Σ , V in O ( d 3 ) time. After that, A, B (1) , B (2) , C (1) , C (2) , D can be all evaluated in O ( d 3 ) time using formulae from Theorem 4.2. The total time complexity of the approximation scheme is therefore O ( L ( Md + d 2 ) + Md 2 + d 3 ) which is subquadratic in L as required.

4.2 Symmetric dense-exponential random features

Define T SDE = ⟨ ν SDE , f (1) SDE , f (2) SDE ⟩ in the same way as T DE with the only difference that B (1) = B (2) = B . From the conditions in Theorem 4.1 it follows immediately that also C (1) = C (2) = C . Hence, f (1) SDE = f (2) SDE and we refer to these RFs as symmetric dense-exponential RFs (SDERFs) . The parameters of T SDE are θ SDE = { A , B , C , D } . By Θ SDE denote a set of all possible θ SDE 's resulting in correct RFs for the kernel K (0) , i.e. satisfying conditions from Theorem 4.1 with B ( k ) = B , C ( k ) = C , k ∈ { 1 , 2 } . The following theorem gives an analytic solution for a global minimum of L ( θ SDE ; X , Y , T SDE ) .

Theorem 4.3. Let X = { x ( i ) ∈ R d } L i =1 , Y = { y ( j ) ∈ R d } L j =1 and let M (1) , M (2) be defined as in Theorem 4.2 (but without the restriction of nonsingularity) and define µ (4) = 1 L ∑ L i =1 x ( i ) ∈ R d , µ (5) = 1 L ∑ L j =1 y ( j ) ∈ R d . Further, let Q (3) Λ (3) ( Q (3) ) ⊤ be eigendecomposition of a symmetric positive semidefinite matrix M (1) + µ (4) ( µ (5) ) ⊤ + µ (5) ( µ (4) ) ⊤ + M (2) where Q (3) ∈ O d and Λ (3) ∈ D d with nonnegative diagonal entries. Let the entries on the diagonal of Λ (3) be sorted in the non-ascending order.

Again, Theorem 4.3 implies an algorithm for finding θ ∗ SDE in a time subquadratic in L . That is, we can compute M (1) , M (2) , µ (3) , µ (4) , µ (5) in O ( Ld 2 ) total time. Then, perform an eigendecomposition to obtain Q (3) , Λ (3) in O ( d 3 ) time. After that, A , B , C , D can be computed in O ( d 3 ) time using formulae from Theorem 4.3. The total time complexity of the approximation scheme is the same as for ADERFs: O ( L ( Md + d 2 ) + Md 2 + d 3 ) or O ( L ( Md + d 2 ) + Md 2 ) if we assume that L ≥ d .

4.3 Simplified ADERFs

While having a compact and closed-form expression, both ADERFs and SDERFs rely on eigendecomposition and SVD decompositions: operations for which implementation has not yet matured in popular deep learning libraries with GPU and TPU support. For this reason, we propose simplified ADERFs (SADERFs) T SADE = ⟨ ν SADE , f (1) SADE , f (2) SADE ⟩ which extend GERFs but require only basic unary operations. SADERFs are defined via GERFs as follows: Ω SADE = R d , ν SADE = N (0 , 1) d , f (1) SADE ( ω , x ) = f (1) GE ( ω , Ψx ) , f (2) SADE ( ω , y ) = f (2) GE ( ω , Ψ -1 y ) where Ψ ∈ D d is a diagonal matrix with nonzero diagonal entries. First of all, T SADE are valid random features for the softmax kernel K (0) since E ν SADE [ f (1) SADE ( ω , x ) f (2) SADE ( ω , y )] = E ν GE [ f (1) GE ( ω , Ψx ) f (2) GE ( ω , Ψ -1 y )] = K (0) ( Ψx , Ψ -1 y ) = K (0) ( x , y ) , where we use K (0) ( x , y ) = exp( x ⊤ y ) by the definition.

We find Ψ by optimizing the objective (8) for T SADE , the form of which is easily deduced from L ( θ GE ; X , Y , T GE ) :

where we move a term not depending on Ψ to the left-hand side. Since 1 -8 A > 0 , we conclude that minimizing (12) is equivalent to minimizing ∑ i,j ∥ Ψx ( i ) + Ψ -1 y ( j ) ∥ 2 =

4

5

6

7

log2(

M

)

12

14

16

18

Accuracy, %

abalone,

L

=4176

TrigRF

PosRF

GERF

ADERF

SDERF

4

5

6

7

log2(

M

)

70

80

90

Accuracy, %

banknote,

L

=1370

4

5

6

7

log2(

M

)

67

68

69

70

71

Accuracy, %

car,

L

=1727

4

5

6

7

log2(

M

)

30

35

40

Accuracy, %

yeast,

L

=1483

4

5

6

7

log2(

M

)

42

44

46

48

Accuracy, %

cmc,

L

=1472

4

5

6

7

log2(

M

)

30

40

50

60

70

80

Accuracy, %

nursery,

L

=12960

4

5

6

7

log2(

M

)

20

40

60

80

100

Accuracy, %

wifi,

L

=1999

4

5

6

7

log2(

M

)

17

18

19

20

Accuracy, %

chess,

L

=28055

4

5

6

7

log2(

M

)

35

40

45

50

55

60

Accuracy, %

average

Optimizing (13) reduces to independent optimization problems with respect to Ψ l,l , 1 ≤ l ≤ d . Each problem is convex and the solution is found trivially by setting the derivative to zero: for all 1 ≤ l ≤ d : Ψ ∗ l,l = ( ∑ j ( y ( j ) l ) 2 / ∑ i ( x ( i ) l ) 2 ) 1 / 4 . The solution can be computed in O ( dL ) time, after which the parameters of f (1) GE , f (2) GE can be found efficiently as described in Section 2.3.

It is easy to see that T SADE are a special case of ADERFs (Section 4.1) which explains their name. Furthermore, the case Ψ = I d reduces T SADE to T GE , hence the latter is a special case of the former. Figure 1 (left) illustrates all the new types of random features in a Venn diagram.

5 Experiments

In this section, we evaluate DERFs experimentally in various machine learning applications. More details about each experiment can be found in Appendix B.

5.1 Variance comparison

We follow the variance comparison setup from [31]: we sample pairs of vectors x , y and compute relative variances of the approximation Var ̂ K (0) ( x , y ) /K (0) ( x , y ) where ̂ K (0) denotes the RF approximation and Var ̂ K (0) ( x , y ) is evaluated via (9). We set d = 64 as in [31] and take 6 different regimes for sampling x , y : normal where x , y are drawn from N ( 0 d , σ 2 I d ) , sphere where x , y are drawn uniformly on a sphere σ S d -1 , heterogen where x , y are drawn from different distributions N ( 0 d , σ 2 I d ) and N ( σ 1 d , σ 2 I d ) . mnist and cifar10 are where x , y are random images from MNIST [19] or CIFAR10 [27], resized to 8 × 8 , scaled by σ > 0 and flattened. Finally, mnist/cifar10 is a regime where x is drawn as in mnist and y is drawn as in cifar10 .

We do not report SADERFs since they're a special case of ADERFs (Figure 2). SDERFs outperform or are on par with other methods in all setups - about e 5 times better than GERFs in heterogen , mnist and mnist/cifar10 and about e 10 times better in cifar10 . Further, ADERFs outperform GERFs by around e 3 times in mnist/cifar10 where x and y are drawn 'asymmetrically'.

TRANS-8

TRANS-32

TRANS-128

FAVOR#

FAVOR#

FAVOR#

FAVOR#

NST-8

0'0

0.036

0.038

000

0.036

0.032

0.034

0.03

0.032

0.03

0.028

0.028

0.026

0.026

0.024

0.024

0.022

0.022

5k

10k

15k20k25k30k

35k

40k

45k

50k

5k

10k15k20k25k30k35k40k45k50k55k60k65k

0.05

0.027

0.046

0.026

0.025

0.042

0.024

0.038

0.023

0.034

0.022

0.03

0.021

0.026

0.02

0.019

0.022

5k

10k

15k

20k

25k

30k

35k

15k

20k

25k

30k

35k

40k

45k

5.2 Kernel classification

In this experiment, we compare accuracy of different RF methods in kernel classification on 8 benchmarks from UCI [21], following the setup of [31]. Kernel regression [33, 47] is applied for predicting class probabilities. Training objects are denoted as u (1) , . . . , u ( L ) ∈ R d and their one-hot labels as r (1) , . . . , r ( L ) ∈ R n . During testing, the goal is to predict the class of a new object u ∗ as argmax 1 ≤ l ≤ n r ∗ where r ∗ = ∑ L i =1 K ( -0 . 5) ( σ u ∗ , σ u ( i ) ) r ( i ) and σ > 0 is tuned on the validation set. With O ( nLM ) preprocessing, RFs are used to find an unbiased approximation of r ∗ in O ( nM ) instead of O ( nL ) exact computation. For each benchmark, we range the values of M from 2 4 to 2 7 (Figure 3). We observe that SDERF, which is proposed in this paper, shows the best accuracy across most of the (benchmark, M ) pairs and also shows the best average performance.

5.3 DERFs for long-sequence Transformers

In this section, we evaluate DERFs for self-attention approximation in Performer-Transformer training setups [16]. We refer to the DERF-based self-attention approximation method as FAVOR# .

5.3.1 Speech modelling

In our first set of experiments, we focus on speech models. We train Performer-encoders and test them on the LibriSpeech corpus [35], commonly used for benchmarking speech models. We considered two Transformers architectures/training setups: (a) Conformer-Transducer [23] trained in a regular way ( TRANS ) as well as: (b) the Noisy Student Training ( NST ) variant introduced in [36]. We compare 'performized' variants of these architectures, applying FAVOR# with SDERF (since it worked best in the previous setups) as well as FA VOR++ [31].

In the first setting, we see that FA VOR# consistently outperforms FA VOR++ for smaller m (where reduced variance of the softmax-kernel estimation is more critical) and both achieve similar scores for larger m . In the NST-experiment, we focused on the smaller m variant, where FAVOR# again beats FAVOR++. All results are presented in Fig. 4.

5.3.2 Natural language processing

The General Language Understanding Evaluation (GLUE) benchmark [46] consists of 8 different natural language understanding tasks with the sequence length ranging from 32 to 128. We use this to test the performance of different low rank attention methods on NLP tasks. We used the same training parameters as mentioned in [20] (see Appendix B.4.2 for details). We warm start all low-rank

Transformers with a pre-trained BERT-base model checkpoint [20], thus contrasting how well the low rank methods approximate the softmax kernel.

We compared FAVOR++ [31], FAVOR+ [16], ELU [26], ReLU [16] variants of the Performers [16] and sparse attention [37, 4] against the FA VOR# variant and report the results in Table 1. We couldn't use SDERF in this setup because eigendecomposition led to errors on TPUs due to a different implementation compared to the speech modelling experiment. For this reason, we used SADERF which doesn't require any matrix decompositions. On most tasks we find that FAVOR# is the best or second best performing variant showcasing its effectiveness in modelling the softmax kernel for Transformers.

6 Conclusion

We proposed an extension of generalized exponential random features (GERFs) for the Gaussian and softmax kernels: dense-exponential random features (DERFs). DERFs employ matrix parameters and are more flexible than GERFs. We evaluated DERFs in several applications such as kernel regression and two Transformers training setups, demonstrating downstream performance benefits.

Limitations & broader impact. Optimizing matrix parameters in the most general formulation of DERFs (Theorem 4.1) could lead to further variance reductions. It remains an open question how to find a closed form solution of the optimum which we leave to future work. Our work is primarily theoretical but has broad applications. A prominent application is efficient Transformers which should be used responsibly due to their potential for misuse and significant societal impact, and carbon footprint [48, 5, 10].

References

Nir Ailon and Edo Liberty. An almost optimal unrestricted fast johnson-lindenstrauss transform. ACM Trans. Algorithms , 9(3):21:1-21:12, 2013.

Haim Avron, Michael Kapralov, Cameron Musco, Christopher Musco, Ameya Velingker, and Amir Zandieh. Random Fourier features for kernel ridge regression: Approximation bounds and statistical guarantees. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017 , volume 70 of Proceedings of Machine Learning Research , pages 253-262. PMLR, 2017.

Haim Avron, Vikas Sindhwani, Jiyan Yang, and Michael W. Mahoney. Quasi-monte carlo feature maps for shift-invariant kernels. J. Mach. Learn. Res. , 17:120:1-120:38, 2016.

Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150 , 2020.

Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? . In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency , FAccT '21, page 610-623, New York, NY, USA, 2021. Association for Computing Machinery.

Nicholas M. Boffi, Stephen Tu, and Jean-Jacques E. Slotine. Nonparametric adaptive control and prediction: Theory and randomized algorithms. In 60th IEEE Conference on Decision and Control, CDC 2021, Austin, TX, USA, December 14-17, 2021 , pages 2935-2942. IEEE, 2021.

James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018.

Romain Brault, Markus Heinonen, and Florence Buc. Random fourier features for operatorvalued kernels. In Robert J. Durrant and Kee-Eung Kim, editors, Proceedings of The 8th Asian Conference on Machine Learning , volume 63 of Proceedings of Machine Learning Research , pages 110-125, The University of Waikato, Hamilton, New Zealand, 16-18 Nov 2016. PMLR.

R.W. Brockett. Dynamical systems that sort lists, diagonalize matrices, and solve linear programming problems. Linear Algebra and its Applications , 146:79-91, 1991.

Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom B. Brown, Dawn Song, Úlfar Erlingsson, Alina Oprea, and Colin Raffel. Extracting training data from large language models. CoRR , abs/2012.07805, 2020.

Kamalika Chaudhuri, Claire Monteleoni, and Anand D. Sarwate. Differentially private empirical risk minimization. J. Mach. Learn. Res. , 12:1069-1109, 2011.

Youngmin Cho and Lawrence K. Saul. Kernel methods for deep learning. In Yoshua Bengio, Dale Schuurmans, John D. Lafferty, Christopher K. I. Williams, and Aron Culotta, editors, Advances in Neural Information Processing Systems 22: 23rd Annual Conference on Neural Information Processing Systems 2009. Proceedings of a meeting held 7-10 December 2009, Vancouver, British Columbia, Canada , pages 342-350. Curran Associates, Inc., 2009.

Krzysztof Choromanski, Haoxian Chen, Han Lin, Yuanzhe Ma, Arijit Sehanobish, Deepali Jain, Michael S. Ryoo, Jake Varley, Andy Zeng, Valerii Likhosherstov, Dmitry Kalashnikov, Vikas Sindhwani, and Adrian Weller. Hybrid random features. In International Conference on Learning Representations (ICLR) , 2022.

Krzysztof Choromanski, Carlton Downey, and Byron Boots. Initialization matters: Orthogonal predictive state recurrent neural networks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings . OpenReview.net, 2018.

Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Jared Davis, Tamás Sarlós, David Belanger, Lucy J. Colwell, and Adrian Weller. Masked language modeling for proteins via linearly scalable long-context transformers. CoRR , abs/2006.03555, 2020.

Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, David Benjamin Belanger, Lucy J Colwell, and Adrian Weller. Rethinking attention with performers. In International Conference on Learning Representations , 2021.

Sankalan Pal Chowdhury, Adamos Solomou, Avinava Dubey, and Mrinmaya Sachan. On learning the transformer kernel. Transactions of Machine Learning Research , 2022.

Sanjoy Dasgupta and Anupam Gupta. An elementary proof of a theorem of johnson and lindenstrauss. Random Struct. Algorithms , 22(1):60-65, 2003.

Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine , 29(6):141-142, 2012.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018.

Dheeru Dua and Casey Graff. UCI machine learning repository, 2017.

Lukas Gonon. Random feature neural networks learn Black-Scholes type PDEs without curse of dimensionality. CoRR , abs/2106.08900, 2021.

Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, and Ruoming Pang. Conformer: Convolutionaugmented transformer for speech recognition. CoRR , abs/2005.08100, 2020.

Insu Han, Haim Avron, Neta Shoham, Chaewon Kim, and Jinwoo Shin. Random features for the neural tangent kernel. CoRR , abs/2104.01351, 2021.

Charles R. Harris, K. Jarrod Millman, Stéfan J. van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fernández del Río, Mark Wiebe, Pearu Peterson, Pierre Gérard-Marchant, Kevin Sheppard, Tyler Reddy, Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis E. Oliphant. Array programming with NumPy. Nature , 585(7825):357-362, September 2020.

Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are RNNs: Fast autoregressive transformers with linear attention. In Proceedings of the 37th International Conference on Machine Learning , 2020.

Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-10 (canadian institute for advanced research).

Valero Laparra, Diego Marcos Gonzalez, Devis Tuia, and Gustau Camps-Valls. Large-scale random features for kernel regression. In 2015 IEEE International Geoscience and Remote Sensing Symposium (IGARSS) , pages 17-20, 2015.

Quoc V. Le, Tamás Sarlós, and Alexander J. Smola. Fastfood - computing hilbert space expansions in loglinear time. In Proceedings of the 30th International Conference on Machine Learning, ICML 2013, Atlanta, GA, USA, 16-21 June 2013 , volume 28 of JMLR Workshop and Conference Proceedings , pages 244-252. JMLR.org, 2013.

Zhu Li, Jean-Francois Ton, Dino Oglic, and Dino Sejdinovic. Towards a unified analysis of random Fourier features. J. Mach. Learn. Res. , 22:108:1-108:51, 2021.

Valerii Likhosherstov, Krzysztof Choromanski, Avinava Dubey, Frederick Liu, Tamas Sarlos, and Adrian Weller. Chefs' random tables: Non-trigonometric random features. In Advances in Neural Information Processing Systems . Curran Associates, Inc., 2022.

Ha Quang Minh. Operator-valued Bochner theorem, Fourier feature maps for operator-valued kernels, and vector-valued learning. CoRR , abs/1608.05639, 2016.

E. A. Nadaraya. On estimating regression. Theory of Probability & Its Applications , 9(1):141142, 1964.

Junier B. Oliva, Willie Neiswanger, Barnabás Póczos, Eric P. Xing, Hy Trac, Shirley Ho, and Jeff G. Schneider. Fast function to function regression. In Guy Lebanon and S. V. N. Vishwanathan, editors, Proceedings of the Eighteenth International Conference on Artificial Intelligence and Statistics, AISTATS 2015, San Diego, California, USA, May 9-12, 2015 , volume 38 of JMLR Workshop and Conference Proceedings . JMLR.org, 2015.

Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: An ASR corpus based on public domain audio books. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2015, South Brisbane, Queensland, Australia, April 19-24, 2015 , pages 5206-5210. IEEE, 2015.

Daniel S. Park, Yu Zhang, Ye Jia, Wei Han, Chung-Cheng Chiu, Bo Li, Yonghui Wu, and Quoc V. Le. Improved noisy student training for automatic speech recognition. In Helen Meng, Bo Xu, and Thomas Fang Zheng, editors, Interspeech 2020, 21st Annual Conference of the International Speech Communication Association, Virtual Event, Shanghai, China, 25-29 October 2020 , pages 2817-2821. ISCA, 2020.

Niki J. Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In International Conference on Machine Learning (ICML) , 2018.

Jeffrey Pennington, Felix X. Yu, and Sanjiv Kumar. Spherical random features for polynomial kernels. In Corinna Cortes, Neil D. Lawrence, Daniel D. Lee, Masashi Sugiyama, and Roman Garnett, editors, Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada , pages 1846-1854, 2015.

A. Rahimi and B. Recht. Uniform approximation of functions with random bases. In 2008 46th Annual Allerton Conference on Communication, Control, and Computing , Los Alamitos, CA, USA, sep 2008. IEEE Computer Society.

Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. In J. Platt, D. Koller, Y. Singer, and S. Roweis, editors, Advances in Neural Information Processing Systems , volume 20. Curran Associates, Inc., 2007.

Ali Rahimi and Benjamin Recht. Weighted sums of random kitchen sinks: Replacing minimization with randomization in learning. In Daphne Koller, Dale Schuurmans, Yoshua Bengio, and Léon Bottou, editors, Advances in Neural Information Processing Systems 21, Proceedings of the Twenty-Second Annual Conference on Neural Information Processing Systems, Vancouver, British Columbia, Canada, December 8-11, 2008 , pages 1313-1320. Curran Associates, Inc., 2008.

Bharath K. Sriperumbudur and Zoltán Szabó. Optimal rates for random Fourier features. In Corinna Cortes, Neil D. Lawrence, Daniel D. Lee, Masashi Sugiyama, and Roman Garnett, editors, Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada , pages 1144-1152, 2015.

Yitong Sun, Anna C. Gilbert, and Ambuj Tewari. But how does it work in theory? Linear SVM with random features. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett, editors, Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada , pages 3383-3392, 2018.

Danica J. Sutherland and Jeff G. Schneider. On the error of random Fourier features. In Marina Meila and Tom Heskes, editors, Proceedings of the Thirty-First Conference on Uncertainty in Artificial Intelligence, UAI 2015, July 12-16, 2015, Amsterdam, The Netherlands , pages 862-871. AUAI Press, 2015.

Lloyd N. Trefethen and David Bau. Numerical Linear Algebra . SIAM, 1997.

Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461 , 2018.

Geoffrey S. Watson. Smooth regression analysis. Sankhy¯ a: The Indian Journal of Statistics, Series A (1961-2002) , 26(4):359-372, 1964.

Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al. Ethical and social risks of harm from language models. arXiv preprint arXiv:2112.04359 , 2021.

Jiaxuan Xie, Fanghui Liu, Kaijie Wang, and Xiaolin Huang. Deep kernel learning via random Fourier features. CoRR , abs/1910.02660, 2019.

Yang, Duraiswami, Gumerov, and Davis. Improved fast Gauss transform and efficient kernel density estimation. In Proceedings Ninth IEEE International Conference on Computer Vision , pages 664-671 vol.1, 2003.

Jiyan Yang, Vikas Sindhwani, Quanfu Fan, Haim Avron, and Michael W. Mahoney. Random Laplace feature maps for semigroup kernels on histograms. In 2014 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2014, Columbus, OH, USA, June 23-28, 2014 , pages 971-978. IEEE Computer Society, 2014.

Tianbao Yang, Yu-Feng Li, Mehrdad Mahdavi, Rong Jin, and Zhi-Hua Zhou. Nyström method vs random Fourier features: A theoretical and empirical comparison. In Peter L. Bartlett, Fernando C. N. Pereira, Christopher J. C. Burges, Léon Bottou, and Kilian Q. Weinberger, editors, Advances in Neural Information Processing Systems 25: 26th Annual Conference on Neural Information Processing Systems 2012. Proceedings of a meeting held December 3-6, 2012, Lake Tahoe, Nevada, United States , pages 485-493, 2012.

Manzil Zaheer, Guru Prashanth Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Minh Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Mahmoud El Houssieny Ahmed. Big bird: Transformers for longer sequences. 2020.

Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In IEEE international conference on computer vision , pages 19-27, 2015.

A Proofs

A.1 Proof of Theorem 4.1

Proof. By the definition of ⟨ ν DE , f (1) DE , f (2) DE ⟩ , we have:

Since 8 A ≺ I d , we have 4 A ≺ 0 . 5 I d ≺ I d , meaning that I d -4 A is positive definite and invertible. The following identity is straightforward to check:

Therefore, we have:

Next, we use the fact that the integral of the multivariate Gaussian distribution with mean µ and variance Σ is 1 :

From that we conclude:

Based on this expression, we conclude that, indeed, E ν DE f (1) DE ( ω , x ) f (2) DE ( ω , y ) = K (0) ( x , y ) for all x , y ∈ R d if the conditions from theorem's statement are satisfied.

Next, we calculate expression for the variance. For any random variable Z , Var Z = E Z 2 -( E Z ) 2 . In particular, if Z = f (1) DE ( ω , x ) f (2) DE ( ω , y ) , ω ∼ ν DE , we get:

We have:

Evaluation of the integral above can be done in the same way as calculation of E ν DE f (1) DE ( ω , x ) f (2) DE ( ω , y ) , noticing that I d -8 A is positive definite and invertible. The result is as follows:

We conclude that the variance expression given in the theorem's statement is correct.

A.2 Important lemma

Below, we prove an important lemma which is used in the subsequent proofs:

Lemma A.1. Consider a function f : ( -∞ , 1 8 ) defined as

where ϕ ≥ 0 . Then, the minimum of f on ( -∞ , 1 8 ) is achieved at

Proof. Set γ = (1 -8 A ) -1 ∈ (0 , + ∞ ) . Note that there is a one-to-one correspondence between γ ∈ (0 , + ∞ ) and A ∈ ( -∞ , 1 8 ) . Hence, we can substitute γ -1 = 1 -8 A and 1 -4 A = ((1 -8 A ) + 1) / 2 = ( γ -1 +1) / 2 = 1+ γ 2 γ in (14) and equivalently perform minimization with respect to γ :

For h ( · ) 's derivative, we have:

Based on (16), we see that h ′ ( γ ) → -∞ as γ → 0 and h ′ ( γ ) > ϕ ≥ 0 for all γ > 1 . Hence, we conclude that h ( · ) is bounded from below on (0 , + ∞ ) and the global minimum γ ∗ on (0 , + ∞ ) exists and it is one of the points satisfying h ′ ( γ ∗ ) = 0 . Hence, it's one of the positive roots of the polynomial in numerator of (17).

If ϕ = 0 , there is a single root γ ∗ = 1 of the polynomial in the numerator of (17), hence it is a global minimum of h ( · ) . If ϕ > 0 , then there are two roots of the polynomial in the numerator of (17):

Note that, if ϕ > 0 , then 2 ϕ + 1 > 0 and (2 ϕ + 1) 2 + 8 ϕ ≥ (2 ϕ + 1) 2 . Hence, γ ∗ -< 0 and γ ∗ + > 0 . We conclude that γ ∗ = γ ∗ + is the minimum of h ( · ) on (0 , + ∞ ) . We multiply numerator and denominator of (18)'s right hand side by (2 ϕ +1) + √ (2 ϕ +1) 2 +8 ϕ > 0 :

Note that the right hand side of (19) is equivalent to (18) when ϕ > 0 but also holds for the case when ϕ = 0 (i.e. when γ ∗ = 1 ). We conclude that f ( · ) is minimized at A ∗ = 1 8 (1 -( γ ∗ ) -1 ) since γ ∗ = (1 -8 A ∗ ) -1 . It's easy to see that (15) follows from (19) directly.

A.3 Proof of Theorem 4.2

Proof. With A = A I d , the conditions from Theorem 4.1 read as

for k ∈ { 1 , 2 } . And the variance expression (9) for all x , y ∈ R d transforms into

We express C ( k ) through A, B ( k ) and D through A using (20) in the equation above:

Since 1 1 -4 A ( B (1) ) ⊤ B (2) is a full-rank matrix I d (20), both B (1) and B (2) are full-rank. Hence, we can express B (2) = (1 -4 A )( B (1) ) -⊤ . Also, note that

We rewrite the expression for the variance using the identity above and the formula for B (2) :

We use the expression above to rewrite (8) for ⟨ ν, f (1) , f (2) ⟩ = ⟨ ν ADE , f (1) ADE , f (2) ADE ⟩ as follows:

Denote E = ( B (1) ) ⊤ B (1) . Then (21) becomes:

We next prove the following lemma:

Lemma A.2. Let B (1) ∗ = √ 1 -4 A Σ 1 / 2 U ⊤ ( Λ (1) ) -1 / 2 ( Q (1) ) ⊤ . When A ( 8 A < 1 ) is fixed, E = E ∗ = ( B (1) ∗ ) ⊤ B (1) ∗ minimizes the right hand side of (22) with respect to E .

Proof. We have:

where we use the cyclic property of trace Trace( · ) and linearity of trace. Analogously, we obtain L -1 ∑ L j =1 ( y ( j ) ) ⊤ E -1 y ( j ) = Trace( E -1 M (2) ) . Assuming that A is fixed, optimization of (22) with respect to E reduces to the following minimization problem:

where β 1 = (1 -8 A ) -1 (1 -4 A ) -1 , β 2 = (1 -8 A ) -1 (1 -4 A ) and the constraint E ∈ S d , E ≻ 0 follows from the fact that E = ( B (1) ) ⊤ B (1) and E is invertible. We have 1 -8 A > 0 and 1 -4 A = (1 -8 A ) / 2+1 / 2 > 0 . Hence, β 1 , β 2 > 0 . For any E ≻ 0 and any ∆ ∈ S d there is t ∈ R small enough such that E + t B is invertible and the following Neumann series is convergent:

We further deduce:

and, therefore,

Further, we have:

We replace M (2) = Q (2) ( Λ (2) ) 1 / 2 ( Λ (2) ) 1 / 2 ( Q (2) ) ⊤ and apply the cyclic property of trace in (25):

where T = ( Λ (2) ) 1 / 2 ( Q (2) ) ⊤ E -1 ∆ . Since E is positive definite, E -1 is also positive definite and TE -1 T ⊤ is at least positive semidefinite. Hence, Trace ( TE -1 T ⊤ ) ≥ 0 and also ∂ 2 ( ∂t ) 2 F ( E + t ∆ ) | t =0 ≥ 0 . We conclude that F ( E ) is a convex function on { E ∈ S d | E ≻ 0 } . Since { E ∈ S d | E ≻ 0 } is an open set, (every) global minimum E of (23) satisfies two conditions

Set t = 1 and assume that ∆ ∈ S d is small enough by norm so that E + ∆ is invertible and the Neumann series for ( I d + ∆E -1 ) -1 is convergent. Then, (24) holds for t = 1 :

Clearly, β 2 ∑ ∞ l =2 ( -1) l Trace ( E -1 ( ∆E -1 ) l M (2) ) = o ( ∥ ∆ ∥ ) where ∥ · ∥ is an L 2 -norm. Also, using the cyclic property of trace, we get:

Therefore, we have:

Since ∆ , E -1 , M (1) , M (2) ∈ S d , from (27) it follows that

Let E ∗ = ( B (1) ∗ ) ⊤ B (1) ∗ ⪰ 0 . Note that

Since √ β 2 /β 1 = 0 , Σ , U , Λ -1 / 2 , Q (1) are full-rank, E ∗ is also full-rank, therefore E ∗ ≻ 0 and it satisfies condition 1 from (26). Observe that

̸

where we use definitions of E ∗ , U , Σ , V and orthogonality of Q (1) , Q (2) , U , V . Hence, we deduce that

by the definition of Q (1) , Λ (1) , Q (2) , Λ (2) and due to orthogonality of V , U . By left- and rightmultiplication of (30) by ( E ∗ ) -1 we deduce that

or, in other words, ∇F ( E ∗ ) = 0 d × d and the condition 2 from (26) is also satisfied. We conclude that the global minimum of (23) is achieved at E ∗ .

According to Lemma A.2, B (1) = B (1) ∗ is a global minimum of (21)'s right hand side when A is fixed. Indeed, if there is B (1) which leads to a smaller value of (21), E = ( B (1) ) ⊤ B (1) would lead to a smaller value of (22)'s right hand side. Also, this E is positive definite by definition (note that B (1) is nonsingular), leading to contradiction with Lemma A.2.

Substituting E ∗ instead of E in (22) corresponds to the minimum value of L ( θ AGE ; α, X , Y , T AGE ) for a fixed A . Our next step is to minimize this expression with respect to A . Denote F = Q (1) ( Λ (1) ) -1 / 2 UΣU ⊤ ( Λ (1) ) -1 / 2 ( Q (1) ) ⊤ . Then E ∗ = (1 -4 A ) F where F doesn't depend on A . We substitute E ∗ into (22) and get:

where we also replace

and

Based on (30) and since F = √ β 1 /β 2 E ∗ , we conclude that FM (1) F = M (2) , or M (1) F = F -1 M (2) . Using the cyclic property of trace, we get:

By the definition of F , Λ (1) , Q (1) and using the cyclic property and orthogonality of Q (1) , U , we have:

Hence, (31) finally becomes:

Next, we use Lemma A.1 ( ϕ = d -1 ∑ d l =1 Σ l,l + µ (3) ≥ 0 ) for deriving expression for A which minimizes (32). This expression coincides with the one in Theorem's statement. The expressions for B (2) , C (1) , C (2) follow directly from (20), optimal B (1) = B (1) ∗ and A . (10) follows from (32). The proof is concluded.

A.4 Proof of Theorem 4.3

Proof. With B (1) = B (2) = B and C (1) = C (2) = C , the conditions from Theorem 4.1 read as

Denote Q = ( I d -4 A ) -1 / 2 B ∈ R d × d . Then, according to (33), Q ⊤ Q = I d , that is Q ∈ O d . We rewrite (9) using (33) and then substitute B = ( I d -4 A ) 1 / 2 Q :

where we denote:

which is in D d since A ∈ D d . Next, we observe:

We plug this into (34) and use the resulting expression to rewrite (8) for ⟨ ν, f (1) , f (2) ⟩ = ⟨ ν SDE , f (1) SDE , f (2) SDE ⟩ as follows:

Using linearity and cyclic property of trace, we deduce that

Observe that

Denote N = M (1) + µ (4) ( µ (5) ) ⊤ + µ (5) ( µ (4) ) ⊤ + M (2) . We conclude that

With A fixed, we minimize the right hand side of (37) with respect to Q which is equivalent to minimizing L ( θ SDE ; X , Y , T SDE ) with respect to B with fixed A , since there is a one-to-one correspondence between B and Q . This is equivalent to maximizing, again using the cyclic property of trace,

with respect to Q . We prove the following lemma first:

Lemma A.3. Suppose that diagonal entries of E are all distinct, and the same holds for Λ (3) . Let Π ∈ { 0 , 1 } d × d be a permutation matrix sorting diagonal entries of E (i.e. by applying ΠEΠ ⊤ ) in a descending order corresponding to a permutation π ∈ N d . Set Q ∗ = Π ⊤ ( Q (3) ) ⊤ ∈ O d . Then we have:

Proof. First of all, we have:

i.e. (39) is satisfied.

Optimization for finding sup Q ∈ O d Trace ( EQNQ ⊤ ) is a well-studied problem [9]. By the definition, Λ (3) has eigenvalues of N on the main diagonal and E ∈ D d hence it contains its eigenvalues on its main diagonal. Then, as proven in [9], Q ∗ is indeed a global maximum of this problem in the case of distinct eigenvalues for E and N . That is, (40) is proven.

Next, we prove a generalization of Lemma A.3 when diagonal entries of E and Λ (3) are not necessarily distinct:

Lemma A.4. Let Π ∈ { 0 , 1 } d × d be a permutation matrix sorting diagonal entries of E (i.e. by applying ΠEΠ ⊤ ) in any non-ascending order corresponding to a permutation π ∈ N d . Set Q ∗ = Π ⊤ ( Q (3) ) ⊤ ∈ O d . Then we have:

Proof. In the same way as (41-42), we show that Trace ( EQ ∗ N ( Q ∗ ) ⊤ ) = ∑ d l =1 E π l , π l Λ (3) l,l , i.e. (43) is satisfied. Next we prove that for any Q ∈ O d ,

which would imply (44).

Our proof is by contradiction. First of all, we can assume that E , Λ (3) are nonzero matrices since otherwise we have (44) trivially. Since Trace ( EQNQ ⊤ ) is a continuous function of Q and O d is compact, sup Q ∈ O d Trace ( EQNQ ⊤ ) is finite. Suppose that there is δ > 0 such that

Let ˜ E , ˜ Λ (3) ∈ D d be matrices with all distinct values on the diagonal such that

̸

where ∥ · ∥ F denotes Frobenius norm and ∥ E ∥ F , ∥ Λ (3) ∥ F = 0 since these are nonzero matrices. Further, we assume that diagonal entries of ˜ Λ (3) are sorted in a descending order and, in addition to Λ (3) , π also sorts entries of ˜ E in a non-ascending (descending) order. Clearly, such ˜ E , ˜ Λ (3) can be obtained by small perturbations of E , Λ (3) . Also, denote ˜ N = Q (3) ˜ Λ (3) ( Q (3) ) ⊤ . Since O d is a compact closed set and Trace ( EQNQ ⊤ ) is a continuous function of Q , there exists Q ∗∗ ∈ O d such that

By the definition of ˜ E , ˜ Λ (3) , ˜ N , we have:

Next, we apply Cauchy-Schwarz inequality to both terms:

where we use invariance of the Frobenius norm under multiplications by orthogonal matrices. Using this invariance again, we deduce that

We conclude that

Next, we apply Lemma A.3 to E = ̂ E , Λ (3) = ̂ Λ (3) and deduce that

We apply Cauchy-Schwarz inequality again to the second and the third term:

where we use invariance of the Frobenius norm under column and row permutations. We conclude that

We combine this inequality with (49) and obtain:

Next, we use triangle inequality and deduce that

Hence, we continue (50):

where in the last transition we use ∥ E -˜ E ∥ F ≤ ∥ E ∥ F which is according to (47). We continue this chain of inequalities using (47) again:

This is a contradiction with (46) taking into account Q ∗∗ 's definition (48). Hence, (44) is proven.

Let Q ∗ be defined as in Lemma A.4's statement. Further, we denote π ( A ) = π , Π ( A ) = Π where π , Π are defined as in Lemma A.4's statement. That is, π ( A ) denotes some permutation which sorts diagonal entries of E in a non-ascending order. It's a function of A since E is a function of A defined in (35). In fact, based on (35), we see that π ( A ) is some permutation which sorts diagonal entries of A in a non-descending order. Π ( A ) denotes a permutation matrix corresponding to π ( A ) . That is, diagonal entries of Π ( A ) AΠ ( A ) ⊤ are sorted in a non-descending order.

Let G ( A ) denote the right hand side of (37) where we substitute Q = Q ∗ . That is, G ( A ) is an optimal value of L ( θ SDE ; X , Y , T SDE ) with A fixed:

where we use E 's definition (35). Let π -1 ( A ) ∈ N d denote a permutation inverse to π ( A ) . By rearranging terms in the sum, we have:

Therefore, we have:

Define a new function G ( ̂ A , A ) , where ̂ A ∈ D d satisfies 8 ̂ A ≺ I d , as follows:

By the definition of G ( ̂ A , A ) , we have:

Hence, it holds:

Next, we show that there is a closed-form expression for the solution of inf ̂ A ∈ D d , 8 ̂ A ≺ I d G ( ̂ A , A ) . Since ̂ A ∈ D d , we have: log det( I d -4 ̂ A ) = ∑ d l =1 log(1 -4 ̂ A l,l ) , log det( I d -8 ̂ A ) = ∑ d l =1 log(1 -8 ̂ A l,l ) . We further have:

From (54), we see that minimization inf ̂ A ∈ D d , 8 ̂ A ≺ I d G ( ̂ A , A ) reduces to d independent minimization problems with respect to ̂ A l,l such that 8 ̂ A l,l < 1 . l 'th problem, 1 ≤ l ≤ d , is solved using Lemma A.1 where we set ϕ = Λ (3) π -1 ( A ) l , π -1 ( A ) l . Let A ∗∗ ∈ D d denote the corresponding solution. Then, for all 1 ≤ l ≤ d , we have:

From (53) it follows that

Denote E ∗∗ = -1 2 I d -1 2 ( I d -8 A ∗∗ ) -1 . Then we have:

where the second transition follows from Lemma A.4 and the fact that π ( A ∗∗ ) sorts diagonal entries of A ∗∗ in a non-descending order, hence its sorts diagonal entries of E ∗∗ in a non-ascending order (recall the definition of π ( A ∗∗ ) and E ∗∗ ). Denote E ∗ = Π ( A ∗∗ ) E ∗∗ Π ( A ∗∗ ) ⊤ . Then E ∗∗ π ( A ∗∗ ) l , π ( A ∗∗ ) l = E ∗ l,l for all 1 ≤ l ≤ d and

Further, we have:

where we denote A ∗ = Π ( A ∗∗ ) A ∗∗ Π ( A ∗∗ ) ⊤ , i.e. A ∗ l,l = A ∗∗ π ( A ∗∗ ) l , π ( A ∗∗ ) l for all 1 ≤ l ≤ d . Given the definition of A ∗∗ (55), for all 1 ≤ l ≤ d we have:

That is, A ∗ is independent of A . Based on (59), we see that smaller values of Λ (3) l,l result in bigger values of A ∗ l,l . Since Λ (3) 1 , 1 , . . . , Λ (3) d,d are ordered in a non-ascending order, we deduce that A ∗ 1 , 1 , . . . , A ∗ d,d are ordered in a non-descending order. By the definition of π ( A ∗ ) , we then have A ∗ l,l = A ∗ π ( A ∗ ) l , π ( A ∗ ) l for all 1 ≤ l ≤ d . Therefore,

Combining this with (58), (57), we can continue the chain of inequalities (56):

where in the second transition we use the fact that

and, similarly, det( I d -8 A ∗ ) = det( I d -8 A ∗∗ ) . In the third transition, we use definition of G ( · ) (51). Note that (60) holds for all A ∈ D d such that 8 A ≺ I d and also 8 A ∗ ≺ I d since 8 A ∗∗ ≺ I d . We conclude that, when B , C , D are chosen optimally with a given A , the minimum of L ( θ SDE ; X , Y , T SDE ) is reached when A = A ∗ . As we have already deduced, diagonal entries of A = A ∗ are sorted in the non-descending order. Hence, using Lemma A.4's notation, diagonal entries of E are already sorted in a non-ascending sorting order and π = (1 , . . . , d ) , Π = I d satisfy requirements of the Lemma. Hence, with A = A ∗ , the optimal B has a form ( I d -4 A ) 1 / 2 Q ∗ where Q ∗ = I d ( Q (3) ) ⊤ = ( Q (3) ) ⊤ . Optimal C and D are further determined by (33). (11) follows from (60) and the fact that, as discussed above, we can replace π ( A ∗ ) l with l in (60), 1 ≤ l ≤ d .

B Additional experimental details

B.1 Compute resources and implementation

We use NumPy [25] in Google Colaboratory the variance comparison and kernel classification experiment. For the Transformer setups, we use TPU cluster and JAX [7] library. All tested Transformer variants were trained and tested on a TPU pods containing 4 TPU v3 chips with JAX and on GPUs (V100).

B.2 Variance comparison

We repeat the setup of [31] closely: we draw 5 pairs of sets { x ( i ) } 1 ≤ i ≤ L , { y ( j ) } 1 ≤ j ≤ L , L = 1024 . On each pair, we compute the relative variance for all pairs of points and for all indicated RF methods. Further, the shifted log-variance is optimized separately on each pair of sets for GERF, ADERF and SDERF.

We take M = 1 since M 's value is not important in this experiment: bigger M would just shift the curves below. The reported curves are means over all pairs of points and over all 5 sets.

B.3 Kernel classification

As in [31], we obtain training, validation and test splits by shuffling the raw dataset and taking 90% , 5% , 5% objects respectively. The splits are fixed for all RF methods. We tune σ on a logarithmic grid of 10 values on [10 -2 , 10 2 ] . For each σ and each RF type, we try 50 seeds for drawing RFs during validation and testing. Testing is performed for the best σ only. Figure 3 reports averages over 50 seeds. We use orthogonal ω 's for all types of RFs as described in [31], since orthogonal random features work better in practice [16, 31].

B.4 DERFs for long-sequence Transformers

B.4.1 Speech modelling

Our Conformer-Transducer variant was characterized by: 20 conformer layers, model _ dim = 512 , relative position embedding dimensionality rped = 512 and h = 8 heads. We used batch size bs = 2048 and trained with the adam optimizer on TPUs. For the regular Conformer-Transducer training, we run ablation studies over different number of random features: m = 8 , 32 , 128 . In the NST setting, we run experiments with m = 8 . We reported commonly used metric: normalized word error rate (NWER).

B.4.2 Natural language processing

We pretrained BERT model on two publicly available datasets (see: Table 3). Following the original BERT training, we mask 15% of tokens in these two datasets, and train to predict the mask. All methods were warm started from exactly the same pre-trained checkpoint after 1M iteration of BERT

pretraining. We used the exact same hyperparameter-setup for all the baselines (FAVOR++[31], FAVOR+ [16], ELU [26], ReLU [16]) and FAVOR++. The hyperparameters for pretraining are shown in Table 2. We finetuned on GLUE task, warm-starting with the weights of the pretrained model. The setup is analogous to the one from the original BERT paper.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of computing the matrix \"KC\". Here, K is the kernel matrix (L-by-L, with L very large, could be reproducing kernel, but OK otherwise), and C is a known constant matrix. The kernel is restricted to be \"scaled softmax kernel\": exp(x-dot-y)F(x)F(y) (for some function F not difficult to handle). The contribution of this paper, is the design of a class of random features, of which the correlation is exactly K, but to some extent, the variance of replacing the mean (for the correlation) by sample mean, can be minimized with closed form. This research, to the best of my knowledge, is innovative, inspiring, and interesting.\n\nSTRENGTHS:\nIn this paper, the idea, and the derived formulation of optimizing sample variance for improved kernel estimation is innovative. Besides the notation problem raised below, the paper has a clear and friendly structure that is easy to follow. The main results are presented clearly. These suggest excellent writing quality and clarity. We do have concerns on significance, but we would like to raise such concerns below, instead of here.\n\nWEAKNESSES:\n1. Concerns about the writing. I feel that the notations are unnecessarily too heavy. For example, I can not see any reason that blocks one from rewriting:\nf^(k) --> f^k (or just f and g), B^(k) --> B_k, C^(k) --> C_k or C^k, x^(i) --> x_i, y^(i) --> y_i, M^(k), Q^(k), Lambda^(k), .... By the rewriting, the number of notations is reduced by 50%! Some notations only appear a few times and can just be replaced by English, e.g., O_d (the set of orthogonal matrices).\n2. The significance of the paper relies on the significance of the exponential kernel exp(x-dot-y). If computing the transformer attention is the only application, the vanilla computational cost should be compared. Otherwise, I feel not certain if this is a widely used kernel family, and (e.g., for Gaussian kernel) in the application scenarios (e.g., regularized least squares, or support vector machines), whether computing KC is really the bottleneck.\n\nQUESTIONS:\n1. Lines 76--77. Could you please provide more details on why the computational complexity of hat-K-times-C is O(LMn)?\n2. In Eq. (8), the variable \"theta\" does not show up in the right-hand side, and is not elsewhere defined. Better if this variable can be defined.\n3. It remains very unclear even after a careful study of Section 3, what are \"homogeneity heuristic\" and \"a certain optimization problem\" mentioned in Lines 124 and 125. Better if these can be explicitly pointed out.\n4. Consider the mission of computing the matrix KC, as elaborated in Section 2.1, whether adopting DERFs would reduce computational complexity? To be specific, whether the amount of variance reduced, deserves the computational cost incurred by optimizing the random features via the approaches introduced in Section 4? Can one achieve the same accuracy and computation complexity simply by using a larger M? For example, Theorem 4.2.\n\nLIMITATIONS:\nfoundational research, limitations well managed.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

comments: SUMMARY:
In this paper, the GERFs are genearilzied to dense exponential random features (DERFs). The paper shows that GERFs and PosRFs are the specific situation of DERFs, from which if follows that with a suitable parameter estimator, DERFs can have better performance.

STRENGTHS:
As far as I know, the DERFs are novel and are a good extension from GERFs.

There are sufficient discussions and experiments from many aspects, all showing the good performance of the proposed method. 

The paper is well organized and clearly represented.

WEAKNESSES:
I did not find significant weakness. One possible is that because this paper include many aspects, it may make the reader lose the focus. 

It is better to clearly express the suggestion: in which case DERFs is recommended.

QUESTIONS:
In different experiments, several random features are compared when the number of random features are the same. How about the real calculation time? Since there could be difference on calculating different random features in the inference.

LIMITATIONS:
not applicable
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

comments: SUMMARY:
The authors propose new random features for Gaussian and softmax kernels, and apply the approximation to learning scalable Transformer networks.

STRENGTHS:
1. The proposed dense exponential random features (DERFs) generalize the current positive random features (PosRFs) and  generalized exponential random features (GERFs).
2. The authors show stronger theoretical results for the proposed DERFs in scalable Transformer networks where the self-attention matrix is approximated as a low-rank matrix when the sequence is long.
3. The utility of the method is demonstrated on a variety of datasets with competitive results.

WEAKNESSES:
1. While the authors argue that significant variance reduction can be achieved, it is not clear whether the proposed approximation is unbiased.
2. The method could be computationally expensive---the term D involves computing the determinant.
3. Theorem 4.1 makes a few assumptions, restrictions of these conditions are not fully discussed.

QUESTIONS:
How feasible are the conditions in Theorem 4.1?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper is the next instalment in a series of works focusing on low-rank transforners, e.g. [15, 30]. They propose Dense Exponential Random Features (DERF) for unbiased Monte Carlo approximation of Gaussian or softmax kernels. This class of features generalizes GERFs from previous work, with the main difference being that the features are parametrized by matrix-valued parameters rather than scalars. Hence, this flexibility allows for capturing a larger class of features for kernel approximation.  The parameters themselves can be analytically optimized in several useful special cases: ADERF, SDERF and SADERF. The parameter optimization is carried out with respect to a so-called shifted log-variance objective averaged over the dataset, and it is shown that optimizing this objective formalizes a \"homogeneity heuristic\" from previous work, which is a nice observation. In particular, this allows to select the same set of parameters across all data points. The experiments are essentially the same as in previous works: variance analysis on synthetic datasets + cifar + mnist, a set of nonparametric kernel regression benchmarks, speech modelling and low-rank uptraining on NLP tasks. Improvements compared to previous work and scalable transformers are achieved.\n\nSTRENGTHS:\nThe paper is overall well written, it does really well at communicating the main issues and ideas on which it is based. There is both a theoretical and experimental component, both of which have some interesting results. The main idea is novel, and the proposed approach for generalizing previous random feature methods is non-trivial; I was impressed that the analytical calculations could also be carried out in the matrix-valued case. I believe it is of interest to both the kernel and efficient transformer communities to have some scalable and expressive random features, and the paper definitely takes another step in this direction.\n\nWEAKNESSES:\nOverall there are not that many weaknesses in my opinion. The impact is slightly less significant than of its predecessor [30], where the main novelty was allowing for low-rank uptraining of pre-trained Transformers, which was already achieved by FAVOR++ (efficient Transformer based on GERFs). Nevertheless, the improved experimental performance should be of interest.\n\nI was missing some more comparisons between the variations: for one, SADERFs were not included in any of them, and only used for the last NLP task. Computationally, this variant seems to be the most favourable since it does not require the eigendecomposition of a $d \\times d$ matrix, hence foregoing an $O(d^3)$ computational cost, which is associated with ADERF and SDERF. It would be interesting to see this variation included in the comparisons to have an idea about the trade-off between the performance and this computational saving. \n\nI was also missing some intuition or hypothesis regarding why SDERF seems to perform best. Is it because it allows for a non-isotropic matrix $A$, hence allowing to adapt the variance of $\\omega$ on a coordinate-by-coordinate basis in the quadratic form containing $\\omega$, if this makes sense? It would be interesting to know which additional parameters result in the biggest improvement compared to GERF, so that focus could be placed on optimizing the scalability-performance trade-off. (See questions for more)\n\nAnother question that seems unaddressed to me is how much information is lost compared to a full Transformer on truly long-range tasks. At the moment, we do not know the \"price\" we pay for the subquadratic scalability in sequence length, and it would also be interesting to include some more long-range tasks, which compares full Transformer vs RF methods.\n\nQUESTIONS:\nSome questions and remarks:\n- Is $\\sigma$ analogous to the bandwidth parameter of the kernel in Figure 1 (right) and Figure 2? This is not stated explicitly.\n- In line 138-139, it is stated that [30] achieved good results by optimizing the shifted log-variance. Given that this objective is only introduced in this work, maybe I would phrase this differently, since in previous work it was only incorporated as a heuristic for homogenizing the solution, rather than an optimization problem.\n- How significant is the $O(d^3)$ computation cost associated with SDERF and ADERF? I guess the previous layer could easily have an overall dimension in the 1000s, where this could become a bottleneck on GPUs in terms of memory? \n- In line 214, the authors make the remark that assuming $L \\geq d$, the $O(Ld^2)$ cost dominates the $O(d^3)$ cost. Is this not already assumed in Theorem 4.2 (and hence, I assume in Theorem 4.3 from the way it is phrased), since this is necessary for the nonsingularity of $M^{(1)}$ and $M^{(2)}$?\n- If the main improvement of SDERF compared to ADERF comes from allowing a non-isotropic $A$, would it be beneficial to define extensions of GERF or SADERF, which add this flexibility to the parametrization? What might be the computational cost of such an RF construction? This is just a hunch, but if this can be solved for without matrix decompositions, then we might have the best of both worlds? No problem if working this out is difficult, just curious.\n- What is the range of sequence lengths in the speech modelling benchmark? Does this count as a long-range task?\n- In the NLP task, what is the number of random features (number of MC samples) used? Is there any intuition, investigation that the authors could report about how to choose this hyperparameter?\n- One more question; in the Appendix it is stated that replacing the standard normal distribution on $\\omega$ with an orthogonal one works better in practice. is this achieved by QR decomposing a set of Gaussian vectors? Is there any intuition about why this improves the performance?\n\nLIMITATIONS:\nThe paper is mainly theoretical, but it has direct implications regarding Transformers, which carry with themselves a variety of societal and environmental impacts, but this seems to be addressed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work focuses on positive linear features for softmax kernels which are relevant for accelerating kernel methods and transformers. The paper observes that the functional form for random Fourier features (GERFs) proposed by prior work in [30] can be seen as optimizing the shift-ed log-variance objective. Building on GERFs, a more expressive and parameterized functional form is proposed, and corresponding shifted-log variance is derived. After that, several simplifications are considered for improved optimization of the log-shifted variance of the random features, which leads to different forms of RFs that are easily computable while keeping subquadratic complexity in the number of sequence lengths. Towards the end, a few empirical results are presented in support of their contributions.\n\nSTRENGTHS:\n* This work studies an important class of problems that has broader applicability. The paper builds on observations/heuristics from the prior work and attempts to justify using theoretical arguments. During this exercise, the proposed methodologies are mathematically elegant and present several exciting results on RFs that are relevant to both kernel methods and transformers.  \n* While I have not examined every mathematical proof, the derivation appears generally accepted, insightful, and praiseworthy.\n\nWEAKNESSES:\n* To establish that GERF minimizes the objective, the supporting argument is the solution of 8 matches with the heuristic proposed by the [30]. If I have understood it correctly, more is needed as for an objective to be reasonable, its behavior beyond the maximizer/minimizer needs to be understood. Also, is there any relation between this objective and the actual variance of the GERF estimate? Can Jensen inequality be used? This is one of the main results of this paper, and it needs to be better motivated. \n* Notations could be improved. E.g., use a consistent format for scalars and matrices. L is a scalar. Avoid unnecessary precision; e.g., the upper subscript in lines 157 and 158 could have been avoided and stated as part of the text. Note that this work is built on math and therefore requires extra effort on styling to make it accessible for a larger audience. \n* Pages 5-6-7 are not used carefully and do not present the best aspects of this work. You are considering simple formulations and repeating more or less the same procedure for obtaining close form solution of the objective, which again needs to be justified better in the first place. Also, before going into these details, give an overview of possible choices on the parameters space and justification. Summarizing these results as a corollary instead of Theorem 4.3 and 4.4 might be helpful. \n* In line 217: “operations for which implementation has not yet matured in popular deep learning libraries with GPU and TPU support.” What is the evidence for this claim? Are you suggesting that eigen decomposition can’t be computed efficiently on GPUs using torch and TensorFlow?\n\nQUESTIONS:\n* Is it possible to repeat the proof of Theorem 4.2 for a setting where the log of features comes from a non-linear function, potentially a neural network? The main idea is to extend from the quadratic functions of w and x, use a more general parametric form, and optimize a relaxation of the variance.  Note that to ensure the unbiasedness of the kernel, estimation can be imposed using constraints similar to the proposed work. How would you compare and contrast with the current approach? If not, why is it not possible?\n* Why are time versus accuracy results not reported anywhere, assuming scalability is one of the main motivations? When will these methods accelerate, and when will they not? Theoretical complexity may not account for constants that stem from implementation. \n* What aspects of the implementation may prohibit practitioners interested in the fast self-attention method from considering these proposed methods?\n* Respond to weakness 1.\n\nLIMITATIONS:\nPlease take a look at the weakness and questions.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-26"}