Large-Scale Learning with Fourier Features and Tensor Decompositions

1 Introduction . Kernel methods such as Support Vector Machines and Gaussian Processes are commonly used to tackle problems such as classification , regression and dimensionality reduction . Since they can be universal function approximators [ 13 ] , kernel methods have received renewed attention in the last few years and have shown equivalent or superior performance to Neural Networks [ 22 , 26 , 11 ] . The main idea behind kernel methods is to lift the data into a higher-dimensional ( or even infinitedimensional ) Reproducing Kernel Hilbert Space H by means of a feature map φ ( · ) : X → H. Considering then the pairwise similarities between the mapped data allows to tackle problems which are highly nonlinear in the original sample space X , but linear inH . This can be done equivalently by considering a kernel function k ( · , · ) : X × X → R such that 〈φ ( x ) , φ ( x′ ) 〉 = k ( x , x′ ) and performing thus said mapping implicitly . Although effective at learning nonlinear patterns in the data , kernel methods are known to scale poorly as the number of data points N increases . For example , when considering Kernel Ridge Regression ( KRR ) , Gaussian Process Regression ( GPR ) [ 33 ] or Least-Squares Support Vector Machine ( LS-SVM ) [ 41 , 42 ] training usually consist in inverting the Gram matrix kij = k ( xi , xj ) , which encodes the pairwise relation between all data . As a consequence , the associated storage complexity is O ( N2 ) and the computational complexity is O ( N3 ) , rendering these methods unfeasible for large data . In order to lower the computational cost , data dependent methods approximate the kernel function by means of M data dependent basis functions . Due to their reduced-rank formulation , the computational complexity is reduced to O ( NM2 ) for N M . However , e.g . for the Nyström method [ 46 ] , the convergence rate is 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . only of O ( 1/√M ) [ 7 ] limiting its effectiveness . As the name suggests , data independent methods approximate the kernel function by M data independent basis functions . A good example is the celebrated Random Fourier Features approach by Rahimi and Recht [ 31 ] , where the authors propose for stationary kernels a low-dimensional random mapping z ( · ) : RD → RM such that k ( x , x′ ) = 〈φ ( x ) , φ ( x′ ) 〉 ≈ 〈z ( x ) , z ( x′ ) 〉 . ( 1 ) As in the case of data dependent methods , the reduced-rank formulation allows for a computational complexity of O ( NM2 ) for N M . Probabilistic error bounds on the approximation are provided which result in a convergence rate of O ( 1/√M ) , which is again the Monte Carlo rate . Improvements in this sense were achieved by considering deterministic features resulting from dense quadrature methods [ 5 , 25 , 36 ] and kernel eigenfunctions [ 16 , 39 ] . These methods are able to achieve exponentially decreasing upper bounds on uniform convergence guarantees when certain conditions are met . However , for a D-dimensional input space these methods take the tensor product of D vectors , resulting in an exponential increase in the number of basis functions and thus of model weights , effectively limiting the applicability of deterministic features to low-dimensional data . In this work we consider deterministic features . In order to take advantage of the tensor product structure which arises when mapping the inputs to H , we represent the weights as a low-rank tensor . This allows us to learn the inter-modal relations in the tensor product of ( low-dimensional ) Hilbert spaces , avoiding the exponential computational and storage complexities in D. In this way we are able to obtain a linear computational complexity in both the number of samples and in the input dimension during training , without having to resort to the use of sparse grids or additive modeling of kernels . The main contribution of this work is in lifting the curse of dimensionality affecting deterministic Fourier features by modeling the weights as a low-rank tensor . This enables the efficient solution of large-scale and high-dimensional kernel learning problems . We derive an iterative algorithm under the exemplifying case of regularized squared loss and test it on regression and classification problems . 2 Related work . Fourier Features ( FF ) are a collection of data independent methods that leverage Bochner ’ s theorem [ 35 ] from harmonic analysis to approximate stationary kernels by numerical integration of their spectral density p ( · ) : k ( x , x′ ) Stationarity : = k ( x− x′ ) Bochner= ∫ p ( ω ) exp ( 〈iω , ( x− x′ ) 〉 ) dω FF≈ 〈z ( x ) , z ( x′ ) 〉 . ( 2 ) Rahimi and Recht [ 31 ] proposed to approximate the integral by Monte Carlo integration i.e . by drawing M random frequencies ω ∼ p ( · ) . In their work they show how the method converges uniformly at the Monte Carlo rate . See [ 23 ] for an overview of random FF . In order to achieve faster convergence and a lower sample complexity , a multitude of approaches that rely on deterministic numerical quadrature of the Fourier integral were developed . These methods generally consider product kernels whose spectral density factors in the frequency domain , which enables in turn to factor the Fourier integral . The resulting deterministic Fourier features are then the tensor product of D one-dimensional deterministic features . For example , in [ 5 ] the authors give an analysis of the sample complexity of features resulting from dense Gaussian Quadrature ( GQ ) . In [ 25 ] the authors present a similar quadrature approach for kernels whose spectral density factors over the dimensions of the inputs . They provide an explicit construction for their Quadrature Fourier Features relying on a dense Cartesian grid , and note that their method , as well as GQ , can attain exponentially decreasing uniform convergence bounds in the total number of basis functions per dimensions M̂ [ 25 , Theorem 1 ] . To avoid the curse of dimensionality , they make use of additive modeling of kernels . Variational Fourier Features ( VFF ) [ 16 ] are derived probabilistically in a one-dimensional setting for Matérn kernels by projecting a Gaussian Process onto a set of Fourier basis . An extension to multiple dimensions when considering product kernels is then achieved by taking the tensor product of the one-dimensional features , incurring however again in exponentially rising computational costs in D. In what they call HilbertGP [ 39 ] the authors diagonalize stationary kernels in terms of the eigenvalues and eigenfunctions of the Laplace operator with Dirichlet boundary conditions . Due to the multiplicative structure of the eigenfunctions of the Laplace operator , the complexity of the basis function increases exponentially in D when one considers product kernels . Like with other deterministic approximations , bounds on the uniform convergence error which decrease exponentially in M̂ can be achieved [ 39 , Theorem 8 ] . Tensor decompositions have been used extensively in machine learning in order to benefit from structure in data [ 38 ] , in particular to obtain an efficient representation of the model parameters . In the context of Gaussian Process Regression , the tensor product structure which arises when the inputs are located on a Cartesian grid have been exploited to speedup inference [ 12 ] . In their variational inference framework , [ 19 ] proposed to parameterize the posterior mean of a Gaussian Process by means of a tensor decomposition in order to exploit the tensor product structure which arises when interpolating the kernel function . In the context of neural networks , [ 27 ] proposed to represent the weights in deep neural networks as a tensor decomposition to speed up training . A similar approach was carried out in case of recurrent neural networks [ 48 ] [ 43 ] . Tensor decompositions have also been used to learn from simple features . In [ 45 ] multivariate functions are learned from a Fourier basis , which corresponds to assuming a uniform spectral density in ( 2 ) . Motivated by spin vectors in quantum mechanics , [ 40 ] consider trigonometric feature maps which they argue induce uninformative kernels . On a similar note , [ 28 ] and [ 1 ] leverage the tensor product structure of polynomial mappings to induce a polynomial kernel and consider model parameters in decomposed form . While these existing approaches are able to overcome the curse of dimensionality affecting tensor product feature maps by modeling the parameter tensor as a tensor network , they consider simple and empirical feature maps which induce uninformative kernels . In the following section we show how deterministic Fourier features can be used for supervised learning in both large and high-dimensional scenarios by assuming that the model weights are a low-rank tensor , thereby linking tensor decompositions with stationary product kernels . 3 Learning with deterministic Fourier features . 3.1 Notation . All tensor operations and mappings can be formulated over both the complex and the real field . For the remainder of this article real-valued mappings will be considered . Vectors and matrices are denoted by boldface lowercase and uppercase letters , respectively . Higher-order tensors are denoted by boldface calligraphic letters . Tensor entries are always written as lightface lowercase letters with a subscript index notation . For example , element i1 , i2 , i3 of the third-order tensorA ∈ RI1×I2×I3 is written as ai1i2i3 . The vectorization vec ( A ) of a tensorA ∈ RI1×I2×···×ID is the vector such that vec ( A ) i = ai1i2···iD , where the relationship between the linear index i and i1i2 . . . iD is given by i = i1 + D∑ d=2 ( id − 1 ) d−1∏ k=1 Ik . The Frobenius inner product between tensorsA ∈ RI1×I2×···×ID and B ∈ RI1×I2×···×ID is defined as 〈A , B〉F : = I1∑ i1=1 I2∑ i2=1 · · · ID∑ iD=1 ai1i2···iDbi1i2···iD = vec ( A ) Tvec ( B ) . Depending on the context , the symbol ⊗ either denotes the tensor outer product or the matrix Kronecker product . The Khatri-Rao productA~B of the matricesA ∈ RN×R andB ∈ RM×R is the NM ×R matrix that is obtained by taking the column-wise Kronecker product . The Hadamard ( element-wise ) matrix product ofA ∈ RN×R andB ∈ RN×R is denoted byA B . 3.2 The model . In this article we assume models of the form f ( x ) = 〈w , φ ( x ) 〉 , y = f ( x ) + . ( 3 ) Here x ∈ RD is the input vector , φ ( · ) : RD → H is a feature map , w ∈ H are the model weights , y ∈ R is the corresponding available observation corrupted by a zero mean i.i.d . noise . Learning such a kernelized model consists in finding as set of weights w such that : N∑ n=1 L ( yn , f ( xn ) ) + λ 〈w , w〉p , ( 4 ) is minimized . Here , L ( · , · ) : R× R→ R+ is a symmetric and positive loss function and λ 〈w , w〉p is a p-th norm regularization term . A variety of primal machine learning formulations arise when considering different combinations of loss functions and regularization terms . For example , considering p = 2 and squared loss leads to Kernel Ridge Regression [ 42 ] , while considering hinge loss leads to SVM [ 4 ] , and so on . Furthermore , applying the kernel trick when considering φ ( · ) recovers the nonparametric formulation which , depending on the choice of kernel , enables to perform inference using infinitely many basis functions with a computational complexity of O ( N3 ) . Considering instead a low-dimensional finite mapping z ( · ) : X → RM such as Random Fourier Features or the Nyström method leads a primal approach and to computational savings with a computational complexity of O ( NM2 ) for N M . However , as already discussed , these low-dimensional random mappings converge at the slow Monte Carlo rate , motivating our approach . In order to approximate the kernel function with faster and possibly exponential convergence , we consider deterministic , finite-dimensional features which are the tensor product of D vectors . For a given D-dimensional input point x we define the deterministic feature mapping Z ( · ) : RD → RM̂1 ⊗ RM̂2 ⊗ · · · ⊗ RM̂D as Z ( x ) = z ( 1 ) ( x1 ) ⊗ z ( 2 ) ( x2 ) · · · ⊗ z ( D ) ( xD ) , ( 5 ) where z ( d ) is a deterministic mapping applied to the d-th dimension . The dimension of the feature space is denoted M = ∏D d=1 M̂d . Without loss of generality it is from here on assumed that M̂1 = · · · = M̂D = M̂ such that M = M̂D . It is easy to verify by applying the kernel trick that the features of ( 5 ) yield product kernels . The mapping Z ( · ) encompasses for instance the mappings derived by quadrature [ 5 , 25 , 36 ] and by projection [ 16 , 39 ] . Note that these mappings cover many popular kernels such as the Gaussian and Matérn kernels . To give a concrete example , in the framework of A. Solin and S. Särkkä [ 39 , Equation 60 ] , for input data centered in a hyperbox X = [ −U1 , U1 ] ⊗ · · · ⊗ [ −UD , UD ] , the Gaussian kernel is approximated by means of D tensor products of M̂ weighted sinusoidal basis functions with frequencies lying on a harmonic scale such that : ( z ( d ) ( xd ) ) id = 1√ Ud p ( πid 2Ud ) sin ( πid ( xd + Ud ) 2Ud ) , id = 1 , . . . , M̂ . ( 6 ) Here p ( · ) is the spectral density of the Gaussian kernel with one-dimensional inputs , which is known in closed-form [ 33 , page 83 ] . This deterministic mapping then approximates the Gaussian kernel function [ 39 , Equation 59 ] such that k ( x , x′ ) ≈ 〈vec ( Z ( x ) ) , vec ( Z ( x′ ) ) 〉 = 〈Z ( x ) , Z ( x′ ) 〉F , and converges uniformly with exponentially decreasing bounds [ 39 , Theorem 8 ] . We therefore use Z ( · ) instead of φ ( · ) in ( 3 ) to obtain f ( x ) = 〈w , vec ( Z ( x ) ) 〉 = 〈W , Z ( x ) 〉F , ( 7 ) where the weight vector w has been reshaped into a D-dimensional tensorW ∈ RM̂1×M̂2×···×M̂D . Learning the exponential number of model parametersW in ( 7 ) under a hinge loss leads to Support Vector Machines , while considering a squared loss leads to the primal formulation of Kernel Ridge Regression , which we will consider as exemplifying case from here on : min W N∑ n=1 ( yn − 〈W , Z ( xn ) 〉F ) 2 + λ〈W , W〉F . ( 8 ) Since the number of elements M inW and Z grows exponentially in D , this primal approach is advantageous compared to the nonparametric dual approach only if M̂D N , limiting it to low-dimensional inputs . In order to lift this curse of dimensionality , we propose to represent and to learnW directly as a low-rank tensor decomposition . A low-rank tensor decomposition allows us to exploit redundancies inW in order to obtain a parsimonious model with a storage complexity that scales linearly in both M̂ and D. The low-rank structure will , as explicitly shown in the experiments , act as a form of regularization by limiting the total number of degrees of freedom of the model .

Summary: Authors present a tensor kernel ridge regression method where the covariance kernel is replaced with random Fourier features to reduce the typical O(N^3) cost down to O(NM^2) for M random frequencies. RFFs typically have slow convergence in approximating the full kernel w.r.t. to the number of random frequencies. The tensor is assumed to have a rank-R CPD decomposition to further reduce the complexity of the problem.