Large-Scale Learning with Fourier Features and Tensor Decompositions

1 Introduction . Kernel methods such as Support Vector Machines and Gaussian Processes are commonly used to tackle problems such as classification , regression and dimensionality reduction . Since they can be universal function approximators [ 13 ] , kernel methods have received renewed attention in the last few years and have shown equivalent or superior performance to Neural Networks [ 22 , 26 , 11 ] . The main idea behind kernel methods is to lift the data into a higher-dimensional ( or even infinitedimensional ) Reproducing Kernel Hilbert Space H by means of a feature map φ ( · ) : X → H. Considering then the pairwise similarities between the mapped data allows to tackle problems which are highly nonlinear in the original sample space X , but linear inH . This can be done equivalently by considering a kernel function k ( · , · ) : X × X → R such that 〈φ ( x ) , φ ( x′ ) 〉 = k ( x , x′ ) and performing thus said mapping implicitly . Although effective at learning nonlinear patterns in the data , kernel methods are known to scale poorly as the number of data points N increases . For example , when considering Kernel Ridge Regression ( KRR ) , Gaussian Process Regression ( GPR ) [ 33 ] or Least-Squares Support Vector Machine ( LS-SVM ) [ 41 , 42 ] training usually consist in inverting the Gram matrix kij = k ( xi , xj ) , which encodes the pairwise relation between all data . As a consequence , the associated storage complexity is O ( N2 ) and the computational complexity is O ( N3 ) , rendering these methods unfeasible for large data . In order to lower the computational cost , data dependent methods approximate the kernel function by means of M data dependent basis functions . Due to their reduced-rank formulation , the computational complexity is reduced to O ( NM2 ) for N M . However , e.g . for the Nyström method [ 46 ] , the convergence rate is 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . only of O ( 1/√M ) [ 7 ] limiting its effectiveness . As the name suggests , data independent methods approximate the kernel function by M data independent basis functions . A good example is the celebrated Random Fourier Features approach by Rahimi and Recht [ 31 ] , where the authors propose for stationary kernels a low-dimensional random mapping z ( · ) : RD → RM such that k ( x , x′ ) = 〈φ ( x ) , φ ( x′ ) 〉 ≈ 〈z ( x ) , z ( x′ ) 〉 . ( 1 ) As in the case of data dependent methods , the reduced-rank formulation allows for a computational complexity of O ( NM2 ) for N M . Probabilistic error bounds on the approximation are provided which result in a convergence rate of O ( 1/√M ) , which is again the Monte Carlo rate . Improvements in this sense were achieved by considering deterministic features resulting from dense quadrature methods [ 5 , 25 , 36 ] and kernel eigenfunctions [ 16 , 39 ] . These methods are able to achieve exponentially decreasing upper bounds on uniform convergence guarantees when certain conditions are met . However , for a D-dimensional input space these methods take the tensor product of D vectors , resulting in an exponential increase in the number of basis functions and thus of model weights , effectively limiting the applicability of deterministic features to low-dimensional data . In this work we consider deterministic features . In order to take advantage of the tensor product structure which arises when mapping the inputs to H , we represent the weights as a low-rank tensor . This allows us to learn the inter-modal relations in the tensor product of ( low-dimensional ) Hilbert spaces , avoiding the exponential computational and storage complexities in D. In this way we are able to obtain a linear computational complexity in both the number of samples and in the input dimension during training , without having to resort to the use of sparse grids or additive modeling of kernels . The main contribution of this work is in lifting the curse of dimensionality affecting deterministic Fourier features by modeling the weights as a low-rank tensor . This enables the efficient solution of large-scale and high-dimensional kernel learning problems . We derive an iterative algorithm under the exemplifying case of regularized squared loss and test it on regression and classification problems . 2 Related work . Fourier Features ( FF ) are a collection of data independent methods that leverage Bochner ’ s theorem [ 35 ] from harmonic analysis to approximate stationary kernels by numerical integration of their spectral density p ( · ) : k ( x , x′ ) Stationarity : = k ( x− x′ ) Bochner= ∫ p ( ω ) exp ( 〈iω , ( x− x′ ) 〉 ) dω FF≈ 〈z ( x ) , z ( x′ ) 〉 . ( 2 ) Rahimi and Recht [ 31 ] proposed to approximate the integral by Monte Carlo integration i.e . by drawing M random frequencies ω ∼ p ( · ) . In their work they show how the method converges uniformly at the Monte Carlo rate . See [ 23 ] for an overview of random FF . In order to achieve faster convergence and a lower sample complexity , a multitude of approaches that rely on deterministic numerical quadrature of the Fourier integral were developed . These methods generally consider product kernels whose spectral density factors in the frequency domain , which enables in turn to factor the Fourier integral . The resulting deterministic Fourier features are then the tensor product of D one-dimensional deterministic features . For example , in [ 5 ] the authors give an analysis of the sample complexity of features resulting from dense Gaussian Quadrature ( GQ ) . In [ 25 ] the authors present a similar quadrature approach for kernels whose spectral density factors over the dimensions of the inputs . They provide an explicit construction for their Quadrature Fourier Features relying on a dense Cartesian grid , and note that their method , as well as GQ , can attain exponentially decreasing uniform convergence bounds in the total number of basis functions per dimensions M̂ [ 25 , Theorem 1 ] . To avoid the curse of dimensionality , they make use of additive modeling of kernels . Variational Fourier Features ( VFF ) [ 16 ] are derived probabilistically in a one-dimensional setting for Matérn kernels by projecting a Gaussian Process onto a set of Fourier basis . An extension to multiple dimensions when considering product kernels is then achieved by taking the tensor product of the one-dimensional features , incurring however again in exponentially rising computational costs in D. In what they call HilbertGP [ 39 ] the authors diagonalize stationary kernels in terms of the eigenvalues and eigenfunctions of the Laplace operator with Dirichlet boundary conditions . Due to the multiplicative structure of the eigenfunctions of the Laplace operator , the complexity of the basis function increases exponentially in D when one considers product kernels . Like with other deterministic approximations , bounds on the uniform convergence error which decrease exponentially in M̂ can be achieved [ 39 , Theorem 8 ] . Tensor decompositions have been used extensively in machine learning in order to benefit from structure in data [ 38 ] , in particular to obtain an efficient representation of the model parameters . In the context of Gaussian Process Regression , the tensor product structure which arises when the inputs are located on a Cartesian grid have been exploited to speedup inference [ 12 ] . In their variational inference framework , [ 19 ] proposed to parameterize the posterior mean of a Gaussian Process by means of a tensor decomposition in order to exploit the tensor product structure which arises when interpolating the kernel function . In the context of neural networks , [ 27 ] proposed to represent the weights in deep neural networks as a tensor decomposition to speed up training . A similar approach was carried out in case of recurrent neural networks [ 48 ] [ 43 ] . Tensor decompositions have also been used to learn from simple features . In [ 45 ] multivariate functions are learned from a Fourier basis , which corresponds to assuming a uniform spectral density in ( 2 ) . Motivated by spin vectors in quantum mechanics , [ 40 ] consider trigonometric feature maps which they argue induce uninformative kernels . On a similar note , [ 28 ] and [ 1 ] leverage the tensor product structure of polynomial mappings to induce a polynomial kernel and consider model parameters in decomposed form . While these existing approaches are able to overcome the curse of dimensionality affecting tensor product feature maps by modeling the parameter tensor as a tensor network , they consider simple and empirical feature maps which induce uninformative kernels . In the following section we show how deterministic Fourier features can be used for supervised learning in both large and high-dimensional scenarios by assuming that the model weights are a low-rank tensor , thereby linking tensor decompositions with stationary product kernels . 3 Learning with deterministic Fourier features . 3.1 Notation . All tensor operations and mappings can be formulated over both the complex and the real field . For the remainder of this article real-valued mappings will be considered . Vectors and matrices are denoted by boldface lowercase and uppercase letters , respectively . Higher-order tensors are denoted by boldface calligraphic letters . Tensor entries are always written as lightface lowercase letters with a subscript index notation . For example , element i1 , i2 , i3 of the third-order tensorA ∈ RI1×I2×I3 is written as ai1i2i3 . The vectorization vec ( A ) of a tensorA ∈ RI1×I2×···×ID is the vector such that vec ( A ) i = ai1i2···iD , where the relationship between the linear index i and i1i2 . . . iD is given by i = i1 + D∑ d=2 ( id − 1 ) d−1∏ k=1 Ik . The Frobenius inner product between tensorsA ∈ RI1×I2×···×ID and B ∈ RI1×I2×···×ID is defined as 〈A , B〉F : = I1∑ i1=1 I2∑ i2=1 · · · ID∑ iD=1 ai1i2···iDbi1i2···iD = vec ( A ) Tvec ( B ) . Depending on the context , the symbol ⊗ either denotes the tensor outer product or the matrix Kronecker product . The Khatri-Rao productA~B of the matricesA ∈ RN×R andB ∈ RM×R is the NM ×R matrix that is obtained by taking the column-wise Kronecker product . The Hadamard ( element-wise ) matrix product ofA ∈ RN×R andB ∈ RN×R is denoted byA B . 3.2 The model . In this article we assume models of the form f ( x ) = 〈w , φ ( x ) 〉 , y = f ( x ) + . ( 3 ) Here x ∈ RD is the input vector , φ ( · ) : RD → H is a feature map , w ∈ H are the model weights , y ∈ R is the corresponding available observation corrupted by a zero mean i.i.d . noise . Learning such a kernelized model consists in finding as set of weights w such that : N∑ n=1 L ( yn , f ( xn ) ) + λ 〈w , w〉p , ( 4 ) is minimized . Here , L ( · , · ) : R× R→ R+ is a symmetric and positive loss function and λ 〈w , w〉p is a p-th norm regularization term . A variety of primal machine learning formulations arise when considering different combinations of loss functions and regularization terms . For example , considering p = 2 and squared loss leads to Kernel Ridge Regression [ 42 ] , while considering hinge loss leads to SVM [ 4 ] , and so on . Furthermore , applying the kernel trick when considering φ ( · ) recovers the nonparametric formulation which , depending on the choice of kernel , enables to perform inference using infinitely many basis functions with a computational complexity of O ( N3 ) . Considering instead a low-dimensional finite mapping z ( · ) : X → RM such as Random Fourier Features or the Nyström method leads a primal approach and to computational savings with a computational complexity of O ( NM2 ) for N M . However , as already discussed , these low-dimensional random mappings converge at the slow Monte Carlo rate , motivating our approach . In order to approximate the kernel function with faster and possibly exponential convergence , we consider deterministic , finite-dimensional features which are the tensor product of D vectors . For a given D-dimensional input point x we define the deterministic feature mapping Z ( · ) : RD → RM̂1 ⊗ RM̂2 ⊗ · · · ⊗ RM̂D as Z ( x ) = z ( 1 ) ( x1 ) ⊗ z ( 2 ) ( x2 ) · · · ⊗ z ( D ) ( xD ) , ( 5 ) where z ( d ) is a deterministic mapping applied to the d-th dimension . The dimension of the feature space is denoted M = ∏D d=1 M̂d . Without loss of generality it is from here on assumed that M̂1 = · · · = M̂D = M̂ such that M = M̂D . It is easy to verify by applying the kernel trick that the features of ( 5 ) yield product kernels . The mapping Z ( · ) encompasses for instance the mappings derived by quadrature [ 5 , 25 , 36 ] and by projection [ 16 , 39 ] . Note that these mappings cover many popular kernels such as the Gaussian and Matérn kernels . To give a concrete example , in the framework of A. Solin and S. Särkkä [ 39 , Equation 60 ] , for input data centered in a hyperbox X = [ −U1 , U1 ] ⊗ · · · ⊗ [ −UD , UD ] , the Gaussian kernel is approximated by means of D tensor products of M̂ weighted sinusoidal basis functions with frequencies lying on a harmonic scale such that : ( z ( d ) ( xd ) ) id = 1√ Ud p ( πid 2Ud ) sin ( πid ( xd + Ud ) 2Ud ) , id = 1 , . . . , M̂ . ( 6 ) Here p ( · ) is the spectral density of the Gaussian kernel with one-dimensional inputs , which is known in closed-form [ 33 , page 83 ] . This deterministic mapping then approximates the Gaussian kernel function [ 39 , Equation 59 ] such that k ( x , x′ ) ≈ 〈vec ( Z ( x ) ) , vec ( Z ( x′ ) ) 〉 = 〈Z ( x ) , Z ( x′ ) 〉F , and converges uniformly with exponentially decreasing bounds [ 39 , Theorem 8 ] . We therefore use Z ( · ) instead of φ ( · ) in ( 3 ) to obtain f ( x ) = 〈w , vec ( Z ( x ) ) 〉 = 〈W , Z ( x ) 〉F , ( 7 ) where the weight vector w has been reshaped into a D-dimensional tensorW ∈ RM̂1×M̂2×···×M̂D . Learning the exponential number of model parametersW in ( 7 ) under a hinge loss leads to Support Vector Machines , while considering a squared loss leads to the primal formulation of Kernel Ridge Regression , which we will consider as exemplifying case from here on : min W N∑ n=1 ( yn − 〈W , Z ( xn ) 〉F ) 2 + λ〈W , W〉F . ( 8 ) Since the number of elements M inW and Z grows exponentially in D , this primal approach is advantageous compared to the nonparametric dual approach only if M̂D N , limiting it to low-dimensional inputs . In order to lift this curse of dimensionality , we propose to represent and to learnW directly as a low-rank tensor decomposition . A low-rank tensor decomposition allows us to exploit redundancies inW in order to obtain a parsimonious model with a storage complexity that scales linearly in both M̂ and D. The low-rank structure will , as explicitly shown in the experiments , act as a form of regularization by limiting the total number of degrees of freedom of the model .

Summary: The authors propose a method for inference of large-scale gaussian process (GP) regressio problems with relatively high-dimensional inputs. The method achieves state of the art performance on real-world datasets using 2 innovations 1) formulation of the regression problem in the primal form with deterministic Fourier features, 2) reducing the dimensionality of the regression weights in the primal space by  defining them as a low-rank tensor. The authors describe the advantages and disadvantages of different choices for the tensor decomposition and demonstrate computational and inferential performance on both synthetic and real datasets. Of note is their ability to achieve comparable performance using a laptop computer to competing methods using clusters.