Random matrices in service of ML footprint: ternary random features with no performance loss

1 INTRODUCTION . Kernel methods are among the most powerful machine learning approaches with a wide range of successful applications ( Schölkopf & Smola , 2018 ) which , however , suffer from scalability issues in large-scale problems , due to their high space and time complexities ( with respect to the number of data n ) . To address this key limitation , a myriad of random features based kernel approximation techniques have been proposed ( Rahimi & Recht , 2008 ; Liu et al. , 2021a ) : random features methods randomly project the data to obtain low-dimensional nonlinear representations that approximate the original kernel features . This allows practitioners to apply them , with a large saving in both time and space , to various kernel-based downstream tasks such as kernel spectral clustering ( Von Luxburg , 2007 ) , kernel principal component analysis ( Schölkopf et al. , 1997 ) , kernel canonical correlation analysis ( Lai & Fyfe , 2000 ) , kernel ridge regression ( Vovk , 2013 ) , to name a few . A wide variety of these kernels can be written , for data points xi , xj ∈ Rp , in the form κ ( xi , xj ) = Ew [ σ ( wTxi ) σ ( wTxj ) ] ( 1 ) with w ∈ Rp having i.i.d . entries , which can be “ well approximated ” by a sample mean 1 m ∑m t=1 σ ( wTt xi ) σ ( wTt xj ) over m independent random features for m sufficiently large . For instance , taking σ ( x ) = [ cos ( x ) , sin ( x ) ] and w with i.i.d . standard Gaussian entries , one obtains the popular Random Fourier Features ( RFFs ) that approximate the Gaussian kernel ( and the Laplacian kernel for Cauchy distributed w with the same choice of σ ) ( Rahimi & Recht , 2008 ) ; for σ ( x ) = max ( x , 0 ) , one approximates the first order Arc-cosine kernel ; and the zeroth order Arc-cosine kernel ( Cho , 2012 ) with σ ( x ) = ( 1 + sign ( x ) ) /2 , etc . As shall be seen subsequently , ( random ) neural networks are , to a large extent , connected to kernel matrices of the form ( 1 ) . More specifically , the classification or regression performance at the output of random neural networks are functionals of random matrices that fall into the wide class of kernel random matrices . Perhaps more surprisingly , this connection still exists for deep neural networks which are ( i ) randomly initialized and ( ii ) trained with gradient descent , as testified by the recent works on neural tangent kernels ( Jacot et al. , 2018 ) , by considering the “ infinitely many neurons ” limit , that is , the limit where the network widths of all layers go to infinity simultaneously . This close connection between neural networks and kernels has triggered a renewed interest for the theoretical investigation of deep neural networks from various perspectives , including optimization ( Du et al. , 2019 ; Chizat et al. , 2019 ) , generalization ( Allen-Zhu et al. , 2018 ; Arora et al. , 2019 ; Bietti & Mairal , 2019 ) , and learning dynamics ( Lee et al. , 2019 ; Advani et al. , 2020 ; Liao & Couillet , 2018a ) . These works shed new light on the theoretical understanding of deep neural network models and specifically demonstrate the significance of studying networks with random weights and their associated kernels to assess the mechanisms underlying more elaborate deep networks . In this article , we consider the random feature kernel of the type ( 1 ) , which can also be seen as the limiting kernel of a single-hidden-layer neural network with a random first layer . By assuming a high-dimensional Gaussian Mixture Model ( GMM ) for the data { xi } ni=1 with xi ∈ Rp , we show that the centered kernel matrix1 K , P { κ ( xi , xj ) } ni , j=1P , P , In − 1 n 1n1 T n , ( 2 ) is asymptotically ( as n , p→∞ with p/n→ c ∈ ( 0 , ∞ ) ) equivalent , in a spectral sense , to another random kernel matrix K̃ which depends on the GMM data statistics and the generalized Gaussian moments E [ σ′ ( z ) ] , E [ σ′′ ( z ) ] of the activation function σ ( · ) , but is independent of the specific law of the i.i.d . entries of the random vector w , as long as they are normalized to have zero mean and unit variance . As such , one can design novel random features schemes with limiting kernels asymptotically equivalent to the original K. For instance , define κter ( xi , xj ) , Ewter [ σter ( xTi w ter ) σter ( xTj w ter ) ] ( 3 ) with wter ∈ Rp having i.i.d . entries taking value wteri = 0 ( with probability ) and value wteri ∈ { − ( 1− ) − 12 , ( 1− ) − 12 } ( 4 ) each with probability 1/2− /2 , where ∈ [ 0 , 1 ) represents the level of sparsity of w , and σter ( t ) = −1 · δt < s− + 1 · δt > s+ ( 5 ) for some thresholds s− < s+ chosen to match the generalized Gaussian moments E [ σ′ ( z ) ] , E [ σ′′ ( z ) ] of any σ function ( e.g. , ReLU , cos , sin ) widely used in random features or neural network contexts . The proposed Ternary Random Features ( TRFs , with limiting kernel matrices defined in ( 3 ) asymptotically “ matching ” any random features kernel matrices in a spectral sense ) have the computational advantage of being sparse and not requiring multiplications but only additions , as well as the storage advantage of being composed of only a finite set of words , e.g. , { −1 , 0 , 1 } for = 0 . Given the urgent need for environmentally-friendly , but still efficient , neural networks such as binary neural networks ( Hubara et al. , 2016 ; Lin et al. , 2015 ; Zhu et al. , 2016 ; Qin et al. , 2020 ; Hubara et al. , 2016 ) , pruned neural networks ( Liu et al. , 2015 ; Han et al. , 2015a ; b ) , weights-quantized neural networks ( Gupta et al. , 2015 ; Gong et al. , 2014 ) , we hope that our analysis will open a new door to a random matrix-improved framework of computationally efficient methods for machine learning and neural network models more generally . 1Left- and right-multiplying the kernel matrices by P is equivalent to centering the data in the kernel feature space , which is a common practice in kernel learning and plays a crucial role in multidimensional scaling ( Joseph & Myron , 1978 ) and kernel PCA ( Schlkopf et al. , 1998 ) . In the remainder of this paper , whenever we use kernel matrices , they are considered to have been centered in this way . 1.1 CONTRIBUTIONS . Our main results are summarized as follows . 1 . By considering a high-dimensional Gaussian mixture model for the data , we show ( Theorem 1 ) that for K defined in ( 2 ) , ‖K− K̃‖ → 0 as n , p→∞ , where K̃ is a random matrix independent of the law of w , and depends on the nonlinear σ ( · ) only via its generalized Gaussian moments E [ σ′ ( z ) ] and E [ σ′′ ( z ) ] for z ∼ N ( 0 , 1 ) . 2 . We exploit this result to propose a computationally efficient random features approach , called Ternary Random Features ( TRFs ) , with asymptotically the same limiting kernel as any random features-type kernel matrices K of the form ( 2 ) , while inducing no multiplication and b times less memory storage for its computation , with b the number of bits to store full precision values , e.g. , b = 32 in the case of a single-precision floating-point format . 3 . We provide empirical evidence on various random-features based algorithms , showing the computational and storage advantages of the proposed TRF method , while achieving competitive performances compared to state-of-the-art random features techniques . As a first telling example , in Figure 1 on the Cov-Type dataset from the UCI ML repository , TRFs achieve similar logistic regression performance as the LP-RFF method proposed by Zhang et al . ( 2019 ) , with 8 times less memory , and 32 or 16 times less memory than the Nyström approximation of the Gaussian kernel ( using full precision of 32 or 16 bits ) ( Williams & Seeger , 2001 ) ; see the shift in the x-axis ( memory ) of Figure 1 . 1.2 RELATED WORK . Random features kernel and random neural networks . Random features methods were first proposed to relieve the computational and storage burden of kernel methods in large-scale problems when the number of training samples n is large ( Schölkopf & Smola , 2018 ; Rahimi & Recht , 2008 ; Liu et al. , 2021a ) . For instance , Random Fourier Features can be used to approximate the popular Gaussian kernel , when the number of random features m is sufficiently large ( Rahimi & Recht , 2008 ) . Since ( deep ) modern neural networks are routinely trained with random initialization , random features method is also considered a stylish model to analyze neural networks ( Neal , 1996 ; Williams , 1997 ; Novak et al. , 2018 ; Matthews et al. , 2018 ; Lee et al. , 2017 ; Louart et al. , 2018 ) . In particular , by focusing on the large n , m regime , the analysis of such models led to the so-called double descent theory ( Advani et al. , 2020 ; Mei & Montanari , 2019 ; Liao et al. , 2020 ) for neural nets . Computationally efficient random features methods . In an effort to further reduce the computation and storage costs of random features models , various quantization and binarization methods were proposed ( Goemans & Williamson , 1995 ; Charikar , 2002 ; Li & Slawski , 2017 ; Li & Li , 2019 ; 2021 ; Agrawal et al. , 2019 ; Zhang et al. , 2019 ; Liao et al. , 2021 ; Couillet et al. , 2021 ) . More precisely , Agrawal et al . ( 2019 ) combined RFFs with a data-dependent feature selection approach to reduce the computational cost , while preserving the statistical guarantees of ( using the original set of ) RFFs . Zhang et al . ( 2019 ) proposed a low-precision approximation of RFFs to significantly reduce the storage while generalizing as well as full-precision RFFs . Li & Li ( 2021 ) designed quantization schemes of RFFs for arbitrary choice of the Gaussian kernel parameter . Our work improves these previous efforts by ( i ) considering a broader family of random features kernels beyond RFFs and by ( ii ) proposing the TRF approach that is both sparse and quantized , while asymptotically yielding the same limiting kernel spectral structure , and thus algorithmic performances ( Cortes et al. ) . Random matrix theory and neural networks . Random matrix theory ( RMT ) , as a powerful and flexible tool to investigate the ( asymptotic ) behavior of large-scale systems , is recently gaining popularity in the analysis of ( deep ) neural networks ( Martin & Mahoney , 2018 ; 2019 ) . In this respect , Pennington & Worah ( 2017 ) derived the eigenvalue distribution of the Conjugate Kernel ( CK ) in a single-hidden-layer random neural network model . This result was then generalized to a broader class of data distributions ( Louart et al. , 2018 ) and to a multi-layer scenario ( Benigni & Péché , 2019 ; Pastur , 2020 ) . Fan & Wang ( 2020 ) went beyond the general i.i.d . assumption ( on the entries of data vectors ) and studied the spectral properties of the CK and neural tangent kernel for data that are approximately “ pairwise orthogonal. ” Our work improves ( Fan & Wang , 2020 ) by studying the random features kernel for more structured GMM data , and is thus more adapted to machine learning applications such as classification . As far as the study of random features kernels under GMM data is concerned , the closest work to ours is ( Liao & Couillet , 2018b ) where the kernel matrix K defined in ( 2 ) is studied for GMM data , but only for a few specific activation functions and Gaussian w , see Footnote 5 below for a detailed account of the technical differences between this work and ( Liao & Couillet , 2018b ) . Here , we provide a universal result with respect to the ( much broader class of ) activation functions and random w , and propose a computation and storage efficient random features technique well tuned to match the performances of any commonly used random features kernels .

Summary: The authors present a resource-efficient approach for approximating kernels using random feature transformations. The random feature approach was introduced in a seminal paper by Rahimi and Recht, where each random feature is extracted from the input $x$ as $\sigma(\langle w, x\rangle )$, for a random (appropriately distributed) vector $w$ and an activation function (e.g. sinusiod) $\sigma$. In this paper, the goal is to save memory and time by restricting $w$ and $\sigma$ to take values in {$-1,0,1$}, meaning that $\sigma(\langle w, x\rangle )$:  1. can be evaluated only with additions/subtractions (no multiplications) and 2. only takes $1$ bit of storage.   The authors prove that under some conditions, their kernel estimate approaches the true kernel at the limit where the number of features and the number of data points are comparable and approach infinity. The assumptions are roughly: 1. The data follows a Gaussian mixture model, 2. the input features are approximately pairwise orthogonal, 3. the entries of $w$ are chosen i.i.d. and have bounded fourth moment, and 4. some boundedness assumptions on the generalized Gaussian moments of $\sigma$.  The theoretical result is accompanied by numerical experiments on various tasks like kernel ridge regression and SVM on both real and synthetic datasets. The results demonstrate a significant advantage of the authors' approach compared to previous quantized random features approches, both in terms of runtime and memory, without a significant drop in accuracy.