Sparse Quantized Spectral Clustering

1 INTRODUCTION . Sparsifying , quantizing , and/or performing other entry-wise nonlinear operations on large matrices can have many benefits . Historically , this has been used to develop iterative algorithms for core numerical linear algebra problems ( Achlioptas & McSherry , 2007 ; Drineas & Zouzias , 2011 ) . More recently , this has been used to design better neural network models ( Srivastava et al. , 2014 ; Dong et al. , 2019 ; Shen et al. , 2020 ) . A concrete example , amenable to theoretical analysis and ubiquitous in practice , is provided by spectral clustering , which can be solved by retrieving the dominant eigenvectors of XTX , for X = [ x1 , . . . , xn ] ∈ Rp×n a large data matrix ( Von Luxburg , 2007 ) . When the amount of data n is large , the Gram “ kernel ” matrix XTX can be enormous , impractical even to form and leading to computationally unaffordable algorithms . For instance , Lanczos iteration that operates through repeated matrix-vector multiplication suffers from an O ( n2 ) complexity ( Golub & Loan , 2013 ) and quickly becomes burdensome . One approach to overcoming this limitation is simple subsampling : dividing X into subsamples of size εn , for some ε ∈ ( 0 , 1 ) , on which one performs parallel computation , and then recombining . This leads to computational gain , but at the cost of degraded performance , since each data point xi looses the cumulative effect of comparing to the whole dataset . An alternative cost-reduction procedure consists in uniformly randomly “ zeroing-out ” entries from the whole matrix XTX , resulting in a sparse matrix with only an ε fraction of nonzero entries . For spectral clustering , by focusing on the eigenspectrum of the “ zeroed-out ” matrix , Zarrouk et al . ( 2020 ) showed that the same computational gain can be achieved at the cost of a much less degraded performance : for n/p rather large , almost no degradation is observed down to very small values of ε ( e.g. , ε ≈ 2 % for n/p & 100 ) . Previous efforts showed that it is often advantageous to perform sparsification/quantization in a nonuniform manner , rather than uniformly ( Achlioptas & McSherry , 2007 ; Drineas & Zouzias , 2011 ) . The focus there , however , is often on ( non-asymptotic bounds of ) the approximation error between the original and the sparsified/quantized matrices . This , however , does not provide a direct access to the actual performance for spectral clustering or other downstream tasks of interest , e.g. , since the top eigenvectors are known to exhibit a phase transition phenomenon ( Baik et al. , 2005 ; Saade et al. , 2014 ) . That is , they can behave very differently from those of the original matrix , even if the matrix after treatment is close in operator or Frobenius norm to the original matrix . Here , we focus on a precise characterization of the eigenstructure of XTX after entry-wise nonlinear transformation such as sparsification or quantization , in the large n , p regime , by performing simultaneously non-uniform sparsification and/or quantization ( down to binarization ) . We consider a simple mixture data model with x ∼ N ( ±µ , Ip ) and let K ≡ f ( XTX/ √ p ) / √ p , where f is an entry-wise thresholding/quantization operator ( thereby zeroing-out/quantizing entries of XTX ) ; and we prove that this leads to significantly improved performances , with the same computational cost , in spectral clustering as uniform sparsification , but for a much reduced cost in storage induced by quantization . The only ( non-negligible ) additional cost arises from the extra need for evaluating each entry of XTX . Our main technical contribution ( of independent interest , e.g. , for those interested in entry-wise nonlinear transformations of feature matrices ) consists in using random matrix theory ( RMT ) to derive the large n , p asymptotics of the eigenspectrum of K = f ( XTX/ √ p ) / √ p for a wide range of functions f , and then comparing to previously-established results for uniform subsampling and sparsification in ( Zarrouk et al. , 2020 ) . Experiments on real-world data further corroborate our findings . Our main contributions are the following . 1 . We derive the limiting eigenvalue distribution of K as n , p→∞ ( Theorem 1 ) , and we identify : ( a ) the existence of non-informative and isolated eigenvectors of K for some f ( Corollary 1 ) ; ( b ) in the absence of such eigenvectors , a phase transition in the dominant eigenvalue-eigenvector ( λ̂ , v̂ ) pair ( Corollary 2 ) : if the signal-to-noise ratio ( SNR ) ‖µ‖2 of the data exceeds a certain threshold γ , then λ̂ becomes isolated from the main bulk ( Von Luxburg , 2007 ; Joseph & Yu , 2016 ; Baik et al. , 2005 ) and v̂ contains data class-structure information exploitable for clustering ; if not , then v̂ contains only noise and is asymptotically orthogonal to the class-label vector . 2 . Letting f be a sparsification , quantization , or binarization operator , we show : ( a ) a selective non-uniform sparsification operator , such that XTX can be drastically sparsified with very little degradation in clustering performance ( Proposition 1 and Section 4.2 ) , which significantly outperforms the random uniform sparsification scheme in ( Zarrouk et al. , 2020 ) ; ( b ) for a given matrix storage budget ( i.e. , fixed number of bits to store K ) , an optimal design of the quantization/binarization operators ( Proposition 2 and Section 4.3 ) , the performances of which are compared against the original XTX and its sparsified but not quantized version . For spectral clustering , the surprisingly small performance drop , accompanied by a huge reduction in computational cost , contributes to improved algorithms for large-scale problems . More generally , our proposed analysis sheds light on the effect of entry-wise nonlinear transformations on the eigenspectra of data/feature matrices . Thus , looking forward ( and perhaps more importantly , given the use of nonlinear transformations in designing modern neural network models as well as the recent interest in applying RMT to neural network analyses ( Dobriban et al. , 2018 ; Li & Nguyen , 2018 ; Seddik et al. , 2018 ; Jacot et al. , 2019 ; Liu & Dobriban , 2019 ) ) , we expect that our analysis opens the door to improved analysis of computationally efficient methods for large dimensional machine learning and neural network models more generally . 2 SYSTEM MODEL AND PRELIMINARIES . Basic setup . Let x1 , . . . , xn ∈ Rp be independently drawn ( not necessarily uniformly ) from a two-class mixture of C1 and C2 with C1 : xi = −µ+ zi , C2 : xi = +µ+ zi ( 1 ) with zi ∈ Rp having i.i.d . zero-mean , unit-variance , κ-kurtosis , sub-exponential entries , µ ∈ Rp such that ‖µ‖2 → ρ ≥ 0 as p → ∞ , and v ∈ { ±1 } n with [ v ] i = −1 for xi ∈ C1 and +1 for xi ∈ C2.1 The data matrix X = [ x1 , . . . , xn ] ∈ Rp×n can be compactly written as X = Z + µvT 1The norm ‖ · ‖ is the Euclidean norm for vectors and the operator norm for matrices . for Z = [ z1 , . . . , zn ] so that ‖v‖ = √ n and both Z , µvT have operator norm of order O ( √ p ) in the n ∼ p regime . In this setting , the Gram ( or linear kernel ) matrix XTX achieves optimal clustering performance on the mixture model ( 1 ) ; see Remark 1 below . However , it consists of a dense n× n matrix , which becomes quickly expensive to store or to perform computation on , as n increases . Thus , we consider instead the following entry-wise nonlinear transformation of XTX : K = { δi 6=jf ( x T i xj/ √ p ) / √ p } n i , j=1 ( 2 ) for f : R → R satisfying some regularity conditions ( see Assumption 1 below ) , where δi 6=j equals 1 for i 6= j and equals 0 otherwise . The diagonal elements f ( xTi xi/ √ p ) ( i ) bring no additional information for clustering and ( ii ) do not scale properly for p large ( xTi xi/ √ p = O ( √ p ) instead of O ( 1 ) ) . Thus , following ( El Karoui , 2010 ; Cheng & Singer , 2013 ) , they are discarded . Most of our technical results hold for rather generic functions f , e.g. , those of interest beyond sparse quantized spectral clustering , but we are particularly interested in f with nontrivial numerical properties ( e.g. , promoting quantization and sparsity ) : Sparsification : f1 ( t ) = t · 1|t| > √2s ( 3 ) Quantization : f2 ( t ) = 22−M ( bt · 2M−2/ √ 2sc+ 1/2 ) · 1|t|≤√2s + sign ( t ) · 1|t| > √2s ( 4 ) Binarization : f3 ( t ) = sign ( t ) · 1|t| > √2s . ( 5 ) Here , s ≥ 0 is some truncation threshold , and M ≥ 2 is a number of information bits.2 The visual representations of these fs are given in Figure 1- ( left ) . For f3 , taking s→ 0 leads to the sign function sign ( t ) . In terms of storage , the quantization f2 consumes 2M−2 +1 bits per nonzero entry , while the binarization f3 takes values in { ±1 , 0 } and thus consumes 1 bit per nonzero entry . Random matrix theory . To provide a precise description of the eigenspectrum of K for the nonlinear f of interest , to be used in the context of spectral clustering , we will provide a large dimensional characterization for the resolvent of K , defined for z ∈ C \ R+ as Q ( z ) ≡ ( K− zIn ) −1 . ( 6 ) This matrix , which plays a central role in RMT analysis ( Bai & Silverstein , 2010 ) , will be used in two primary ways . First , the normalized trace 1n tr Q ( z ) is the so-called Stieltjes transform of the eigenvalue distribution of K , from which the eigenvalue distribution can be recovered , and be further used to characterize the phase transition beyond which spectral clustering becomes theoretically possible ( Corollary 2 ) . Second , for ( λ̂ , v̂ ) , an “ isolated ” eigenvalue-eigenvector pair of K , and a ∈ Rn , a deterministic vector , by Cauchy ’ s integral formula , the “ angle ” between v̂ and a is given by |v̂Ta|2 = − 12πı ∮ Γ ( λ̂ ) aTQ ( z ) a dz , where Γ ( λ̂ ) is a positively oriented contour surrounding λ̂ only . Letting a = v , this will be exploited to characterize the spectral clustering error rate ( Proposition 1 ) . From a technical perspective , unlike linear random matrix models , K ( and thus Q ( z ) ) involves nonlinear dependence between its entries . To break this difficulty , following the ideas of ( Cheng & Singer , 2013 ) , we exploit the fact that , by the central limit theorem , zTi zj/ √ p → N ( 0 , 1 ) in distribution as p → ∞ . As such , up to µvT , which is treated separately with a perturbation argument , the entries of K asymptotically behave like a family of dependent standard Gaussian variables to which f is applied . Expanding f in a series of orthogonal polynomials with respect to the Gaussian measure allows for “ unwrapping ” this dependence . A few words on the theory of orthogonal polynomials ( Andrews et al. , 1999 ) are thus convenient to pursue our analysis . Orthogonal polynomial framework . For a probability measure µ , let { Pl ( x ) , l ≥ 0 } be the orthonormal polynomials with respect to 〈f , g〉 ≡ ∫ fgdµ obtained by the Gram-Schmidt procedure on the monomials { 1 , x , x2 , . . . } , such that P0 ( x ) = 1 , Pl is of degree l and 〈Pl1 , Pl2〉 = δl1−l2 . By the Riesz-Fischer theorem ( Rudin , 1964 , Theorem 11.43 ) , for any function f ∈ L2 ( µ ) , the set of square-integrable functions with respect to 〈· , ·〉 , one can formally expand f as f ( x ) ∼ ∞∑ l=0 alPl ( x ) , al = ∫ f ( x ) Pl ( x ) µ ( dx ) ( 7 ) 2Also , here , b·c denotes the floor function , while erf ( x ) = 2√ π ∫ x 0 e−t 2 dt and erfc ( x ) = 1−erf ( x ) denotes the error function and complementary error function , respectively . where “ f ∼ ∑∞ l=0 alPl ” indicates that ‖f − ∑L l=0 alPl‖ → 0 as L→∞ with ‖f‖2 = 〈f , f〉 . To investigate the asymptotic behavior of K , as n , p→∞ , we make the following assumption on f . Assumption 1 ( Square-integrable in Gauss space ) . Let ξp = zTi zj/ √ p and Pl , p ( x ) be the orthonor- mal polynomials with respect to the measure µp of ξp . For f ∈ L2 ( µp ) , f ( x ) ∼ ∑∞ l=0 al , pPl , p ( x ) with al , p in ( 7 ) such that ( i ) ∑∞ l=0 al , pPl , p ( x ) µp ( dx ) converges in L 2 ( µp ) to f ( x ) uniformly over large p ; and ( ii ) as p→∞ , ∑∞ l=1 a 2 l , p → ν and , for l = 0 , 1 , 2 , al , p → al converges with a0 = 0 . Since ξp → N ( 0 , 1 ) in distribution , the parameters a0 , a1 , a2 and ν are simply moments of the standard Gaussian measure involving f . More precisely , for ξ ∼ N ( 0 , 1 ) , a0 = E [ f ( ξ ) ] , a1 = E [ ξf ( ξ ) ] , √ 2a2 = E [ ξ2f ( ξ ) ] − a0 , ν = E [ f2 ( ξ ) − a20 ] ≥ a21 + a22 . ( 8 ) Imposing the condition a0 = 0 simply discards the non-informative rank-one matrix a01n1Tn/ √ p from K. The three parameters ( a1 , a2 , ν ) are of crucial significance in determining the spectral behavior of K ( see Theorem 1 below ) .The sparse f1 , quantized f2 , and binary f3 of our primary interest all satisfy Assumption 1 ( counterexample exists though , for example f ( x ) = eP ( x ) for polynomial P ( x ) of degree greater than two ) , with their corresponding a2 = 0 ( as we shall see in Corollary 1 below , this is important for the spectral clustering use case ) and a1 , ν given in Figure 1- ( right ) . With these preliminary comments , we are in position to present our main technical results .

Summary: This work considers the effect of sparsification, quantization and non-linear transormations on the spectrum of a random matrix with respect to performance in downstream applications like clustering. Eigen decomposition of large matrices is computationally very expensive and therefore methods like sparsification is used in order to reduce the computational complexity, while adding some amount of error. Therefore, theoretical bounds which quantize the amount of deviation because of such processing steps is essential to understand how much/little degraded the performance of machine learning applications are because of this step. This work considers the important case of spectral clustering with a 2 class mixture of subexponential models using