A Kernel Random Matrix-Based Approach for Sparse PCA

1 INTRODUCTION . Principal component analysis ( PCA ) is extensively used in data analysis and machine learning applications . It is a dimension reduction technique that aims to project a given dataset onto principal subspaces spanned by the leading eigenvectors of the sample covariance matrix ( Wold et al. , 1987 ) , which represent the principal modes of variance . Basically , the statistical interpretation of PCA lies in the fact that most of the variance in the data is captured by these modes . Consequently , PCA reduces the dimension of the feature space while keeping most of the information in the data . It is well-known ( Anderson , 1963 ) that PCA performs efficiently in the traditional data setting where the number of features is small and the number of samples is large . Consider a data matrix Y ∈ Rp×n consisting of n centered samples , each sample having p features . The standard PCA method requires the computation of the sample covariance matrix Ĉ = Y Y ᵀ/n and estimates the first principal components u1 , u2 , . . . ( i.e. , the successive dominant eigenvectors of C = E [ Y Y ᵀ/n ] ) by the ordered eigenvectors û1 , û2 , . . . of Ĉ . ( Johnstone & Lu , 2009 ) demonstrated that , in the high dimensional regime where n , p→∞ with p/n→ c > 0 , the principal component û1 estimated by standard PCA is inconsistent . Essentially , if p/n 9 0 then ‖û1 − u1‖2 9 0 in the high-dimensional asymptotic regime . This phenomenon is well investigated within the field of random matrix theory for covariance models of the form Ĉ = 1nΣ 1/2 p XXᵀΣ 1/2 p , where Σp is a positive semi-definite matrix and X is a p × n matrix with random i.i.d . entries . One of the main results from random matrix theory concerns the so-called spiked models , where Σp is a low-rank perturbation of the identity matrix , namely Σp = Ip + ∑k i=1 ωiuiu ᵀ i with k fixed with respect to p , n. ( Baik et al. , 2005 ) and ( Paul , 2007 ) notably exhibited a phase transition phenomenon : as p/n→ c , if ωi < √ c the estimated principal component ûi using standard PCA is ( almost surely ) asymptotically orthogonal to the true principal component ui ( i.e. , û ᵀ i ui → 0 ) ; on the other hand , if ωi > √ c , lim infn |ûᵀi ui| > 0 . This phase transition phenomenon has attracted recently much attention within the random matrix community ( Benaych-Georges & Nadakuditi , 2011 ; Capitaine et al. , 2009 ; Féral & Péché , 2007 ; Knowles & Yin , 2013 ) . The inconsistency of standard PCA in high dimensions motivated the idea to look for more structural information on the principal components . In particular , considering that the principal components ∗Couillet ’ s work is supported by the GSTATS UGA IDEX Datascience chair and the ANR RMT4GRAPH ( ANR-14-CE28-0006 ) . are sparse in an appropriate basis ( e.g. , in the wavelet domain ) , a large body of works have emerged and proposed improved PCA approaches that account for sparsity . One of the most consistent sparse PCA methods in the literature is the covariance thresholding ( CT ) algorithm ( Krauthgamer et al. , 2015 ) . Based on the intuition that the small entries of the empirical covariance matrix Ĉ induce noise in its principal components , this method consists in applying the popular soft-thresholding function ( with threshold τ > 0 ) ; soft ( · ; τ ) : t 7→ sign ( t ) · ( |t| − τ ) + , entry-wise to the empirical covariance matrix Ĉ and performing PCA on the resulting matrix . ( Deshpande & Montanari , 2014 ; 2016 ) have theoretically demonstrated that the covariance thresholding algorithm recovers the sought-for principal components with high probability under controlled growth rates between p , n and the sparsity level . In this paper , we show that the soft-thresholding method in fact falls within a broader class of kernel-based1 PCA algorithms that are particularly suited to sparse PCA recovery . This method consists in considering the matrix f ( Ĉ ) instead of Ĉ where f is a function applied entry-wise . By imposing some constraints on f , most importantly that f ′ ( 0 ) = f ′′ ( 0 ) = 0 , sparse PCA can be performed with provably high accuracy for sufficiently large n , p. The rest of the paper is organized as follows . In Section 2 , we present the related work on sparse PCA . We recall some necessary concentration of measure tools and notions about sparse matrices in Section 3 . Our main theoretical results are then provided in Section 4 . Section 5 discusses the practical aspects and provides experimental results . Section 6 concludes the article . Notation : In following , the notation [ n ] denotes the set { 1 , . . . , n } , bac denotes the integer part of a . Given a vector x ∈ Rn , the ` 2-norm of x is denoted as ‖x‖2 = ∑n i=1 x 2 i . Given an n × n matrix M , Mij or [ M ] ij denote the entry of the matrix M at line i and column j . [ M ] · , j denotes the j-th column vector of M and [ M ] i , · its i-th line vector . The Frobenius norm ‖M‖F of the matrix M is defined as ‖M‖2F = ∑n i , j=1M 2 ij , and the operator norm ‖M‖op of M is defined as ‖M‖op = max‖x‖=1 ‖Mx‖ . Finally , denotes the Hadamard product , with [ M N ] ij = MijNij . 2 RELATED WORK . The problem of sparse PCA has been tackled with a large range of techniques . Mainly , three classes of approaches emerge in the literature . Most popular techniques are optimization-based algorithms ( d ’ Aspremont et al. , 2005 ; Moghaddam et al. , 2006 ; Zass & Shashua , 2007 ; Zou et al. , 2006 ; Wright et al. , 2009 ) , where the idea is to see the problem of sparse PCA through an optimization perspective , and to propose methods to solve the latter by either considering a different formulation – e.g . semi-definite programming ( SDP ) or convex relaxations – or adding penalties to the original optimization problem such as a LASSO regularization . The second class of approaches covers matrix decomposition-based techniques ( Asteris et al. , 2014 ; Papailiopoulos et al. , 2013 ; Shen & Huang , 2008 ) , where sparse principal components are extracted through solving a low rank matrix approximation problem based on Singular Value Decomposition . Finally , most consistent sparse PCA methods adopt thresholding-based approaches : initial heuristics used factor rotation techniques and thresholding of eigenvectors to obtain sparsity ( Cadima & Jolliffe , 1995 ) . Based on the well-known power method , ( Yuan & Zhang , 2013 ) introduced an efficient sparse PCA approximation to obtain the exact level of required sparsity , by truncating to zero the principal components iteratively except for their largest entries . A step further , under a spiked covariance model ( see Section 4.1 ) , ( Ma et al. , 2013 ) proposed a very efficient iterative thresholding approach for estimating principal subspaces in the sparse setting . Similarly , assuming a single-spike model , ( Krauthgamer et al. , 2015 ) proved that , when the sparsity level s ≥ Ω ( √n ) , a standard SDP approach can not recover consistently the sparse spike ; in particular , the authors presented empirical results suggesting that for s = O ( √n ) , recovery is possible by a simple covariance thresholding algorithm . More recently , ( Deshpande & Montanari , 2016 ) analyzed and theoretically proved , under a spiked model , that indeed the covariance thresholding algorithm ( Krauthgamer et al. , 2015 ) succeeds with high probability under controlled growth rates between p , n and s. In this work , while restricting ourselves to a setting where p and n grow at a controlled joint rate , we provide an elementary argument , based on a matrix-wise Taylor expansion controlled through a concentration of measure approach , that generalizes the CT method to a large family of kernel-based 1We use the kernel-based terminology to highlight that our work falls within the framework of kernel random matrices and should not be confused with the standard kernel PCA . methods , by means of a kernel random matrix approach ( El Karoui , 2010b ; a ) . Concretely , we study kernel random matrices of the form f ( Y Y ᵀ/n ) where Y = Σ1/2p X and X is a random matrix with N ( 0 , 1 ) i.i.d . entries . ( El Karoui , 2010b ) studied kernel matrices of the form f ( Y ᵀY/n ) ( i.e. , the socalled inner-product kernel matrices ) , which is equivalent to the case Σp = Ip when considering the form f ( Y Y ᵀ/n ) . In particular , we elaborate from El Karoui ’ s study by Taylor expanding f ( Y Y ᵀ/n ) in the vicinity of Σp entry-wise and controlling the resulting matrices via concentration arguments . 3 PRELIMINARIES . Before introducing our model setting we recall some definitions and notions of the concentration of measure theory ( Ledoux , 2005 ) that are at the heart of our main results . Furthermore , we recall a definition , introduced by ( El Karoui , 2008 ) , of sparse matrices in the large-dimensional context that will also be exploited in this paper . 3.1 CONCENTRATION OF MEASURE RESULTS . We start by a definition of the notion of concentration for a real random variable . Definition 1 ( Concentration of a Random Variable ) . Given a function δ : R+ → R+ , a random variable Z is said to be δ-concentrated ( around its mean ) and we write Z ∈ δ , if for all t > 0 , P { |Z − EZ| ≥ t } ≤ δ ( t ) . In particular , Z is said to be normally ( resp. , exponentially ) concentrated when δ ( t ) = Ce−c t 2 ( resp. , δ ( t ) = Ce−c t ) and we write Z ∈ CN ( c · ) ( resp. , Z ∈ CE ( c · ) ) , where C , c > 0 are some absolute constants . In particular , δ-concentration remains stable by application of Lipschitz functions : Proposition 1 ( Concentration of Lipschitz Functions ) . Given a λ-Lipschitz function f : R→ R and a concentrated random variable Z ∈ δ , we have f ( Z ) ∈ δ ( ·/λ ) . As a consequence , linear combinations of δ-concentrated random variables remain concentrated . However , products of δ-concentrated random variables are more technical to handle , but we still have the following proposition in the case of normally concentrated random variables and which will be essential in this article . Proposition 2 ( Square of Normally Concentrated Random Variables ) . Given Z ∈ CN ( c · ) , the random variable Z2 is exp-normally concentrated , precisely Z2 ∈ KCE ( c 2 · ) +KCN ( c 16E [ Z ] 2 · ) , ( 1 ) where KC > 0 is a constant depending only on C. The extension of the notion of concentration to random vectors Z ∈ Rp demands that Rp → R Lipschitz functions are concentrated random variables . Definition 2 ( Concentration of a Random Vector ) . Given a function δ : R+ → R+ and a normal space ( E , ‖.‖ ) , a random vector Z ∈ E is said to be δ-concentrated if for any 1-Lipschitz function f : E → R , the random variable f ( Z ) is δ-concentrated . We note again Z ∈ δ . In particular , we have the concentration of Gaussian random vectors in the sense of Definition 2 in the following proposition ( Tao , 2012 , Theorem 2.1.12 ) . Proposition 3 ( Normal Concentration of Gaussian Random Vectors ) . A Gaussian vector Z ∈ Rp , with independent and identically distributed N ( 0 , 1 ) entries , is normally concentrated independently on the dimension p. Furthermore , Z ∈ 2N ( ·/2 ) . Remark 1 . According to Definition 2 , given a Lipschitz application F : Rp → Rq for q ∈ N∗ , Proposition 3 provides the normal concentration of all the random vectors F ( Z ) . In particular , note that our results are extensible to this family of vectors and random vectors with independent entries . 3.2 ε-SPARSE MATRICES When considering a large-dimensional random matrix setting , the notion of sparsity for such matrices is particularly attached to the choice of the matrix norm.2 ( El Karoui , 2008 ) introduced a definition ( ε-sparsity ) for sparsity of matrices that is compatible with spectral analysis , and specifically adapted to the operator norm . The ε-sparsity definition requires some notions from graph theory that we present in the following : to each p× p symmetric matrix M , we define its corresponding adjacency matrix as A ( M ) = { 1Mij 6=0 } pi , j=1 , which corresponds to a graph Gp with p vertices . A walk is said to be closed on this graph if it starts and finishes at the same vertex and the number of edges traversed by a walk defines the length of this walk . Denote Cp ( k ) the set of closed walks of length k on Gp . Definition 3 ( ε-sparse matrices ( El Karoui , 2008 , Definition 1 ) ) . A sequence of covariance matrices { Σp } ∞p=1 is said to be ε-sparse if the sequence of their associated graphs { Gp } ∞p=1 satisfies , for all k ∈ 2N , |Cp ( k ) | ≤ Ck pε ( k−1 ) +1 where ε ∈ [ 0 , 1 ] , Ck > 0 independent of p and |S| denotes the cardinality of the set S. The ε-sparsity is both useful and convenient to out study for the following reasons : 1 ) it is adapted to the analysis of the operator norm of large sparse matrices ( as we give concentration results on the operator norm ) ; 2 ) it is also more general than other sparsity notions such as in ( Bickel & Levina , 2008 ) . In the latter , the authors developed a natural permutation-invariant notion of sparsity which is more specific than Definition 3 as pointed out in the introduction of their article . Furthermore , note that both sparsity notions ( Definition 3 and the one in ( Bickel & Levina , 2008 ) ) provide equivalent bounds for ε < 12 and when considering the large dimensional p ∼ n setting ( see subsection 2.4 in ( Bickel & Levina , 2008 ) ) ; this is precisely the setting considered in Corollary 2 introduced subsequently ( cf . µ > 0 ) . Remark 2 . As Definition 3 is based on a graph defined by its corresponding adjacency matrix , we have the following property : given an ε-sparse matrix M and a function f such that f ( 0 ) = 0 and f ( x ) 6=x 6=0 0 , the matrix f ( M ) , resulting from the application of f entry-wise to M , remains ε-sparse ; this is simply a consequence of A ( M ) = A ( f ( M ) ) .

Summary: This paper discusses the sparse PCA problem from the random matrix perspective. First, it establishes a theorem that connect a f(sample covariance) to f(actual covariance) in Theorem 1, and shows that when f if three-times continuous differentiable,  f(sample covariance) can be written in terms of actual covariance and $1/n XX^T-I$. Then, based on this theorem, it shows that if f'(0)=f''(0)=0, then f(sample covariance) is well approximated by f(actual covariance).