Approximation Algorithms for Sparse Principal Component Analysis

1 INTRODUCTION . Principal Component Analysis ( PCA ) and the related Singular Value Decomposition ( SVD ) are fundamental data analysis and dimension reduction tools in a wide range of areas including machine learning , multivariate statistics and many others . They return a set of orthogonal vectors of decreasing importance that are often interpreted as fundamental latent factors that underlie the observed data . Even though the vectors returned by PCA and SVD have strong optimality properties , they are notoriously difficult to interpret in terms of the underlying processes generating the data ( Mahoney & Drineas , 2009 ) , since they are linear combinations of all available data points or all available features . The concept of Sparse Principal Components Analysis ( SPCA ) was introduced in the seminal work of ( d ’ Aspremont et al. , 2007 ) , where sparsity constraints were enforced on the singular vectors in order to improve interpretability . A prominent example where sparsity improves interpretability is document analysis , where sparse principal components can be mapped to specific topics by inspecting the ( few ) keywords in their support ( d ’ Aspremont et al. , 2007 ; Mahoney & Drineas , 2009 ; Papailiopoulos et al. , 2013 ) . Formally , given a positive semidefinite ( PSD ) matrix A ∈ Rn×n , SPCA can be defined as follows:1 Z∗ = maxx∈Rn , ‖x‖2≤1 x > Ax , subject to ‖x‖0 ≤ k. ( 1 ) In the above formulation , A is a covariance matrix representing , for example , all pairwise feature or object similarities for an underlying data matrix . Therefore , SPCA can be applied for either the object or feature space of the data matrix , while the parameter k controls the sparsity of the resulting vector and is part of the input . Let x∗ denote a vector that achieves the optimal value Z∗ in the above formulation . Then intuitively , the optimization problem of eqn . ( 1 ) seeks a sparse , unit norm vector x∗ that maximizes the data variance . It is well-known that solving the above optimization problem is NP-hard ( Moghaddam et al. , 2006a ) and that its hardness is due to the sparsity constraint . Indeed , if the sparsity constraint was removed , then the resulting optimization problem can be easily solved by computing the top left or right singular vector of A and its maximal value Z∗ is equal to the top singular value of A . Notation . We use bold letters to denote matrices and vectors . For a matrix A ∈ Rn×n , we denote its ( i , j ) -th entry by Ai , j ; its i-th row by Ai∗ and its j-th column by A∗j ; its 2-norm by 1Recall that the p-th power of the ` p norm of a vector x ∈ Rn is defined as ‖x‖pp = ∑n i=1 |xi| p for 0 < p < ∞ . For p = 0 , ‖x‖0 is a semi-norm denoting the number of non-zero entries of x . ‖A‖2 = maxx∈Rn , ‖x‖2=1 ‖Ax‖2 ; and its ( squared ) Frobenius norm by ‖A‖ 2 F = ∑ i , j A 2 i , j . We use the notation A 0 to denote that the matrix A is symmetric positive semidefinite ( PSD ) and Tr ( A ) = ∑ iAi , i to denote its trace , which is also equal to the sum of its singular values . Given a PSD matrix A ∈ Rn×n , its Singular Value Decomposition is given by A = UΣUT , where U is the matrix of left/right singular vectors and Σ is the diagonal matrix of singular values . 1.1 OUR CONTRIBUTIONS . We present three algorithms for SPCA and associated quality-of-approximation results ( Theorems 2.2 , 3.1 , and 4.1 ) . All three algorithms are simple , intuitive , and run in O ( n3.5 ) or less time . They return a vector that is provably sparse and , when applied to the input covariance matrix A , provably captures a fraction of the optimal solution Z∗ . We note that in all three algorithms , the output vector has a sparsity that depends on k ( the target sparsity of the original SPCA problem of eqn . ( 1 ) ) and ( an accuracy parameter between zero and one ) . The first algorithm is based on randomized , approximate matrix multiplication : it randomly ( but non-uniformly ) selects a subset of O ( k/ 2 ) columns of A1/2 ( the square root of the PSD matrix A ) and computes its top right singular vector . The output of this algorithm is precisely this singular vector , padded with zeros to become a vector in Rn . It turns out that this simple algorithm , which , surprisingly has not been analyzed in prior work , returns an O ( k/ 2 ) sparse vector y ∈ Rn that satisfies ( with constant probability that can be amplified as desired , see Section 2 for details ) : y > Ay ≥ 1 2 Z∗ − √ Z∗ · √ Tr ( A ) k . Notice that the above bound depends on both Z∗ and it square root and therefore is not a relative error bound . The second term scales as a function of the trace of A divided by k , which depends on the properties of the matrix A and the target sparsity . The second algorithm is a deterministic thresholding scheme . It computes a small number of the top singular vectors of the matrix A and then applies a deterministic thresholding scheme on those singular vectors to ( eventually ) construct a sparse vector z ∈ Rn that satisfies z > Az ≥ ( 1/2 ) Z∗− ( 3/2 ) Tr ( A ) . Our analysis provides unconditional guarantees for the accuracy of the solution of this simple thresholding scheme . To the best of our knowledge , no such analyses have appeared in prior work ( see Section 1.2 for details ) . The error bound of the second algorithm is weaker than the one provided by the first algorithm , but the second algorithm is deterministic and does not need to compute the square root ( i.e. , all singular vectors and singular values ) of the matrix A . Our third algorithm provides novel bounds for the following standard convex relaxation of the problem of eqn . ( 1 ) . max Z∈Rn×n , Z 0 Tr ( AZ ) s.t . Tr ( Z ) ≤ 1 and ∑ |Zi , j | ≤ k. ( 2 ) It is well-known that the optimal solution to eqn . ( 2 ) is at least the optimal solution to eqn . ( 1 ) . We present a novel , two-step rounding scheme that converts the optimal solution matrix Z ∈ Rn×n to a vector z ∈ Rn that has expected sparsity2 Õ ( k2/ 2 ) and satisfies z > Az ≥ γZ ( 1− ) · Z∗ − . Here , γZ is a constant that precisely depends on the top singular value of Z , the condition number of Z , and the extent to which the SDP relaxation of eqn . ( 2 ) is able to capture the original problem ( see Theorem 4.1 and the following discussion for details ) . To the best of our knowledge , this is the first analysis of a rounding scheme for the convex relaxation of eqn . ( 2 ) that does not assume a specific model for the covariance matrix A . Applications to Sparse Kernel PCA . Our algorithms have immediate applications to sparse kernel PCA ( SKPCA ) , where the input matrix A ∈ Rn×n is instead implicitly given as a kernel matrix whose entry ( i , j ) is the value k ( i , j ) : = 〈φ ( Xi∗ ) , φ ( Xj∗ ) 〉 for some kernel function φ that implicitly maps an observation vector into some high-dimensional feature space . Although A is not explicit , 2For simplicity of presentation and following the lines of ( Fountoulakis et al. , 2017 ) , we assume that the rows and columns of the matrix A have unit norm ; this assumption was not necessary for the previous two algorithms and can be removed as in ( Fountoulakis et al. , 2017 ) . We are also hiding a poly-logarithmic factor for simplicity , hence the Õ ( · ) notation . See Theorem 4.1 for a detailed statement . we can query all O ( n2 ) entries of A using O ( n2 ) time assuming an oracle that computes the kernel function k. We can then subsequently apply our SPCA algorithms and achieve polynomial time runtime with the same approximation guarantees . 1.2 PRIOR WORK . SPCA was formally introduced by ( d ’ Aspremont et al. , 2007 ) ; however , previously studied PCA approaches based on rotating ( Jolliffe , 1995 ) or thresholding ( Cadima & Jolliffe , 1995 ) the top singular vector of the input matrix seemed to work well , at least in practice , given sparsity constraints . Following ( d ’ Aspremont et al. , 2007 ) , there has been an abundance of interest in SPCA . ( Jolliffe et al. , 2003 ) considered LASSO ( SCoTLASS ) on an ` 1 relaxation of the problem , while ( Zou & Hastie , 2005 ) considered a non-convex regression-type approximation , penalized similar to LASSO . Additional heuristics based on LASSO ( Ando et al. , 2009 ) and non-convex ` 1 regularizations ( Zou & Hastie , 2005 ; Zou et al. , 2006 ; Sriperumbudur et al. , 2007 ; Shen & Huang , 2008 ) have also been explored . Random sampling approaches based on non-convex ` 1 relaxations ( Fountoulakis et al. , 2017 ) have also been studied ; we highlight that unlike our approach , ( Fountoulakis et al. , 2017 ) solved a non-convex relaxation of the SPCA problem and thus perhaps relied on locally optimal solutions . Additionally , ( Moghaddam et al. , 2006b ) considered a branch-and-bound heuristic motivated by greedy spectral ideas . ( Journée et al. , 2010 ; Papailiopoulos et al. , 2013 ; Kuleshov , 2013 ; Yuan & Zhang , 2013 ) further explored other spectral approaches based on iterative methods similar to the power method . ( Yuan & Zhang , 2013 ) specifically designed a sparse PCA algorithm with early stopping for the power method , based on the target sparsity . Another line of work focused on using semidefinite programming ( SDP ) relaxations ( d ’ Aspremont et al. , 2007 ; d ’ Aspremont et al. , 2008 ; Amini & Wainwright , 2009 ) . Notably , ( Amini & Wainwright , 2009 ) achieved provable theoretical guarantees regarding the SDP and thresholding approach of ( d ’ Aspremont et al. , 2007 ) in a specific , high-dimensional spiked covariance model , in which a base matrix is perturbed by adding a sparse maximal eigenvector . In other words , the input matrix is the identity matrix plus a “ spike ” , i.e. , a sparse rank-one matrix . Despite the variety of heuristic-based sparse PCA approaches , very few theoretical guarantees have been provided for SPCA ; this is partially explained by a line of hardness-of-approximation results . The sparse PCA problem is well-known to be NP-Hard ( Moghaddam et al. , 2006a ) . ( Magdon-Ismail , 2017 ) shows that if the input matrix is not PSD , then even the sign of the optimal value can not be determined in polynomial time unless P = NP , ruling out any multiplicative approximation algorithm . In the case where the input matrix is PSD , ( Chan et al. , 2016 ) shows that it is NP-hard to approximate the optimal value up to multiplicative ( 1 + ) error , ruling out any polynomialtime approximation scheme ( PTAS ) . Moreover , they show Small-Set Expansion hardness for any polynomial-time constant factor approximation algorithm and also that the standard SDP relaxation might have an exponential gap . We conclude by summarizing prior work that offers provable guarantees ( beyond the work of ( Amini & Wainwright , 2009 ) ) , typically given some assumptions about the input matrix . ( d ’ Aspremont et al. , 2014 ) showed that the SDP relaxation can be used to find provable bounds when the covariance input matrix is formed by a number of data points sampled from Gaussian models with a single sparse singular vector . ( Papailiopoulos et al. , 2013 ) presented a combinatorial algorithm that analyzed a specific set of vectors in a low-dimensional eigenspace of the input matrix and presented relative error guarantees for the optimal objective , given the assumption that the input covariance matrix has a decaying spectrum . ( Asteris et al. , 2011 ) gave a polynomial-time algorithm that solves sparse PCA exactly for input matrices of constant rank . ( Chan et al. , 2016 ) showed that sparse PCA can be approximated in polynomial time within a factor of n−1/3 and also highlighted an additive PTAS of ( Asteris et al. , 2015 ) based on the idea of finding multiple disjoint components and solving bipartite maximum weight matching problems . This PTAS needs time npoly ( 1/ ) , whereas all of our algorithms have running times that are a low-degree polynomial in n .

Summary: This paper proposed three simple algorithms for sparse principal component analysis (SPCA): a) randomized matrix multiplication; b) deterministic thresholding scheme; and c) semidefinite programming relaxation. All of the proposed algorithms look like native combinations of existing techniques and simple sparsification steps. However, it is somewhat interesting to have novel theoretical guarantees for these simple strategies whose error bounds depend on the properties of the input matrix and the target sparsity.