A Note on Sparse Generalized Eigenvalue Problem

1 Introduction . The sparse generalized eigenvalue problem ( SGEP ) is to solve the following constrained optimization problem : max x∈Rp xTÃx xTB̃x , subject to ‖x‖0 ≤ k , ( 1 ) where Ã , B̃ are both p-by-p symmetric matrices and B̃ is ( semi ) positive definite ; ‖x‖0 denotes the ` 0-norm of x , i.e. , the number of nonzero entries of x ; k is an integer which may be much smaller than p. For a fixed k , we let x∗ be the solution to ( 1 ) . Moreover , matrices Ã and B̃ usually have the following decomposition : Ã = A+ E , B̃ = B + F , ( 2 ) where A , B are underlying symmetric matrices and B is positive definite . However , A and B are unobserved in practice and they can be contaminated by some noise/perturbation matrices E and F , respectively . Let λ1 be the largest generalized eigenvalue of matrix pair ( A , B ) and u1 be its corresponding eigenvector which is also known as the leading eigenvector . In addition , u1 is always assumed to be sparse , i.e. , ‖u1‖0 p. Therefore , the SGEP is essentially to find an approximation of u1 based on observation ( Ã , B̃ ) . The SGEP is challenging due to the following aspects . First , the sparsity constraint optimization problem ( 1 ) is NP-hard , since it is essentially a subset selection problem [ 19 , 20 ] . To recover the underlying sparsity structure , we need to try different k which is computationally expensive . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Second , due to insufficient number of samples , Ã and B̃ can be both ill-conditioned or even share an approximate common null space , and perturbations E and F can be large . As a result , the leading eigenvector of pair ( Ã , B̃ ) may not be a good approximation of u1 . Though challenging , especially when there is a limited number of data samples , SGEP and related problems have received more and more attentions in the past decade and various numerical methods are proposed to solve the SGEP . For example , in [ 24 ] , the SGEP is framed as a d.c. ( difference of convex functions ) program and is solved as a sequence of convex programs by invoking the majorization-minimization method . In [ 23 ] , the ` 0 norm constraint is approximated by a continuous surrogate function , an algorithm inspired by the majorization-minimization method is developed via iteratively majorizing the surrogate function by a quadratic separable function , and a systematic way based on smoothing is proposed to deal with the singularity issue . [ 9 ] proposes a semidefinite programming method for sparse canonical correlation analysis which is one of important SGEP applications . In [ 30 ] , a two-stage computational framework is proposed to solve the SGEP , in which the first stage computes an initial guess via convex relaxation , and the second stage solves a nonconvex optimization via a fixed step size steepest ascent method , and followed by the simple truncation . In [ 22 ] , a general framework called sparse estimation with linear programming is proposed for sparse canonical correlation analysis . Combining the idea of ` 1-penalized adaptive normalized quasi-Newton algorithm with nested orthogonal complement structure , an algorithm is proposed in [ 32 ] to solve the SGEP . [ 13 ] solves the SGEP by modifying the standard generalized orthogonal iteration with a sparsity-inducing penalty for the eigenvectors . For more comprehensive review of computations in the SGEP , please see [ 3 , 42 ] and references therein . Despite the efforts mentioned above , there is very little work investigating the optimal estimation error and no work provides theoretical guarantees whether the underlying sparsity structure of the leading eigenvector of SGEP can be recovered . In this paper , we bridge the theoretical gap in the literature . First , we provide both upper and lower bounds of the approximation error given noisy observations in the sparse generalized eigenvalue problem setting . To be specific , we prove that the difference between u1 and x∗ is bounded by two parts , the perturbation error and error of not recovering the true support of u1 . Furthermore , we show that the error bound we obtained is rate optimal . That is , the lower bound matches the upper bound up to a multiplicative constant . Second , we consider a family of non-convex penalty functions and reformulate ( 1 ) into a regularization problem . We show that such regularized estimator enjoys the merits of sparsity and nearly unbiasedness . With these nice properties , the estimator can be shown to achieve the optimal estimation error rate and can recover the true support of leading eigenvector as well . In addition , we also present a new computational algorithm . The estimation procedure adopts the Alternating Direction Method of Multipliers ( ADMM , [ 37 , 21 ] ) and can recover the sparsity structure well under mild conditions . Local convergence theory is established for the proposed method . The rest of paper is organized as follows . In Section 2 , we introduce the sparse generalized eigenvalue problem with its applications and technical tools . In Section 3 , we provide a sensitivity analysis for SGEP and establish its upper and lower perturbation bounds . In Section 4 , we propose a regularized estimator via using non-convex penalization method and established the estimation bound . We also propose a computational method , non-convex SGEP algorithm ( NC-SGEP ) , to tackle with the estimation issues . Multiple numerical results are given in Section 5 and corroborate our theory . The concluding remark is given in Section 6 . Notations : The calligraphic letters I , J , K , and S are usually used to denote index sets . |I| denotes the cardinality of I , e.g. , I = { i1 , i2 , . . . , is } , where i1 , i2 , . . . , is are distinct integers , then |I| = s. We use x to denote a generic vector in Rp and xT to denote its transpose . x [ j ] is the jth element in x. x [ I ] stands for the subvector of x with indices in set I . We use A to denote a matrix in Rp×p . A [ i , j ] is the entry in the ith row and the jth column . A [ I , J ] stands for the submatrix of A with row indices in set I and column indices in set J . If I = J , A [ I , I ] is also denoted by AI . We also let ( I , J ) be the union of sets I and J . For x ∈ Rp , supp ( x ) denotes the index set of all nonzero entries of x . For matrix A , supp ( A ) denotes the index set of all nonzero rows of A. Ip is the p× p identity matrix . diag ( d1 , . . . , dp ) represents a diagonal matrix with d1 , . . . , dp being its diagonal elements . ‖x‖0 represents the number of non-zero entries in x ; ‖x‖2 : = √∑ j x 2 [ j ] ; ‖A‖2 denotes the largest singular value of A ; ‖A‖F and ‖A‖∗ is the Frobenius norm and nuclear norm of matrix A respectively . ‖A‖∞ : = maxi , j |A [ i , j ] | . We write a b ( a b ) if a ≥ Kb ( a ≤ b/K ) for some sufficiently large number K . 2 Preliminary . Applications Sparse generalized eigenvalue problem has a wide application . Many high-dimensional multivariate statistical problems can be formulated as special instances of ( 1 ) . Example 1 [ Sparse Principal Component Analysis [ 44 , 5 , 1 , 35 , 36 ] ] Given n observations with p features , sparse principal component analysis ( SPCA ) seeks the best low dimensional projection of the observed data for increasing interpretability , minimizing information loss and achieving sparsity structure . Let Σ̃ be the empirical covariance matrix . SPCA aims to solve max x∈Rp xTΣ̃x , subject to ‖x‖0 ≤ k , ‖x‖2 = 1 . ( 3 ) Such problem is a special case of SGEP where Ã = Σ̃ and B̃ = I . Example 2 [ Sparse Fisher ’ s Discriminant Analysis [ 31 , 12 , 17 , 14 , 16 ] ] Given n observations with p features from K different classes , Fisher ’ s discriminant analysis seeks a projection for mapping observations to the low dimensional space on which the between-class variance Σb is large while the within-class variance Σw is small . Let Σ̃b and Σ̃w be the sample estimates of Σb and Σw , respectively . To obtain the sparse leading discriminant vector , one can solve max x∈Rp xTΣ̃bx , subject to xTΣ̃wx = 1 , ‖x‖0 ≤ k. ( 4 ) Such a problem can be formulated as an SGEP with Ã = Σ̃b and B̃ = Σ̃w . Example 3 [ Sparse Canonical Correlation Analysis [ 39 , 9 , 10 , 41 ] ] Given two random vectors X ∈ Rp , Y ∈ Rp , let Σxx , Σyy , Σxy be the covariance matrices for X , Y , and the cross-covariance matrix between X and Y , respectively . Let Σ̃xx , Σ̃yy , Σ̃xy be estimators ( constructed from samples ) for Σxx , Σyy , Σxy , respectively . Sparse canonical correlation analysis ( SCCA ) aims to solve the constrained optimization problem : max ux , uy uTx Σ̃xyuy , subject to u T x Σ̃xxux = 1 , u T y Σ̃yyuy = 1 , ‖ux‖0 ≤ kx , ‖uy‖0 ≤ ky , where kx and ky are two small integers . Such a problem can be recast as an SGEP with Ã = [ 0 Σ̃xy Σ̃Txy 0 ] , B̃ = [ Σ̃xx 0 0 Σ̃yy ] , u1 = [ ux uy ] . 1 Strictly speaking , two problems are not exactly equivalent , because ‖u‖0 ≤ kx + ky does not necessarily imply ‖ux‖0 ≤ kx and ‖uy‖0 ≤ ky . Technical preparation We first introduce some useful terminologies for describing the generalized eigenvalue problem ( GEP ) . Let A ∈ Rp×p be a symmetric matrix and B ∈ Rp×p be a symmetric and positive definite matrix . Then GEP for the matrix pair ( A , B ) is defined as Aui = λiBui ; i = 1 , . . . , p. Without loss of generality , we can always assume that λ1 ≥ · · · ≥ λp . Here ui is ith eigenvector and λi is ith eigenvalue ; ( λi , ui ) is called ith eigenpair of ( A , B ) . Notice that ui is only determined up to a scale . In this paper , we always assume that ‖ui‖2 = 1 . The perturbation analysis of generalized eigenvalue problem has been extensively studied since 1970s and many perturbation bounds have been developed [ 25 , 27 , 28 , 33 ] . For reader convenience , we present several existing fundamental results before moving to our main theory immediately . Definition 1 The angle between x , y ( 6= 0 ) ∈ Rp is defined as θ ( x , y ) : = arccos |x Ty| ‖x‖2‖y‖2 . We assume x and y ∈ Rp have unit l2 norms and let [ x , X2 ] be an orthogonal matrix . Then it holds | sin θ ( x , y ) | = ‖XT2 y‖2 . ( 5 ) Furthermore , it can be checked that | sin θ ( x , y ) | ≤ ‖x−y‖2 ≤ √ 2| sin θ ( x , y ) | . Thus | sin θ ( x , y ) | is a measure to quantify the distance between two vectors . In generalized eigenvalue problem , Crawford number is an important quantity for characterizing the perturbation bound . Definition 2 The Crawford number of a symmetric matrix pair ( A , B ) is defined as c ( A , B ) : = min ‖x‖2=1 √ ( xTAx ) 2 + ( xTBx ) 2 . A symmetric matrix pair ( A , B ) is referred to as “ definite ” if c ( A , B ) > 0 . Obviously , the symmetric matrix pair ( A , B ) is definite for any positive definite B . By [ 25 ] , it is known that the Crawford number is continuous with respect to the matrix pair .

Summary: The paper studies a class of sparse generalized eigenproblems which encompasses several common problems in machine learning. For this class of problems it gives upper and lower perturbation bounds which are shown to be rate-optimal. The authors then present an optimization problem with a non-convex penalty function and show that it can be used to estimate the sparse leading eigenvector. The theoretical results are validated numerically using synthetic data in the experimental section.