Unique sparse decomposition of low rank matrices

1 Introduction . The problem of matrix decomposition has been a popular and fundamental topic under extensive investigations across several disciplines , including signal processing , machine learning , natural language processing [ 10 , 11 , 31 , 46 , 32 , 8 ] . From the decomposition , one can construct efficient representation of the original data matrix . However , for any matrix Y ∈ Rp×n that can be factorized as a product of two matrices A ∈ Rp×r and X ∈ Rr×n , there exist infinitely many decompositions , simply because one can use any r× r invertible matrix Q to construct A′ = AQ and X ′ = Q−1X such that Y = AX = A′X ′ , while A′ 6= A andX ′ 6= X . Thus , in various applications , additional structures and priors are being exploited to find a preferred representation [ 22 , 15 ] . For example , principal component analysis ( PCA ) aims to find orthogonal representations which retain as much variations in Y as possible [ 17 , 23 ] , whereas independent component analysis ( ICA ) targets the representations of statistically independent non-Gaussian signals [ 26 ] . In this paper , we are interested in finding a unique sparse low-dimensional representation of Y . To this end , we study the decomposition of a low rank matrix Y ∈ Rp×n that satisfies Y = AX , ( 1.1 ) where A ∈ Rp×r is an unknown deterministic matrix , with r < min { n , p } , and X ∈ Rr×n is an unknown sparse matrix . Formulation ( 1.1 ) is an important model problem in many applications . As columns of Y are viewed as linear combinations of columns of A with X being the sparse coefficient , ( 1.1 ) can be used to form overlapping clusters of the n columns of Y via the support of . X with columns of A being viewed as r cluster centers [ 12 , 7 ] . When we form a p × r low-dimensional representation of Y via sparse combinations , this greatly enhance the interpretability of the resulting representations [ 28 , 21 , 4 ] , in the same spirit as the sparse PCA , but ( 1.1 ) generalizes to the factorization of non-orthogonal matrices . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . To motivate our approach , we first consider the simple case that A has orthonormal columns , namely , ATA = Ir1 . Then it is easy to see that the sparse coefficient matrix X is recovered by multiplying Y on the left byAT , ATY = ATAX = X . ( 1.2 ) The problem of finding such orthonormal matrix A boils down to successively finding a unit-norm direction q that renders qTY as sparse as possible [ 34 , 39 , 35 ] , min q ∥ ∥qTY ∥ ∥ sparsity s. t. ‖q‖2 = 1 . ( 1.3 ) However , the natural choice of sparsity penalty , either ℓ0 or ℓ1 , leads to trivial and meaningless solutions , as there always exists q in the null space ofAT such that qTY = 0 . To avoid the null space ofAT , we instead choose to find the unit direction q that maximizes the ℓ4 norm of qTY as max q ∥ ∥qTY ∥ ∥ 4 s. t. ‖q‖2 = 1 . ( 1.4 ) The above formulation is based on the key observation that the objective value ismaximized when q coincideswith one column ofA ( see , Section 2 , for details ) while the objective value is zero when q lies in the null space of AT . The ℓ4 norm objective function and its variants have been adopted as a sparsity regularizer in a line of recent works [ 30 , 44 , 43 , 35 , 42 ] . However , even with this new objective function , the null space ofAT persists as a challenge for solving the optimization problem : they form a flat region of saddle points . This paper characterizes the nonconvex optimization landscape of ( 1.4 ) and proposes a guaranteed procedure that avoids the flat region and provably recovers the global solution to ( 1.4 ) , which corresponds to one column of A . More specifically , we demonstrate that , despite the non-convexity , ( 1.4 ) still possesses benign geometric property in the sense that any strict local solution with large objective value is globally optimal and recovers one column ofA , up to its sign . See , Theorem 3.1 in Section 3.1 for the population level result and Theorem 3.4 for the finite sample result . We further extend these results to the general case when A only has full column rank in Theorem 3.6 of Section 3.2 . To recover a general A with full column rank , our procedure first resorts to a preconditioning procedure of Y proposed in Section 2.3 and then solves a optimization problem similar to ( 1.4 ) . From our analysis of the optimization landscape , the intriguing problem boils down to developing algorithms to recover the nontrivial local solutions by avoiding regions with small objective values . We thus propose a simple initialization scheme in Section 4.1 and prove in Theorem 4.3 that such initialization , proceeded with any second order descent algorithm [ 20 , 27 ] , suffices to find the global solution , up to some statistical error . Our theoretical analysis provides the explicit convergence rate of the statistical error and characterizes its dependence on various dimensions , such as p , r and n , as well as the sparsity of X . Numerical simulation results are provided in Section 5 . Due to the space limitation , we defer all the proof along with our conclusions and discussion of several future directions of our work to Appendix . Notations Throughout this paper , we use bold lowercase letters , like a , to represent vectors and bold uppercase letters , like A , to represent matrices . For matrix X , Xij denotes the entry at the i-th row and j-th column ofX , withXi· andX·j denoting the i-th row and j-th column ofX , respectively . Oftentimes , we writeX·j = Xj for simplicity . We use grad and Hess to represent the Riemannian gradient and Hessian . For any vector v ∈ Rd , we use ‖v‖q to denote its ℓq norm , for 1 ≤ q ≤ ∞ . The notation v◦q stands for { v q i } i . For matrices , we use ‖·‖F and ‖·‖op to denote the Frobenius norm and the operator norm , respectively . For any positive integer d , wewrite [ d ] = { 1 , 2 , . . . , d } . The unit sphere in d-dimensional real space Rd is written as Sd−1 . For two sequences an and bn , we write an À bn if there exists some constant C > 0 such that an ≤ Cbn for all n. Both uppercase C and lowercase c are reserved to represent numerical constants , whose values may vary line by line . 1 Ir is the identity matrix of size r × r . 1.1 Related work . Finding the unique factorization of a matrix is an ill-posed problem in general due to infinitely many solutions . There exist several strands of studies from different contexts on finding the unique decomposition of Y by imposing additional structures onA andX . We start by reviewing the literature which targets the sparse decomposition of Y . Dictionary learning The problems of dictionary learning ( DL ) [ 2 , 38 , 19 , 39 ] and sparse blind deconvolution or convolutional dictionary learning [ 13 , 29 ] study the unique decomposition of Y = AX whereX is sparse andA has full row rank . In this case , the row space of Y lies in the row space ofX , suggesting to recover the sparse rows ofX via solving the following problem , min q ∥ ∥qTY ∥ ∥ 1 s. t. q 6= 0 . ( 1.5 ) Under certain scaling and incoherence conditions onA , the objective achieves theminimum value when q is equal to one column ofA , at the same time qTY recovers one sparse row of X . This idea has been studied andmodified in a strand of papers whenA has full row rank [ 38 , 39 , 45 , 30 , 44 , 35 , 42 , 37 , 47 ] . In our context , the major difference rises in the matrix A , which has full column rank rather than row rank , therefore minimizing ∥ ∥qTY ∥ ∥ 1 as before only leads to some vector in the null space of AT , yielding the trivial zero objective value . We would love to note that [ 35 ] uses the same objective function in ( 1.4 ) to study the problem of overcomplete dictionary learning ( where A has full row rank ) , however the optimization landscape when A has full column rank is significantly different from that in the overcomplete setting . The more complicated optimization landscape in our setting brings additional difficulty of the analysis and requires a proper initialization in our proposed algorithm . We refer to Appendix B for detailed technical comparison with [ 35 ] . Sparse PCA Sparse principal component analysis ( SPCA ) is a popular method that recovers a unique decomposition of a low-rank matrix Y by utilizing the sparsity of its singular vectors . However , as being said , under Y = AX , SPCA is only applicable when X coincides with the right singular vectors of Y . Indeed , one formulation of SPCA is to solve max U∈Rn×r tr ` UTY TY U ˘ − λ‖U‖1 , s. t. U TU = Ir , ( 1.6 ) which is promising only if X corresponds to the right singular vectors of Y . It is worth mentioning that among the various approaches of SPCA , the following one might be used to recover one sparse row of X , min u , v ∥ ∥Y − uvT ∥ ∥ 2 2 + λ‖v‖1 s. t. ‖u‖2= 1 . ( 1.7 ) This procedure was originally proposed by [ 49 ] and [ 36 ] together with an efficient algorithm by alternating the minimization between u and v. However , there is no guarantee that the resulting solution recovers the ground truth . Factor analysis Factor analysis is a popular statistical tool for constructing low-rank representations of Y by postulating Y = AX + E where A ∈ Rp×r is the so-called loading matrix with r = rank ( A ) < min { n , p } , X ∈ Rr×n contains n realizations of a r-dimensional factor and E is some additive noise . Here only Y is observable . Factor analysis is mainly used to recover the low-dimension column space ofA or the row space ofX , rather than to identify and recover the unique decomposition . Recently , [ 7 ] studied the unique decomposition of Y when the columns of X are i.i.d . realizations of a r-dimensional latent random factor . The unique decomposition is further used for ( overlapping ) clustering the rows of Y via the assignment matrix A . To uniquely identify A , [ 7 ] assumes that A contains at least one r × r identity matrix , coupled with other scaling conditions onA ( we refer to [ 7 ] for detailed discussions of other existing conditions in the literature of factor models that ensure the unique decomposition of Y but require strong prior information on either A or X ) . By contrast , we rely on the sparsity of X instead of A which is more general than requiring the existence of a r × r identity matrix . NMF and topic models Such existence condition of identity matrix in either A or X has a variant in non-negative matrix factorization ( NMF ) [ 14 ] and topic models [ 3 , 8 , 9 ] , also see the references therein , where Y , A and X have non-negative entries . Since all Y , A andX from model ( 1.1 ) are allowed to have arbitrary signs in our context , the approaches designed for NMF and topic models are inapplicable .

Summary: This paper studies the unique decomposition of a low-rank matrix into sparse and full column-rank components. Explicit conditions under which such decomposition is possible (up to a signed permutation) are derived. Moreover, an initialization scheme is proposed that, together a generic second order method, guarantees the exact and unique recovery of the sparse and full column-rank components. The authors first study the setting where the full column-rank component has orthogonal columns. Then they extend their results to the general setting where this component is no longer column-orthogonal. To the best of my knowledge, the technical contributions of the paper are sound and correct.