Unique sparse decomposition of low rank matrices

1 Introduction . The problem of matrix decomposition has been a popular and fundamental topic under extensive investigations across several disciplines , including signal processing , machine learning , natural language processing [ 10 , 11 , 31 , 46 , 32 , 8 ] . From the decomposition , one can construct efficient representation of the original data matrix . However , for any matrix Y ∈ Rp×n that can be factorized as a product of two matrices A ∈ Rp×r and X ∈ Rr×n , there exist infinitely many decompositions , simply because one can use any r× r invertible matrix Q to construct A′ = AQ and X ′ = Q−1X such that Y = AX = A′X ′ , while A′ 6= A andX ′ 6= X . Thus , in various applications , additional structures and priors are being exploited to find a preferred representation [ 22 , 15 ] . For example , principal component analysis ( PCA ) aims to find orthogonal representations which retain as much variations in Y as possible [ 17 , 23 ] , whereas independent component analysis ( ICA ) targets the representations of statistically independent non-Gaussian signals [ 26 ] . In this paper , we are interested in finding a unique sparse low-dimensional representation of Y . To this end , we study the decomposition of a low rank matrix Y ∈ Rp×n that satisfies Y = AX , ( 1.1 ) where A ∈ Rp×r is an unknown deterministic matrix , with r < min { n , p } , and X ∈ Rr×n is an unknown sparse matrix . Formulation ( 1.1 ) is an important model problem in many applications . As columns of Y are viewed as linear combinations of columns of A with X being the sparse coefficient , ( 1.1 ) can be used to form overlapping clusters of the n columns of Y via the support of . X with columns of A being viewed as r cluster centers [ 12 , 7 ] . When we form a p × r low-dimensional representation of Y via sparse combinations , this greatly enhance the interpretability of the resulting representations [ 28 , 21 , 4 ] , in the same spirit as the sparse PCA , but ( 1.1 ) generalizes to the factorization of non-orthogonal matrices . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . To motivate our approach , we first consider the simple case that A has orthonormal columns , namely , ATA = Ir1 . Then it is easy to see that the sparse coefficient matrix X is recovered by multiplying Y on the left byAT , ATY = ATAX = X . ( 1.2 ) The problem of finding such orthonormal matrix A boils down to successively finding a unit-norm direction q that renders qTY as sparse as possible [ 34 , 39 , 35 ] , min q ∥ ∥qTY ∥ ∥ sparsity s. t. ‖q‖2 = 1 . ( 1.3 ) However , the natural choice of sparsity penalty , either ℓ0 or ℓ1 , leads to trivial and meaningless solutions , as there always exists q in the null space ofAT such that qTY = 0 . To avoid the null space ofAT , we instead choose to find the unit direction q that maximizes the ℓ4 norm of qTY as max q ∥ ∥qTY ∥ ∥ 4 s. t. ‖q‖2 = 1 . ( 1.4 ) The above formulation is based on the key observation that the objective value ismaximized when q coincideswith one column ofA ( see , Section 2 , for details ) while the objective value is zero when q lies in the null space of AT . The ℓ4 norm objective function and its variants have been adopted as a sparsity regularizer in a line of recent works [ 30 , 44 , 43 , 35 , 42 ] . However , even with this new objective function , the null space ofAT persists as a challenge for solving the optimization problem : they form a flat region of saddle points . This paper characterizes the nonconvex optimization landscape of ( 1.4 ) and proposes a guaranteed procedure that avoids the flat region and provably recovers the global solution to ( 1.4 ) , which corresponds to one column of A . More specifically , we demonstrate that , despite the non-convexity , ( 1.4 ) still possesses benign geometric property in the sense that any strict local solution with large objective value is globally optimal and recovers one column ofA , up to its sign . See , Theorem 3.1 in Section 3.1 for the population level result and Theorem 3.4 for the finite sample result . We further extend these results to the general case when A only has full column rank in Theorem 3.6 of Section 3.2 . To recover a general A with full column rank , our procedure first resorts to a preconditioning procedure of Y proposed in Section 2.3 and then solves a optimization problem similar to ( 1.4 ) . From our analysis of the optimization landscape , the intriguing problem boils down to developing algorithms to recover the nontrivial local solutions by avoiding regions with small objective values . We thus propose a simple initialization scheme in Section 4.1 and prove in Theorem 4.3 that such initialization , proceeded with any second order descent algorithm [ 20 , 27 ] , suffices to find the global solution , up to some statistical error . Our theoretical analysis provides the explicit convergence rate of the statistical error and characterizes its dependence on various dimensions , such as p , r and n , as well as the sparsity of X . Numerical simulation results are provided in Section 5 . Due to the space limitation , we defer all the proof along with our conclusions and discussion of several future directions of our work to Appendix . Notations Throughout this paper , we use bold lowercase letters , like a , to represent vectors and bold uppercase letters , like A , to represent matrices . For matrix X , Xij denotes the entry at the i-th row and j-th column ofX , withXi· andX·j denoting the i-th row and j-th column ofX , respectively . Oftentimes , we writeX·j = Xj for simplicity . We use grad and Hess to represent the Riemannian gradient and Hessian . For any vector v ∈ Rd , we use ‖v‖q to denote its ℓq norm , for 1 ≤ q ≤ ∞ . The notation v◦q stands for { v q i } i . For matrices , we use ‖·‖F and ‖·‖op to denote the Frobenius norm and the operator norm , respectively . For any positive integer d , wewrite [ d ] = { 1 , 2 , . . . , d } . The unit sphere in d-dimensional real space Rd is written as Sd−1 . For two sequences an and bn , we write an À bn if there exists some constant C > 0 such that an ≤ Cbn for all n. Both uppercase C and lowercase c are reserved to represent numerical constants , whose values may vary line by line . 1 Ir is the identity matrix of size r × r . 1.1 Related work . Finding the unique factorization of a matrix is an ill-posed problem in general due to infinitely many solutions . There exist several strands of studies from different contexts on finding the unique decomposition of Y by imposing additional structures onA andX . We start by reviewing the literature which targets the sparse decomposition of Y . Dictionary learning The problems of dictionary learning ( DL ) [ 2 , 38 , 19 , 39 ] and sparse blind deconvolution or convolutional dictionary learning [ 13 , 29 ] study the unique decomposition of Y = AX whereX is sparse andA has full row rank . In this case , the row space of Y lies in the row space ofX , suggesting to recover the sparse rows ofX via solving the following problem , min q ∥ ∥qTY ∥ ∥ 1 s. t. q 6= 0 . ( 1.5 ) Under certain scaling and incoherence conditions onA , the objective achieves theminimum value when q is equal to one column ofA , at the same time qTY recovers one sparse row of X . This idea has been studied andmodified in a strand of papers whenA has full row rank [ 38 , 39 , 45 , 30 , 44 , 35 , 42 , 37 , 47 ] . In our context , the major difference rises in the matrix A , which has full column rank rather than row rank , therefore minimizing ∥ ∥qTY ∥ ∥ 1 as before only leads to some vector in the null space of AT , yielding the trivial zero objective value . We would love to note that [ 35 ] uses the same objective function in ( 1.4 ) to study the problem of overcomplete dictionary learning ( where A has full row rank ) , however the optimization landscape when A has full column rank is significantly different from that in the overcomplete setting . The more complicated optimization landscape in our setting brings additional difficulty of the analysis and requires a proper initialization in our proposed algorithm . We refer to Appendix B for detailed technical comparison with [ 35 ] . Sparse PCA Sparse principal component analysis ( SPCA ) is a popular method that recovers a unique decomposition of a low-rank matrix Y by utilizing the sparsity of its singular vectors . However , as being said , under Y = AX , SPCA is only applicable when X coincides with the right singular vectors of Y . Indeed , one formulation of SPCA is to solve max U∈Rn×r tr ` UTY TY U ˘ − λ‖U‖1 , s. t. U TU = Ir , ( 1.6 ) which is promising only if X corresponds to the right singular vectors of Y . It is worth mentioning that among the various approaches of SPCA , the following one might be used to recover one sparse row of X , min u , v ∥ ∥Y − uvT ∥ ∥ 2 2 + λ‖v‖1 s. t. ‖u‖2= 1 . ( 1.7 ) This procedure was originally proposed by [ 49 ] and [ 36 ] together with an efficient algorithm by alternating the minimization between u and v. However , there is no guarantee that the resulting solution recovers the ground truth . Factor analysis Factor analysis is a popular statistical tool for constructing low-rank representations of Y by postulating Y = AX + E where A ∈ Rp×r is the so-called loading matrix with r = rank ( A ) < min { n , p } , X ∈ Rr×n contains n realizations of a r-dimensional factor and E is some additive noise . Here only Y is observable . Factor analysis is mainly used to recover the low-dimension column space ofA or the row space ofX , rather than to identify and recover the unique decomposition . Recently , [ 7 ] studied the unique decomposition of Y when the columns of X are i.i.d . realizations of a r-dimensional latent random factor . The unique decomposition is further used for ( overlapping ) clustering the rows of Y via the assignment matrix A . To uniquely identify A , [ 7 ] assumes that A contains at least one r × r identity matrix , coupled with other scaling conditions onA ( we refer to [ 7 ] for detailed discussions of other existing conditions in the literature of factor models that ensure the unique decomposition of Y but require strong prior information on either A or X ) . By contrast , we rely on the sparsity of X instead of A which is more general than requiring the existence of a r × r identity matrix . NMF and topic models Such existence condition of identity matrix in either A or X has a variant in non-negative matrix factorization ( NMF ) [ 14 ] and topic models [ 3 , 8 , 9 ] , also see the references therein , where Y , A and X have non-negative entries . Since all Y , A andX from model ( 1.1 ) are allowed to have arbitrary signs in our context , the approaches designed for NMF and topic models are inapplicable .

Summary: The authors propose a new method and analysis to decompose a given matrix $Y$ as $Y=AX$ with $X$ sparse and $A$ full column rank. This is similar but different to sparse PCA and dictionary learning, and has interesting applications.  The proposed algorithm essentially recovers $A$ one column at a time (after deflation) and maximizes the $\ell_4$ norm of the overlap of a unit-norm vector with $Y$. As the authors also state, using the $\ell_4$ norm is not entirely new but it gives a benign optimization landscape (in particular, strict local minimizers that are close to the desired solutions).  The best results (theoretically and experimentally) are obtained when $A$ is an orthonormal matrix and $X$ is a random matrix from a Bernouilli-Gaussian distribution. The authors propose a procedure (called preconditioning) that generalizes their results (again theoretically and experimentally) to any $A$ but it is difficult to see how good the results are. In any case, the provided experiment clearly show that we do no longer have exact recovery in this case.  While the theoretical results seem -- I did not check the proofs in the 40(!) page appendix -- nontrivial and interesting, they only apply when recovering one vector and for a random $X$ from the Bernouilli-Gaussian distribution. No results are given for general $X$. In addition, the proposed deflation strategy is a typical technique to extend to multiple vector but typically performs quite poorly when $A$ is ill-conditioned (near linear depednent columns). The authors should therefore restate their contributions in the abstract and introduction to better reflect their main contributions.