Star-Convexity in Non-Negative Matrix Factorization

1 INTRODUCTION . Non-negative matrix factorization ( NMF ) is a ubiquitous technique for data analysis where one attempts to factorize a measurement matrix X into the product of non-negative matrices U , V ( Lee and Seung , 1999 ) . This simple problem has applications in recommender systems ( Luo et al. , 2014 ) , scientific analysis ( Berne et al. , 2007 ; Trindade et al. , 2017 ) , computer vision ( Gillis , 2012 ) , internet distance prediction ( Mao et al. , 2006 ) , audio processing ( Schmidt et al. , 2007 ) and many more domains . Often , the non-negativity is crucial for interpretability , for example , in the context of crystallography , the light sources , which are represented as matrix factors , have non-negative intensity ( Suram et al. , 2016 ) . Like many other non-convex optimization problems , finding the exact solution to NMF is NP-hard ( Pardalos and Vavasis , 1991 ; Vavasis , 2009 ) . NMF ’ s tremendous practical success is however at odds with such worst-case analysis , and simple algorithms based upon gradient descent are known to find good solutions in real-world settings ( Lee and Seung , 2001 ) . At the time when NMF was proposed , most analyses of optimization problems within machine learning focused on convex formulations such as SVMs ( Cortes and Vapnik , 1995 ) , but owing to the success of neural networks , non-convex optimization has experienced a resurgence in interest . Here , we revisit NMF from a fresh perspective , utilizing recent tools developed in the context of optimization in deep learning . Specifically , our main inspiration is the recent work of Kleinberg et al . ( 2018 ) and Zhou et al . ( 2019 ) that empirically demonstrate that gradients typically point towards the final minimizer for neural networks trained on real-world datasets and analyze the implications of such convexity properties for efficient optimization . In this paper , we show theoretically and empirically that a similar property called star-convexity holds in NMF . From a theoretical perspective , we consider an NMF instance with planted solution , inspired by the stochastic block model for social networks ( Holland et al. , 1983 ; Decelle et al. , 2011 ) and the planted clique problem studied in sum-of-squares literature ( Barak et al. , 2016 ) . We prove that between two points the loss is convex with high probability , and conclude that the loss surface is star-convex in the typical case — even if the loss is computed over unobserved data . From an empirical perspective , we verify that our theoretical results hold for an extensive collection < latexit sha1_base64= '' /JI5iynvNWOEKZjLq6ujHWhdsHw= '' > AAAB7nicbVDLSsNAFL2pr1pfVZduBovgqiQi1GXRjcsK9gFtKJPJpB06mYSZG6GEfoQbF4q49Xvc+TdO2yy09cDA4ZxzmXtPkEph0HW/ndLG5tb2Tnm3srd/cHhUPT7pmCTTjLdZIhPdC6jhUijeRoGS91LNaRxI3g0md3O/+8S1EYl6xGnK/ZiOlIgEo2il7kDaaEiH1Zpbdxcg68QrSA0KtIbVr0GYsCzmCpmkxvQ9N0U/pxoFk3xWGWSGp5RN6Ij3LVU05sbPF+vOyIVVQhIl2j6FZKH+nshpbMw0Dmwypjg2q95c/M/rZxjd+LlQaYZcseVHUSYJJmR+OwmF5gzl1BLKtLC7EjammjK0DVVsCd7qyeukc1X33Lr3cF1r3hZ1lOEMzuESPGhAE+6hBW1gMIFneIU3J3VenHfnYxktOcXMKfyB8/kDPMWPfQ== < /latexit > < latexit sha1_base64= '' /JI5iynvNWOEKZjLq6ujHWhdsHw= '' > AAAB7nicbVDLSsNAFL2pr1pfVZduBovgqiQi1GXRjcsK9gFtKJPJpB06mYSZG6GEfoQbF4q49Xvc+TdO2yy09cDA4ZxzmXtPkEph0HW/ndLG5tb2Tnm3srd/cHhUPT7pmCTTjLdZIhPdC6jhUijeRoGS91LNaRxI3g0md3O/+8S1EYl6xGnK/ZiOlIgEo2il7kDaaEiH1Zpbdxcg68QrSA0KtIbVr0GYsCzmCpmkxvQ9N0U/pxoFk3xWGWSGp5RN6Ij3LVU05sbPF+vOyIVVQhIl2j6FZKH+nshpbMw0Dmwypjg2q95c/M/rZxjd+LlQaYZcseVHUSYJJmR+OwmF5gzl1BLKtLC7EjammjK0DVVsCd7qyeukc1X33Lr3cF1r3hZ1lOEMzuESPGhAE+6hBW1gMIFneIU3J3VenHfnYxktOcXMKfyB8/kDPMWPfQ== < /latexit > < latexit sha1_base64= '' /JI5iynvNWOEKZjLq6ujHWhdsHw= '' > AAAB7nicbVDLSsNAFL2pr1pfVZduBovgqiQi1GXRjcsK9gFtKJPJpB06mYSZG6GEfoQbF4q49Xvc+TdO2yy09cDA4ZxzmXtPkEph0HW/ndLG5tb2Tnm3srd/cHhUPT7pmCTTjLdZIhPdC6jhUijeRoGS91LNaRxI3g0md3O/+8S1EYl6xGnK/ZiOlIgEo2il7kDaaEiH1Zpbdxcg68QrSA0KtIbVr0GYsCzmCpmkxvQ9N0U/pxoFk3xWGWSGp5RN6Ij3LVU05sbPF+vOyIVVQhIl2j6FZKH+nshpbMw0Dmwypjg2q95c/M/rZxjd+LlQaYZcseVHUSYJJmR+OwmF5gzl1BLKtLC7EjammjK0DVVsCd7qyeukc1X33Lr3cF1r3hZ1lOEMzuESPGhAE+6hBW1gMIFneIU3J3VenHfnYxktOcXMKfyB8/kDPMWPfQ== < /latexit > < latexit sha1_base64= '' /JI5iynvNWOEKZjLq6ujHWhdsHw= '' > AAAB7nicbVDLSsNAFL2pr1pfVZduBovgqiQi1GXRjcsK9gFtKJPJpB06mYSZG6GEfoQbF4q49Xvc+TdO2yy09cDA4ZxzmXtPkEph0HW/ndLG5tb2Tnm3srd/cHhUPT7pmCTTjLdZIhPdC6jhUijeRoGS91LNaRxI3g0md3O/+8S1EYl6xGnK/ZiOlIgEo2il7kDaaEiH1Zpbdxcg68QrSA0KtIbVr0GYsCzmCpmkxvQ9N0U/pxoFk3xWGWSGp5RN6Ij3LVU05sbPF+vOyIVVQhIl2j6FZKH+nshpbMw0Dmwypjg2q95c/M/rZxjd+LlQaYZcseVHUSYJJmR+OwmF5gzl1BLKtLC7EjammjK0DVVsCd7qyeukc1X33Lr3cF1r3hZ1lOEMzuESPGhAE+6hBW1gMIFneIU3J3VenHfnYxktOcXMKfyB8/kDPMWPfQ== < /latexit > < latexit sha1_base64= '' /JI5iynvNWOEKZjLq6ujHWhdsHw= '' > AAAB7nicbVDLSsNAFL2pr1pfVZduBovgqiQi1GXRjcsK9gFtKJPJpB06mYSZG6GEfoQbF4q49Xvc+TdO2yy09cDA4ZxzmXtPkEph0HW/ndLG5tb2Tnm3srd/cHhUPT7pmCTTjLdZIhPdC6jhUijeRoGS91LNaRxI3g0md3O/+8S1EYl6xGnK/ZiOlIgEo2il7kDaaEiH1Zpbdxcg68QrSA0KtIbVr0GYsCzmCpmkxvQ9N0U/pxoFk3xWGWSGp5RN6Ij3LVU05sbPF+vOyIVVQhIl2j6FZKH+nshpbMw0Dmwypjg2q95c/M/rZxjd+LlQaYZcseVHUSYJJmR+OwmF5gzl1BLKtLC7EjammjK0DVVsCd7qyeukc1X33Lr3cF1r3hZ1lOEMzuESPGhAE+6hBW1gMIFneIU3J3VenHfnYxktOcXMKfyB8/kDPMWPfQ== < /latexit > < latexit sha1_base64= '' /JI5iynvNWOEKZjLq6ujHWhdsHw= '' > AAAB7nicbVDLSsNAFL2pr1pfVZduBovgqiQi1GXRjcsK9gFtKJPJpB06mYSZG6GEfoQbF4q49Xvc+TdO2yy09cDA4ZxzmXtPkEph0HW/ndLG5tb2Tnm3srd/cHhUPT7pmCTTjLdZIhPdC6jhUijeRoGS91LNaRxI3g0md3O/+8S1EYl6xGnK/ZiOlIgEo2il7kDaaEiH1Zpbdxcg68QrSA0KtIbVr0GYsCzmCpmkxvQ9N0U/pxoFk3xWGWSGp5RN6Ij3LVU05sbPF+vOyIVVQhIl2j6FZKH+nshpbMw0Dmwypjg2q95c/M/rZxjd+LlQaYZcseVHUSYJJmR+OwmF5gzl1BLKtLC7EjammjK0DVVsCd7qyeukc1X33Lr3cF1r3hZ1lOEMzuESPGhAE+6hBW1gMIFneIU3J3VenHfnYxktOcXMKfyB8/kDPMWPfQ== < /latexit > < latexit sha1_base64= '' /JI5iynvNWOEKZjLq6ujHWhdsHw= '' > AAAB7nicbVDLSsNAFL2pr1pfVZduBovgqiQi1GXRjcsK9gFtKJPJpB06mYSZG6GEfoQbF4q49Xvc+TdO2yy09cDA4ZxzmXtPkEph0HW/ndLG5tb2Tnm3srd/cHhUPT7pmCTTjLdZIhPdC6jhUijeRoGS91LNaRxI3g0md3O/+8S1EYl6xGnK/ZiOlIgEo2il7kDaaEiH1Zpbdxcg68QrSA0KtIbVr0GYsCzmCpmkxvQ9N0U/pxoFk3xWGWSGp5RN6Ij3LVU05sbPF+vOyIVVQhIl2j6FZKH+nshpbMw0Dmwypjg2q95c/M/rZxjd+LlQaYZcseVHUSYJJmR+OwmF5gzl1BLKtLC7EjammjK0DVVsCd7qyeukc1X33Lr3cF1r3hZ1lOEMzuESPGhAE+6hBW1gMIFneIU3J3VenHfnYxktOcXMKfyB8/kDPMWPfQ== < /latexit > < latexit sha1_base64= '' /JI5iynvNWOEKZjLq6ujHWhdsHw= '' > AAAB7nicbVDLSsNAFL2pr1pfVZduBovgqiQi1GXRjcsK9gFtKJPJpB06mYSZG6GEfoQbF4q49Xvc+TdO2yy09cDA4ZxzmXtPkEph0HW/ndLG5tb2Tnm3srd/cHhUPT7pmCTTjLdZIhPdC6jhUijeRoGS91LNaRxI3g0md3O/+8S1EYl6xGnK/ZiOlIgEo2il7kDaaEiH1Zpbdxcg68QrSA0KtIbVr0GYsCzmCpmkxvQ9N0U/pxoFk3xWGWSGp5RN6Ij3LVU05sbPF+vOyIVVQhIl2j6FZKH+nshpbMw0Dmwypjg2q95c/M/rZxjd+LlQaYZcseVHUSYJJmR+OwmF5gzl1BLKtLC7EjammjK0DVVsCd7qyeukc1X33Lr3cF1r3hZ1lOEMzuESPGhAE+6hBW1gMIFneIU3J3VenHfnYxktOcXMKfyB8/kDPMWPfQ== < /latexit > of real-world datasets spanning collaborative filtering ( Zhou et al. , 2008 ; Kula , 2017 ; Harper and Konstan , 2016 ) , signal decomposition ( Zhu , 2016 ; Li and Ngom , 2013 ; Li et al. , 2001 ; Erichson et al. , 2018 ) and audio processing ( Flenner and Hunter , 2017 ; canto Foundation ) , and demonstrate that the star-convex behavior results in efficient optimization . Finally , we show that star-convex behavior becomes more likely with growing number of parameters , suggesting that a similar result may hold as neural networks become wider . We provide supporting empirical evidence for this hypothesis on modern network architectures . 2 NMF AND STAR-CONVEXITY . The aim of NMF is to decompose some large measurement matrix X P Rnˆm into two non-negative matrices U P Rnˆr ` and V P Rrˆm ` such that X « UV . The canonical formulation of NMF is min U , Vě0 ` pU , Vq , where ` pU , Vq “ 1 2 } UV ´X } 2F ( 1 ) NMF is commonly used in recommender systems where entries pi , jq of X for example correspond to the rating user i has given to movie j ( Luo et al. , 2014 ) . In such settings , data might be missing as all users have not rated all movies . In those cases , it is common to only consider the loss over observed data ( Zhang et al. , 2006 ; Candès and Recht , 2009 ) . We let 1̂pi , jq be an indicator variable that is 1 if entry pi , jq is `` observed '' and 0 otherwise . The loss function is then min U , Vě0 ` pU , Vq “ 1 2 ÿ i , j 1̂pi , jq ˆ “ UV ‰ ij ´Xij ˙2 ( 2 ) NMF is similar to PCA which admits spectral strategies ; however , the non-negative constraints in NMF prevent such solutions and result in NP-hardness ( Vavasis , 2009 ) . Work on the computational complexity of NMF has shown that the problem is tractable for small constant dimensions r via algebraic methods ( Arora et al. , 2012 ) . In practice , however , these algorithms are not used , and simple variants of gradient descent , possibly via multiplicative updates ( Lee and Seung , 2001 ) , are popular and are known to work reliably ( Koren et al. , 2009 ) . This gap between theoretical hardness and practical performance is also found in deep learning . Optimizing neural networks is NP-hard in general ( Blum and Rivest , 1989 ) , but in practice they can be optimized with simple stochastic gradient descent algorithms to outmatch humans in tasks such as face verification ( Lu and Tang , 2015 ) and playing Atari-games ( Mnih et al. , 2015 ) . Recent work on understanding the geometry of neural network loss surfaces has promoted the idea of convexity properties . The work of Izmailov et al . ( 2018 ) shows that the loss surface is convex around the local optimum , while Zhou et al . ( 2019 ) and Kleinberg et al . ( 2018 ) show that the gradients during optimization typically point towards the local minima the network eventually converges to . Of central importance in this line of work is star-convexity , which is a property of a function f that guarantees that it is convex along straight paths towards the optima x˚ . See Figure 2 for an example . Formally , it is defined as follows . Definition 1 . A function f : Rn Ñ R is star-convex towards x˚ if for all λ P r0 , 1s and x P Rn , we have f ` λx ` p1´ λq x˚ ˘ ď λfpxq ` p1´ λqfpx˚q . Optimizing star-convex functions can be done in polynomial time ( Lee and Valiant , 2016 ) , in Kleinberg et al . ( 2018 ) it is shown how the function only needs to be star-convex under a natural noise model . NMF is not star-convex in general as it is NP-hard , however , it is natural to conjecture that NMF is starconvex in the typical case . Such a property could explain the practical success of NMF on real-world datasets , which is not worst-case . This will be the working hypothesis of this paper , where the typical case is formalized in Theorem 1 . Indeed , one can verify numerically that NMF is typically star-convex for natural distributions and realistically sized matrices , see Figure 1 where we consider a rank 10 decomposition of p100 , 100qmatrices with iid half-normal entries and a planted solution , sampled as per Assumption 1 given in the next section . Following sections will be dedicated to proving that NMF is starconvex with high probability in a planted model , and to confirm that this phenomenon generalizes to datasets from the real world , which are far from worst-case . 3 PROVING TYPICAL-CASE STAR-CONVEXITY . Our aim now is to prove that the NMF loss-function is typically star-convex for natural non-worstcase distribution of NMF instances . We will consider a slighly weaker notation of star-convexity where f ` λx ` p1 ´ λq x˚ ˘ ď λfpxq ` p1 ´ λqfpx˚q holds not for all x , but for random x with high probability . This is in fact the best achievable , an NMF instance with u1 “ 1 , u˚ “ 0 and v1 “ 0 , v˚ “ 1 is not star-convex . Our results hold with high probability in high dimensions , similar to Dvoretzky ’ s theorem in convex geometry ( Dvoredsky , 1961 ; Davis ) . Inspired by the stochastic block model of social networks ( Holland et al. , 1983 ; Decelle et al. , 2011 ) and the planted clique problem ( Barak et al. , 2016 ) , we will focus on a setting with a planted random solution . In section 4 we verify conclusions drawn from this model transfers to real-world datasets . We will assume that there is a planted optimal solution pU˚ , V˚q , where entries of these matrices are sampled iid from a class of distributions with good concentration properties that include the halfnormal distribution and bounded distributions . As is standard in random matrix theory ( Vershynin , 2010 ) , we will develop non-asymptotic results that hold with a probability that grows as the matrices of shape pn , rq and pr , mq become large . For this reason , we will need to specify how r and m depend on n. Assumption 1 . For pU , Vq P Rnˆr ˆ Rrˆm we assume that the entries of the matrices U , V are sampled iid from a continuous distribution with non-negative support that either piq is bounded or piiq can be expressed as a 1-Lipschitz function of a Gaussian distribution . As n Ñ 8 , we assume that r grows as nγ up to a constant factor for γ P r1 { 2 , 1s , and m as n up to a constant factor . We are now ready to state our main results , that the loss function equation 1 is convex on a straight line between points samples as per Assumption 1 , and thus satisfy our slightly weaker notion of star-convexity , with high probability . The probability increases as the size of the problem increases , suggesting a surprising benefit of high dimensionality . We also show similar results for the loss function of equation 2 with unobserved data under the assumption that the event that any entry is observed occurs independently with constant probability p. Below we sketch the proof idea and key ingredients , the formal proof is given in Appendix D. Theorem 1 . ( Main ) Let matrices U1 , V1 , U2 , V2 , U˚ , V˚ be sampled according to Assumption 1 . Then there exists positive constants c1 , c2 , such that with probability ě 1 ´ c1 expp´c2n1 { 3q , the loss function ` pU , Vq in equation 1 is convex on the straight line pU1 , V1q Ñ pU2 , V2q . The same holds along the line pU1 , V1q Ñ pU˚ , V˚q . It also holds if any entry pi , jq is observed independently with constant probability p , but with probability ě 1´ c1 expp´c2r1 { 3q .

Summary: This paper studies loss landscape of Non-negative matrix factorization (NMF) when the matrix is very large. It shows that with high probability, the landscape is quasi-convex under some conditions. This suggests that the optimization problem would become easier as the size of the matrix becomes very large. Implications on deep networks are also discussed.