Star-Convexity in Non-Negative Matrix Factorization

1 INTRODUCTION . Non-negative matrix factorization ( NMF ) is a ubiquitous technique for data analysis where one attempts to factorize a measurement matrix X into the product of non-negative matrices U , V ( Lee and Seung , 1999 ) . This simple problem has applications in recommender systems ( Luo et al. , 2014 ) , scientific analysis ( Berne et al. , 2007 ; Trindade et al. , 2017 ) , computer vision ( Gillis , 2012 ) , internet distance prediction ( Mao et al. , 2006 ) , audio processing ( Schmidt et al. , 2007 ) and many more domains . Often , the non-negativity is crucial for interpretability , for example , in the context of crystallography , the light sources , which are represented as matrix factors , have non-negative intensity ( Suram et al. , 2016 ) . Like many other non-convex optimization problems , finding the exact solution to NMF is NP-hard ( Pardalos and Vavasis , 1991 ; Vavasis , 2009 ) . NMF ’ s tremendous practical success is however at odds with such worst-case analysis , and simple algorithms based upon gradient descent are known to find good solutions in real-world settings ( Lee and Seung , 2001 ) . At the time when NMF was proposed , most analyses of optimization problems within machine learning focused on convex formulations such as SVMs ( Cortes and Vapnik , 1995 ) , but owing to the success of neural networks , non-convex optimization has experienced a resurgence in interest . Here , we revisit NMF from a fresh perspective , utilizing recent tools developed in the context of optimization in deep learning . Specifically , our main inspiration is the recent work of Kleinberg et al . ( 2018 ) and Zhou et al . ( 2019 ) that empirically demonstrate that gradients typically point towards the final minimizer for neural networks trained on real-world datasets and analyze the implications of such convexity properties for efficient optimization . In this paper , we show theoretically and empirically that a similar property called star-convexity holds in NMF . From a theoretical perspective , we consider an NMF instance with planted solution , inspired by the stochastic block model for social networks ( Holland et al. , 1983 ; Decelle et al. , 2011 ) and the planted clique problem studied in sum-of-squares literature ( Barak et al. , 2016 ) . We prove that between two points the loss is convex with high probability , and conclude that the loss surface is star-convex in the typical case — even if the loss is computed over unobserved data . From an empirical perspective , we verify that our theoretical results hold for an extensive collection < latexit sha1_base64= '' /JI5iynvNWOEKZjLq6ujHWhdsHw= '' > AAAB7nicbVDLSsNAFL2pr1pfVZduBovgqiQi1GXRjcsK9gFtKJPJpB06mYSZG6GEfoQbF4q49Xvc+TdO2yy09cDA4ZxzmXtPkEph0HW/ndLG5tb2Tnm3srd/cHhUPT7pmCTTjLdZIhPdC6jhUijeRoGS91LNaRxI3g0md3O/+8S1EYl6xGnK/ZiOlIgEo2il7kDaaEiH1Zpbdxcg68QrSA0KtIbVr0GYsCzmCpmkxvQ9N0U/pxoFk3xWGWSGp5RN6Ij3LVU05sbPF+vOyIVVQhIl2j6FZKH+nshpbMw0Dmwypjg2q95c/M/rZxjd+LlQaYZcseVHUSYJJmR+OwmF5gzl1BLKtLC7EjammjK0DVVsCd7qyeukc1X33Lr3cF1r3hZ1lOEMzuESPGhAE+6hBW1gMIFneIU3J3VenHfnYxktOcXMKfyB8/kDPMWPfQ== < /latexit > < latexit sha1_base64= '' /JI5iynvNWOEKZjLq6ujHWhdsHw= '' > AAAB7nicbVDLSsNAFL2pr1pfVZduBovgqiQi1GXRjcsK9gFtKJPJpB06mYSZG6GEfoQbF4q49Xvc+TdO2yy09cDA4ZxzmXtPkEph0HW/ndLG5tb2Tnm3srd/cHhUPT7pmCTTjLdZIhPdC6jhUijeRoGS91LNaRxI3g0md3O/+8S1EYl6xGnK/ZiOlIgEo2il7kDaaEiH1Zpbdxcg68QrSA0KtIbVr0GYsCzmCpmkxvQ9N0U/pxoFk3xWGWSGp5RN6Ij3LVU05sbPF+vOyIVVQhIl2j6FZKH+nshpbMw0Dmwypjg2q95c/M/rZxjd+LlQaYZcseVHUSYJJmR+OwmF5gzl1BLKtLC7EjammjK0DVVsCd7qyeukc1X33Lr3cF1r3hZ1lOEMzuESPGhAE+6hBW1gMIFneIU3J3VenHfnYxktOcXMKfyB8/kDPMWPfQ== < /latexit > < latexit sha1_base64= '' /JI5iynvNWOEKZjLq6ujHWhdsHw= '' > AAAB7nicbVDLSsNAFL2pr1pfVZduBovgqiQi1GXRjcsK9gFtKJPJpB06mYSZG6GEfoQbF4q49Xvc+TdO2yy09cDA4ZxzmXtPkEph0HW/ndLG5tb2Tnm3srd/cHhUPT7pmCTTjLdZIhPdC6jhUijeRoGS91LNaRxI3g0md3O/+8S1EYl6xGnK/ZiOlIgEo2il7kDaaEiH1Zpbdxcg68QrSA0KtIbVr0GYsCzmCpmkxvQ9N0U/pxoFk3xWGWSGp5RN6Ij3LVU05sbPF+vOyIVVQhIl2j6FZKH+nshpbMw0Dmwypjg2q95c/M/rZxjd+LlQaYZcseVHUSYJJmR+OwmF5gzl1BLKtLC7EjammjK0DVVsCd7qyeukc1X33Lr3cF1r3hZ1lOEMzuESPGhAE+6hBW1gMIFneIU3J3VenHfnYxktOcXMKfyB8/kDPMWPfQ== < /latexit > < latexit sha1_base64= '' /JI5iynvNWOEKZjLq6ujHWhdsHw= '' > AAAB7nicbVDLSsNAFL2pr1pfVZduBovgqiQi1GXRjcsK9gFtKJPJpB06mYSZG6GEfoQbF4q49Xvc+TdO2yy09cDA4ZxzmXtPkEph0HW/ndLG5tb2Tnm3srd/cHhUPT7pmCTTjLdZIhPdC6jhUijeRoGS91LNaRxI3g0md3O/+8S1EYl6xGnK/ZiOlIgEo2il7kDaaEiH1Zpbdxcg68QrSA0KtIbVr0GYsCzmCpmkxvQ9N0U/pxoFk3xWGWSGp5RN6Ij3LVU05sbPF+vOyIVVQhIl2j6FZKH+nshpbMw0Dmwypjg2q95c/M/rZxjd+LlQaYZcseVHUSYJJmR+OwmF5gzl1BLKtLC7EjammjK0DVVsCd7qyeukc1X33Lr3cF1r3hZ1lOEMzuESPGhAE+6hBW1gMIFneIU3J3VenHfnYxktOcXMKfyB8/kDPMWPfQ== < /latexit > < latexit sha1_base64= '' /JI5iynvNWOEKZjLq6ujHWhdsHw= '' > AAAB7nicbVDLSsNAFL2pr1pfVZduBovgqiQi1GXRjcsK9gFtKJPJpB06mYSZG6GEfoQbF4q49Xvc+TdO2yy09cDA4ZxzmXtPkEph0HW/ndLG5tb2Tnm3srd/cHhUPT7pmCTTjLdZIhPdC6jhUijeRoGS91LNaRxI3g0md3O/+8S1EYl6xGnK/ZiOlIgEo2il7kDaaEiH1Zpbdxcg68QrSA0KtIbVr0GYsCzmCpmkxvQ9N0U/pxoFk3xWGWSGp5RN6Ij3LVU05sbPF+vOyIVVQhIl2j6FZKH+nshpbMw0Dmwypjg2q95c/M/rZxjd+LlQaYZcseVHUSYJJmR+OwmF5gzl1BLKtLC7EjammjK0DVVsCd7qyeukc1X33Lr3cF1r3hZ1lOEMzuESPGhAE+6hBW1gMIFneIU3J3VenHfnYxktOcXMKfyB8/kDPMWPfQ== < /latexit > < latexit sha1_base64= '' /JI5iynvNWOEKZjLq6ujHWhdsHw= '' > AAAB7nicbVDLSsNAFL2pr1pfVZduBovgqiQi1GXRjcsK9gFtKJPJpB06mYSZG6GEfoQbF4q49Xvc+TdO2yy09cDA4ZxzmXtPkEph0HW/ndLG5tb2Tnm3srd/cHhUPT7pmCTTjLdZIhPdC6jhUijeRoGS91LNaRxI3g0md3O/+8S1EYl6xGnK/ZiOlIgEo2il7kDaaEiH1Zpbdxcg68QrSA0KtIbVr0GYsCzmCpmkxvQ9N0U/pxoFk3xWGWSGp5RN6Ij3LVU05sbPF+vOyIVVQhIl2j6FZKH+nshpbMw0Dmwypjg2q95c/M/rZxjd+LlQaYZcseVHUSYJJmR+OwmF5gzl1BLKtLC7EjammjK0DVVsCd7qyeukc1X33Lr3cF1r3hZ1lOEMzuESPGhAE+6hBW1gMIFneIU3J3VenHfnYxktOcXMKfyB8/kDPMWPfQ== < /latexit > < latexit sha1_base64= '' /JI5iynvNWOEKZjLq6ujHWhdsHw= '' > AAAB7nicbVDLSsNAFL2pr1pfVZduBovgqiQi1GXRjcsK9gFtKJPJpB06mYSZG6GEfoQbF4q49Xvc+TdO2yy09cDA4ZxzmXtPkEph0HW/ndLG5tb2Tnm3srd/cHhUPT7pmCTTjLdZIhPdC6jhUijeRoGS91LNaRxI3g0md3O/+8S1EYl6xGnK/ZiOlIgEo2il7kDaaEiH1Zpbdxcg68QrSA0KtIbVr0GYsCzmCpmkxvQ9N0U/pxoFk3xWGWSGp5RN6Ij3LVU05sbPF+vOyIVVQhIl2j6FZKH+nshpbMw0Dmwypjg2q95c/M/rZxjd+LlQaYZcseVHUSYJJmR+OwmF5gzl1BLKtLC7EjammjK0DVVsCd7qyeukc1X33Lr3cF1r3hZ1lOEMzuESPGhAE+6hBW1gMIFneIU3J3VenHfnYxktOcXMKfyB8/kDPMWPfQ== < /latexit > < latexit sha1_base64= '' /JI5iynvNWOEKZjLq6ujHWhdsHw= '' > AAAB7nicbVDLSsNAFL2pr1pfVZduBovgqiQi1GXRjcsK9gFtKJPJpB06mYSZG6GEfoQbF4q49Xvc+TdO2yy09cDA4ZxzmXtPkEph0HW/ndLG5tb2Tnm3srd/cHhUPT7pmCTTjLdZIhPdC6jhUijeRoGS91LNaRxI3g0md3O/+8S1EYl6xGnK/ZiOlIgEo2il7kDaaEiH1Zpbdxcg68QrSA0KtIbVr0GYsCzmCpmkxvQ9N0U/pxoFk3xWGWSGp5RN6Ij3LVU05sbPF+vOyIVVQhIl2j6FZKH+nshpbMw0Dmwypjg2q95c/M/rZxjd+LlQaYZcseVHUSYJJmR+OwmF5gzl1BLKtLC7EjammjK0DVVsCd7qyeukc1X33Lr3cF1r3hZ1lOEMzuESPGhAE+6hBW1gMIFneIU3J3VenHfnYxktOcXMKfyB8/kDPMWPfQ== < /latexit > of real-world datasets spanning collaborative filtering ( Zhou et al. , 2008 ; Kula , 2017 ; Harper and Konstan , 2016 ) , signal decomposition ( Zhu , 2016 ; Li and Ngom , 2013 ; Li et al. , 2001 ; Erichson et al. , 2018 ) and audio processing ( Flenner and Hunter , 2017 ; canto Foundation ) , and demonstrate that the star-convex behavior results in efficient optimization . Finally , we show that star-convex behavior becomes more likely with growing number of parameters , suggesting that a similar result may hold as neural networks become wider . We provide supporting empirical evidence for this hypothesis on modern network architectures . 2 NMF AND STAR-CONVEXITY . The aim of NMF is to decompose some large measurement matrix X P Rnˆm into two non-negative matrices U P Rnˆr ` and V P Rrˆm ` such that X « UV . The canonical formulation of NMF is min U , Vě0 ` pU , Vq , where ` pU , Vq “ 1 2 } UV ´X } 2F ( 1 ) NMF is commonly used in recommender systems where entries pi , jq of X for example correspond to the rating user i has given to movie j ( Luo et al. , 2014 ) . In such settings , data might be missing as all users have not rated all movies . In those cases , it is common to only consider the loss over observed data ( Zhang et al. , 2006 ; Candès and Recht , 2009 ) . We let 1̂pi , jq be an indicator variable that is 1 if entry pi , jq is `` observed '' and 0 otherwise . The loss function is then min U , Vě0 ` pU , Vq “ 1 2 ÿ i , j 1̂pi , jq ˆ “ UV ‰ ij ´Xij ˙2 ( 2 ) NMF is similar to PCA which admits spectral strategies ; however , the non-negative constraints in NMF prevent such solutions and result in NP-hardness ( Vavasis , 2009 ) . Work on the computational complexity of NMF has shown that the problem is tractable for small constant dimensions r via algebraic methods ( Arora et al. , 2012 ) . In practice , however , these algorithms are not used , and simple variants of gradient descent , possibly via multiplicative updates ( Lee and Seung , 2001 ) , are popular and are known to work reliably ( Koren et al. , 2009 ) . This gap between theoretical hardness and practical performance is also found in deep learning . Optimizing neural networks is NP-hard in general ( Blum and Rivest , 1989 ) , but in practice they can be optimized with simple stochastic gradient descent algorithms to outmatch humans in tasks such as face verification ( Lu and Tang , 2015 ) and playing Atari-games ( Mnih et al. , 2015 ) . Recent work on understanding the geometry of neural network loss surfaces has promoted the idea of convexity properties . The work of Izmailov et al . ( 2018 ) shows that the loss surface is convex around the local optimum , while Zhou et al . ( 2019 ) and Kleinberg et al . ( 2018 ) show that the gradients during optimization typically point towards the local minima the network eventually converges to . Of central importance in this line of work is star-convexity , which is a property of a function f that guarantees that it is convex along straight paths towards the optima x˚ . See Figure 2 for an example . Formally , it is defined as follows . Definition 1 . A function f : Rn Ñ R is star-convex towards x˚ if for all λ P r0 , 1s and x P Rn , we have f ` λx ` p1´ λq x˚ ˘ ď λfpxq ` p1´ λqfpx˚q . Optimizing star-convex functions can be done in polynomial time ( Lee and Valiant , 2016 ) , in Kleinberg et al . ( 2018 ) it is shown how the function only needs to be star-convex under a natural noise model . NMF is not star-convex in general as it is NP-hard , however , it is natural to conjecture that NMF is starconvex in the typical case . Such a property could explain the practical success of NMF on real-world datasets , which is not worst-case . This will be the working hypothesis of this paper , where the typical case is formalized in Theorem 1 . Indeed , one can verify numerically that NMF is typically star-convex for natural distributions and realistically sized matrices , see Figure 1 where we consider a rank 10 decomposition of p100 , 100qmatrices with iid half-normal entries and a planted solution , sampled as per Assumption 1 given in the next section . Following sections will be dedicated to proving that NMF is starconvex with high probability in a planted model , and to confirm that this phenomenon generalizes to datasets from the real world , which are far from worst-case . 3 PROVING TYPICAL-CASE STAR-CONVEXITY . Our aim now is to prove that the NMF loss-function is typically star-convex for natural non-worstcase distribution of NMF instances . We will consider a slighly weaker notation of star-convexity where f ` λx ` p1 ´ λq x˚ ˘ ď λfpxq ` p1 ´ λqfpx˚q holds not for all x , but for random x with high probability . This is in fact the best achievable , an NMF instance with u1 “ 1 , u˚ “ 0 and v1 “ 0 , v˚ “ 1 is not star-convex . Our results hold with high probability in high dimensions , similar to Dvoretzky ’ s theorem in convex geometry ( Dvoredsky , 1961 ; Davis ) . Inspired by the stochastic block model of social networks ( Holland et al. , 1983 ; Decelle et al. , 2011 ) and the planted clique problem ( Barak et al. , 2016 ) , we will focus on a setting with a planted random solution . In section 4 we verify conclusions drawn from this model transfers to real-world datasets . We will assume that there is a planted optimal solution pU˚ , V˚q , where entries of these matrices are sampled iid from a class of distributions with good concentration properties that include the halfnormal distribution and bounded distributions . As is standard in random matrix theory ( Vershynin , 2010 ) , we will develop non-asymptotic results that hold with a probability that grows as the matrices of shape pn , rq and pr , mq become large . For this reason , we will need to specify how r and m depend on n. Assumption 1 . For pU , Vq P Rnˆr ˆ Rrˆm we assume that the entries of the matrices U , V are sampled iid from a continuous distribution with non-negative support that either piq is bounded or piiq can be expressed as a 1-Lipschitz function of a Gaussian distribution . As n Ñ 8 , we assume that r grows as nγ up to a constant factor for γ P r1 { 2 , 1s , and m as n up to a constant factor . We are now ready to state our main results , that the loss function equation 1 is convex on a straight line between points samples as per Assumption 1 , and thus satisfy our slightly weaker notion of star-convexity , with high probability . The probability increases as the size of the problem increases , suggesting a surprising benefit of high dimensionality . We also show similar results for the loss function of equation 2 with unobserved data under the assumption that the event that any entry is observed occurs independently with constant probability p. Below we sketch the proof idea and key ingredients , the formal proof is given in Appendix D. Theorem 1 . ( Main ) Let matrices U1 , V1 , U2 , V2 , U˚ , V˚ be sampled according to Assumption 1 . Then there exists positive constants c1 , c2 , such that with probability ě 1 ´ c1 expp´c2n1 { 3q , the loss function ` pU , Vq in equation 1 is convex on the straight line pU1 , V1q Ñ pU2 , V2q . The same holds along the line pU1 , V1q Ñ pU˚ , V˚q . It also holds if any entry pi , jq is observed independently with constant probability p , but with probability ě 1´ c1 expp´c2r1 { 3q .

Summary: The paper derives results for nonnegative-matrix factorization along the lines of recent results on SGD for DNNs, showing that the loss is star-convex towards randomized planted solutions. The star-convexity property is also shown to hold to some degree on real world datasets. The paper argues that these results explain the good performance that usual gradient descent procedures achieve in practice. The paper also puts forward a conjecture that more parameters make the loss function easier to optimize by making it more likely that star convexity holds, and that a similar conclusion could hold for DNNs.