This paper provides a theoretical explanation on the clustering aspect of nonnegative matrix factorization (NMF). We prove that even without imposing orthogonality nor sparsity constraint on the basis and or coefficient matrix, NMF still can give clustering results, thus providing a theoretical support for many works, e.g., [1] and [2], that show the superiority of the standard NMF as a clustering method.
@cite_10 provides the theoretical analysis on the equivalences between orthogonal NMF to @math -means clustering for both rectangular data matrices and symmetric matrices. However as their proofs utilize the zero gradient conditions, the hidden assumptions (setting the Lagrange multipliers to zeros) are not revealed there. Actually it can be easily shown that their approach is the KKT conditions applied to the unconstrained version of eq. . Thus there is no guarantee that minimizing eq. by using the zero gradient conditions leads to the stationary point located on the nonnegative orthant as required by the objective.
@cite_10
2043545458
Let B i be deterministic real symmetric m × m matrices, and ξ i be independent random scalars with zero mean and “of order of one” (e.g., @math ). We are interested to know under what conditions “typical norm” of the random matrix @math is of order of 1. An evident necessary condition is @math , which, essentially, translates to @math ; a natural conjecture is that the latter condition is sufficient as well. In the paper, we prove a relaxed version of this conjecture, specifically, that under the above condition the typical norm of S N is @math : @math for all Ω > 0 We outline some applications of this result, primarily in investigating the quality of semidefinite relaxations of a general quadratic optimization problem with orthogonality constraints @math , where F is quadratic in X = (X 1,... ,X k ). We show that when F is convex in every one of X j , a natural semidefinite relaxation of the problem is tight within a factor slowly growing with the size m of the matrices @math . Covariance matrices from stationary time series are Toeplitz. Multichannel and multidimensional processes have covariance matrices of block Toeplitz form. In these cases and many other situations, one knows that the actual covariance matrix belongs to a particular subclass of covariance matrices. This paper discusses a method for estimating a covariance matrix of specified structure from vector samples of the random process. The theoretical foundation of the method is to assume that the random process is zero-mean multivariate Gaussian, and to find the maximum-likelihood covariance matrix that has the specified structure. An existence proof is given and the solution is interpreted in terms of a minimum-entropy principle. The necessary gradient conditions that must be satisfied by the maximum-likelihood solution are derived and unique and nonunique analytic solutions for some simple problems are presented. A major contribution of this paper is an iterative algorithm that solves the necessary gradient equations for moderate-sized problems with reasonable computational ease. Theoretical convergence properties of the basic algorithm are investigated and robust modifications discussed. In doing maximum-entropy spectral analysis of a sine wave in white noise from a single vector sample, this new estimation procedure causes no splitting of the spectral line in contrast to the Burg technique. Spectral clustering aims to partition a data set into several groups by using the Laplacian of the graph such that data points in the same group are similar while data points in different groups are dissimilar to each other. Spectral clustering is very simple to implement and has many advantages over the traditional clustering algorithms such as k-means. Non-negative matrix factorization (NMF) factorizes a non-negative data matrix into a product of two non-negative (lower rank) matrices so as to achieve dimension reduction and part-based data representation. In this work, we proved that the spectral clustering under some conditions is equivalent to NMF. Unlike the previous work, we formulate the spectral clustering as a factorization of data matrix (or scaled data matrix) rather than the symmetrical factorization of the symmetrical pairwise similarity matrix as the previous study did. Under the NMF framework, where regularization can be easily incorporated into the spectral clustering, we propose several non-negative and sparse spectral clustering algorithms. Empirical studies on real world data show much better clustering accuracy of the proposed algorithms than some state-of-the-art methods such as ratio cut and normalized cut spectral clustering and non-negative Laplacian embedding. HighlightsWe proved that the spectral clustering is equivalent to NMF.Under the NMF framework, we propose nonnegative sparse spectral clustering model.We propose several algorithms to solve the proposed models.Experiment results on real world data show much better performance. Motivated by some applications in signal processing and machine learning, we consider two convex optimization problems where, given a cone @math K, a norm @math ?·? and a smooth convex function @math f, we want either (1) to minimize the norm over the intersection of the cone and a level set of @math f, or (2) to minimize over the cone the sum of @math f and a multiple of the norm. We focus on the case where (a) the dimension of the problem is too large to allow for interior point algorithms, (b) @math ?·? is "too complicated" to allow for computationally cheap Bregman projections required in the first-order proximal gradient algorithms. On the other hand, we assume that it is relatively easy to minimize linear forms over the intersection of @math K and the unit @math ?·?-ball. Motivating examples are given by the nuclear norm with @math K being the entire space of matrices, or the positive semidefinite cone in the space of symmetric matrices, and the Total Variation norm on the space of 2D images. We discuss versions of the Conditional Gradient algorithm capable to handle our problems of interest, provide the related theoretical efficiency estimates and outline some applications.