Proposal High-Dimensional Analysis of Unsupervised Learning Problems

High dimensional inference problems can present both statistical and computational challenges. In this thesis, we are concerned with developing computationally tractable methods for dimensionality reduction in unsupervised settings with optimal statistical properties. First, we consider the problem of learning Gaussian Mixtude Models (GMMs), under a sparsity assumption on the subspace separating the mixture components. Previous work has shown that, ignoring computational concerns, such sparsity conditions can reduce the sample complexity of mixture learning from polynomial to logarithmic in the dimension of the problem. Existing methods for sparse GMM learning either fail to guarantee such a reduction in sample complexity, or require the covariance of each mixture component to be spherical. We provide to our knowledge the first polynomial time algorithm which can resolve sparse Gaussian mixtures of non-spherical components, with statistical risk that is only logarithmic in ambient dimension. We valiade the algorithm empirically on several synthetic datasets. Second, we consider learning the principal subspace of a large set of vectors from an extremely small number of compressive measurements of each vector. We design and study an estimator based on back-projections of these compressive samples. We show that observing even a constant number of compressive measurement of each vector, we can consistently estimate the principal subspace to arbitrary precision, provided that the number of vectors is large. This result is achieved by a simple algorithm that computes the eigenvectors of an estimate of the covariance matrix. The main insight is to exploit an averaging effect that arises from applying a different random projection to each vector. We also consider the problem of detecting whether a point lies in a known subspace from compressive measurements, and design a method which succeeds with high probability given the number of compressive measurements is slightly larger than the dimension of the subspace. Next, we create a minimax framework for analyzing the performance of semisupervised regression methods. We define a smoothness condition on the regression function in terms of a density-sensitive distance metric, which corresponds to assuming that the function is smoother in regions of the explanatory variables with higher density. We prodive an estimator for this distance metric which utilizes only the unlabeled data, and show that kernel regression using this estimated metric has minimax optimal risk. Also, we characterize conditions under which any estimator which only uses the labeled data – i.e., a fully supervised method – is strictly outperformed by the semisupervised estimator. Intuitively, this occurs when the distribution of the explanatory variables lies on or near a low-dimensional manifold. Furthermore, we show that a cross-validation procedure can be used to adapt to an (unknown) strength parameter of the semisupervised smoothness assumption, so our method does not suffer when the assumption does not hold in practice. Finally, we study the risk of density mode clustering – a nonparametric clustering method where the clusters are the basins of attraction of the modes of a density estimator. We show that the risk of mode clustering over the high-density regions of the clusters can be small even in high dimensions. We also state a low noise condition under which the overall clustering risk is small in high dimensions even beyond the high density regions.