Algorithms for minimally supervised learning

The past few decades have brought substantial progress in the mathematical analysis of supervised learning. This is a paradigm in which a learner is provided with a data set consisting of points x and their labels (or response values) y, and is tasked with finding a suitable classifier (or regressor) that maps x → y. There are many popular types of classifiers—decision trees, linear separators, boosted ensembles, and so on—and for almost all of these, there are by now fairly well-developed analyses of how they may be learned from data (the computational or algorithmic task of learning the function) and of how well they are likely to perform on future data (the statistical problem of generalization). Some good textbooks on this material include those of Mitchell (1997), Hastie et al. (2009), and Mohri et al. (2012). This broad theoretical understanding quickly vanishes, however, as one moves away from supervised learning towards some other paradigms of statistics and machine learning. Take clustering, for example: the problem of partitioning a data set into groups. Like supervised learning, this task has a long history—recall the work of Pearson (1894) on clustering crabs using Gaussian mixture models—and has been studied just as intensively. Yet even the most basic computational and statistical questions around clustering are unresolved. A common way to define a clustering task is via a cost function, such as k-means or the log likelihood of a mixture model. This seems like it might bring clarity to the computational aspect of clustering, but in fact these optimization problems are typically NP-hard, meaning that they are computationally intractable to solve optimally. The heuristics used in practice are generally hill-climbing methods whose solutions can be arbitrarily far from optimal in the worst case, and there is little understanding of how well they perform on average, or on “well-behaved” data sets. The statistical picture is equally murky. Consistency results are absent for most clustering methods, and in the few instances where they do exist, such as for the k-means cost function, they apply only if the estimator is able to optimally solve its (intractable) optimization problem. In these notes, we’ll start by covering two recent strains of work on clustering that address the computational and statistical questions raised above. They are promising beginnings, that combine efficient algorithms with strong statistical guarantees of convergence, and they open the door to a more general theoretical understanding of clustering. We will then move to another topic that has seen recent progress: exploiting the intrinsic dimensionality of data. High-dimensional spaces present many statistical challenges, most of all for nonparametric estimators, which in general require a number of samples exponential in the dimension in order to achieve a specified error rate. This would seem to entirely rule out nonparametric methods in today’s world, where data sets are growing steadily larger and higher-dimensional. Yet there has been a resurgence of interest in these methods, based on the insight that a lot of data have fewer “degrees of freedom” than their dimension would suggest. Perhaps the most popular manifestation of this idea is in the field of “manifold learning”, which looks for ways of handling data that are arbitrarily high dimensional but lie on, or near, a low-dimensional manifold. There are many elegant heuristics that embed a data set of this type in a lower-dimensional space while attempting to preserve certain types of local geometry such as geodesic distances. This embedding can be