A UNIFIED FRAMEWORK FOR UNSUPERVISED LEARNING

We describe a common framework under which a number of widely used, and seemingly disparate algorithms for unsupervised learning can be cast. In this framework, an unsupervised algorithm simply estimates the support of the distribution generating the data. The formulation makes clear the separation between the approximation and statistical components of the algorithms, leading to bounds on their performance. The analysis further allows to set the free parameters of the algorithms to optimize the performance bounds. We show a precise relation between the support estimation view, and the alternative (generative) view of unsupervised learning as approximating the data-generating distribution, establishing a formal relation between two interpretations of unsupervised learning. Interpretation of unsupervised learning. The problem of unsupervised learning is widely described as that of recovering, or finding structure (patterns) in data sampled from a probability distribution [6]. The importance of unsupervised learning is clear both from the availability of unlabeled data, as well as from its central role in practical supervised learning systems. In particular, due to the recent progress in sensory systems and other dataacquisition technology, the amount of unlabeled data is typically much more abundant, and often much more cheaply acquired, than labeled data. Even for supervised learning tasks, many state-of-the-art systems use a preliminary unsupervised stage, which is often of central importance [7, 13, 15]. From a theoretical perspective, it is widely accepted that an unsupervised algorithm that is able to accurately approximate the input data while reducing its dimensionality can have a significant impact on downstream supervised stages, whose finite sample performance may be subject to some form of curse of dimensionality [4]. A unified view. The structure inferred in unsupervised learning has been variously interpreted to be, possibly depending on the application, a clustering or partitioning of the data [9], a, possibly sparse, encoding of the data [12], or learning a lower-dimensional approximation (manifold learning) [1, 14]. All these interpretations have in common that they produce: 1) a compact representation, or estimate of the support of the data, in the form an approximating set, and 2) an associated encoding with respect to this representation. For instance, the k-means algorithm approximates the data by a discrete set of points (means), and encodes a sample by the index of its closest mean (in way that resembles vector quantization [5]). Similarly, in PCA, the computed approximating set is a lower-dimensional affine space, where each data point is encoded using its local coordinates in this space. Several other unsupervised learning algorithms, such as sparse coding [12], k-flats [2], or non-negative matrix factorization [8], can be easily shown to conform to this model [10, 11]. Empirical and expected error. In practice, an unsupervised learning algorithm typically has a tunable parameter that controls the size, or complexity of the approximating set. For example, this parameter may be the number of means in k-means, the dimension of the affine spaces in PCA, or the number of dictionary elements for sparse coding. Once the parameter is chosen, most algorithms, including all the ones mentioned above, proceed by computing the set that best approximates the available samples. In other words, they find the set that minimizes the distance from the samples to their projection onto the set (in some cases, with some additional constraint in the encoding, such as sparsity). Clearly, as the size of the set is increased, the distance from the available samples to the approximating set cannot increase, and typically decreases. However, since we are ultimately interested in the expected performance of the algorithm with respect to a random sample drawn from the true distribution, a question naturally arises of whether a good approximation of the samples necessarily implies a good approximation with respect to the distribution. Regularization in unsupervised learning and bias-variance tradeoff. In this work, we show that, in general, the answer to this question is negative, and thus that there is in general an optimal size/complexity for the approximating set. In other words there is in general a tradeoff in which small sets may be of insufficient complexity to accurately approximate the input, while sufficiently large sets that reproduce the empirical samples with high fidelity may fail to perform well on the true distribution. Indeed, we show that, for widely-used unsupervised algorithms, there is an inherent tradeoff between approximation accuracy and model size, which can be precisely