Towards a Theory of Learning in High-Dimensional Spaces

With advances in computing technology, it is becoming feasible to consider automated learning tasks on high-dimensional data, i.e., data with very large numbers of variables. Such datasets are challenging for learning methods because of (1) the ‘curse of dimensionality’, that learning rates may be slow in high-dimensional problems, and (2) that the computational resources needed to work on such datasets can be very large. Developing theoretical results towards understanding such problems, specifically in the areas of classification and regression, is the focus of this thesis. This thesis takes two approaches towards considering the matter of performance in high-dimensional problems. In the first approach, the motivatation is that the data may lie on a lower-dimensional subspace. Towards the goal of utilizing that assumption, this thesis includes a chapter—which can be seen in a more general setting, not just a highdimensional one—regarding local learning. Specifically, a localizable learning method is defined as one that, when estimating a particular point based on a training set, behaves similarly to how it would have behaved had the training set been limited to only those elements that are close to the point being estimated; call a new learning method purposefully limited to those elements a ‘local version’ of the first. This is a generalization of previous definitions of local learning which were concerned with Empirical Risk Minimization (ERM) and the re-weighting of cost functions to emphasize local training points. We will consider a more generally-applicable set of definitions, in the context of which it is proven that, on the one hand, not every consistent learning method is localizable; yet, on the other hand, if regularization is applied to the local version, prompting the new term ‘approximately localizable’, then in fact every consistent learning method is approximately localizable. This general learning-theoretic result guarantees that we can take a consistent learning method and localize it without ending up with a non-consistent method. In addition, it is proven that consistency is logically equivalent to the combination of two properties, approximate localizability and correctly estimating the global mean. Thus, approximate localizability can be seen as the ‘local’ aspect of consistency, since the other property is a ‘global’ one. In the second approach towards investigating the matter of high-dimensional performance, the behavior of an important existing learning method, the Support Vector Machine (SVM), is considered on high-dimensional problems. This is of interest both because of the possibility of using SVMs on high-dimensional problems, and because the ‘kernel trick’ used in SVMs implies the use of a high-dimensional space even for low-dimensional problems. We consider a specific high-dimensional setup in which the variables are i.i. d standard normal, which allows analytic investigation. Both the number of samples and the number of variables are taken to infinity. We will see that the maximal-margin hyperplane (i.e., a hard-margin SVM) will tend to have all training examples as support vectors when the number of variables is large, specifically, greater than the square of the number of samples. Motivated by these theoretical results, local learning is then applied to a particular type of high-dimensional problem, that of text categorization. By working locally, some of the problems of high-dimensionality can be reduced, since we no longer need to train on high-dimensional datasets—local datasets are not only smaller, but also lowerdimensional. Experiments were conducted on the standard Reuters-21578 text categorization database. The results show that a localized SVM’s performance is greater than that of the K-Nearest-Neighbors algorithm (K-NN), while having significantly lower computational requirements than SVMs in certain types of problems. In addition, it is shown that ‘local information’ exists in small local training sets that is sufficient for performance close to that of global SVMs, and that that information is not fully utilized by K-NN.