Mini-Workshop: Mathematics of Machine Learning

This is a report for a mini-workshop on the mathematical theory of learning. The purpose of the workshop was to bring together internationally recognized experts and young researchers to discuss new approaches, trends, and problems of the area. Mathematics Subject Classification (2000): 68T05, 68Q32, 68T10, 62H30, 62G08, 62L12, 62M20, 47G10; International Mathematical Union: 10, 15. Introduction by the Organisers Learning theory encompasses the mathematical, statistical, and algorithmic aspects arising from problems that appear frequently when one aims at deducing prediction and classification rules from massive and often high-dimensional data. Such problems abound in an increasing number of diverse areas such as bioinformatics, computer vision, data mining, speech processing, and finance, in which learning methods have successfully been applied. As a result of this development, there is now a widely interdisciplinary and highly international community interested in all aspects of learning. During the last decade, the field of learning theory has witnessed an enormous advance and growth. This progress was both triggered and made possible by successfully merging quite different communities, such as the machine learning community, which traditionally resides in computer science and engineering, on the one hand and the mathematicians coming with diverse background as nonparametric statistics, high-dimensional geometry, theoretical computer science, etc. on the other hand. 2366 Oberwolfach Report 42/2011 While there are many conferences aiming at either applications of learning methods or the development of new learning methods, only very few conferences have focused on the mathematical aspects of learning, so far. The most prominent ones in this direction were probably the conferences “Mathematical Learning Theory” held in Barcelona (2004), Paris (2006), and Dagstuhl (2011). Furthermore, there have been only a few workshops that focused on one or two aspects of mathematical learning theory. Among the most important ones are probably the Oberwolfach workshops “Learning Theory and Approximation” (2008, 2012), “Sparse Recovery Problems in High Dimensions: Statistical Inference and Learning Theory” (2009). We believe that the proposed mini-workshop has an important role in this quickly evolving field. Moreover, the mini-workshop aims in a direction, the above mentioned workshops did or will not cover, and therefore, it nicely complements the MFO program in this field. Nonparametric classification and regression are probably the mathematically best understood learning problems since their rigorous statistical investigation dates back to the 1970’s. While at the beginning, mostly relatively simple statistical procedures were considered, this later changed, e.g. with the development of neural networks and kernel methods. At their beginning, the latter were crudely motivated by heuristics from computer science, but during the last decade they have been intensively studied, so that nowadays their mathematical theory, in particular for classification and regression, is quite mature, and the same is true for some other classification and regression methods. Therefore, the proposed miniworkshop aims at discussing learning problems and successful learning methods of machine learning that have not attracted as much attention from the mathematical community, yet. In particular, the following topics will be covered: Spectral methods for clustering and semi-supervised learning. : During the last decade, a new class of algorithms for cluster analysis and semi-supervised learning have been developed, which on the one hand, show promising results in applications, and on the other hand, have not been mathematically well-understood. These methods use a similarity measure between data points to construct a similarity graph, which is then used to compute the first eigenvectors of the associated graph Laplacian. While some progress has already been made in understanding these methods, the large-sample behavior of these methods is, so far, mostly unknown. However, recent results indicate that at least for some of the methods, spectral techniques that have already been developed for the analysis of kernel methods may be crucial for a deeper understanding. We will invite experts from both domains to discuss this question. Merging online and kernel methods. : While kernel methods often produce state-of-the-art results, their computation is rather expensive, which prohibits their use for data sets consisting of millions of data points, a quite common situation in recent practice. On the other hand, online algorithms, which are inherently fast, do not possess strong statistical guarantees such as universal consistency or minmax learning rates, as they Mini-Workshop: Mathematics of Machine Learning 2367 are typically investigated in a conditional worst-case scenario. However, in recent years, some progress has been made in merging the best of the two worlds, by either designing online inspired optimization algorithms for kernel methods, or “kernelizing” online algorithms. Our preliminary list of participants, contains experts for online learning and kernel methods, to achieve synergies. Prediction based on structured classes of experts: Online learning has been an increasingly active area of learning theory. In online learning one is interested in prediction algorithms that act as well as the best among a given class of reference forecasters (the so-called experts). Recently there a lot of research has been carried out in understanding large and structured classes of experts. Such problems give rise to interesting combinatorial, probabilistic, and algorithmic questions. Even though some interesting progress has been achieved recently, many important questions remain unanswered and one of the aims of the workshop is to discuss these problems. Prediction of stationary time series: The existing prediction algorithms are mainly analyzed for special parametric stochastic processes, where the optimality means mean square optimality. Here the problem is how to construct prediction rules, which are optimal with probability 1, i.e., for any stationary time series, the average of squared errors converges, almost surely, to that of the optimum, given by the Bayes predictor. Such algorithms can be introduced and studied by combining the principles of nonparametric statistics and machine learning. Mini-Workshop: Mathematics of Machine Learning 2369 Mini-Workshop: Mathematics of Machine Learning