Large-scale Structured Learning

In this thesis we study large-scale structured learning in the context of supervised, unsupervised and semisupervised settings. In the big data era, it is increasingly important to automatically infer structure from the data or leverage human provided structures in the learning process. The first part of this work focuses on how to leverage human provided structures, specifically hierarchical or graph based dependencies between classlabels, in a tractable manner to improve the classification performance. We propose two frameworks in this context (a) A Hierarchical Bayesian model that can leverage hierarchical dependencies using Bayesian priors (b) A non-Bayesian Risk minimization framework that incorporates hierarchical and graphical dependencies into the regularization structure. For both frameworks we develop fast inference and training methods that can easily handle large-scale structures with hundreds of thousands of classes and millions of dimensions. When hand-generated structures for organizing data is not readily available, we focus on unsupervised techniques to infer structures. Using the von Mises-Fisher distribution as the building block we propose three increasingly powerful Bayesian models that can recover flat, hierarchical as well as temporal structures from data on unit-sphere manifolds. Finally, we propose to incorporate human supervision and enable the use of partially available structures to improve structure discovery. We propose a suite of transformation based clustering methods to leverage such input from users in the process of learning the complete structure. New algorithms for large-scale inference and thorough evaluations on a broad range of data collections are significant parts of the thesis. Although we will predominantly focus on text data, the techniques and methods are generalizable and have been applied to non-text data as well. Chapter 1 Thesis Statement With the advent of big-data, there is a growing need to provide a structured and organized view of the data for effective search, browsing and data mining. The large taxonomies of Yahoo! for organizing all the web-pages on the web, the International patent classification hierarchy for easy browsing of patents, the Wikipedia graph that connects related Wikipedia categories with each other, and the Medical Subject Heading hierarchy for millions of articles in PubMed, are all examples of human generated structures to organize data. Such structures can provide valuable information while trying to classify incoming data. For example, consider the Yahoo! Directory for webpages, from the hierarchical structure between the class-labels one can say that a webpage that belongs to class ‘medicine’ is unlikely to also belong to class ‘mutual funds’. Such complex dependencies cannot be directly incorporated into simple one-against-rest classifiers as they lack the necessary expressive power. Moreover, most real world hierarchies and graphs are characterized by a large-number of class-labels. For example, the Yahoo! Directory comprises of approximately 300,000 classes arranged in a 16 level hierarchy, or the English Wikipedia which consists of approximately 325,000 classes with graphical dependencies. Training models on such large-scale structures, even with simple multiclass (K-way) classifiers is computationally challenging. How can we develop models that are expressive enough to leverage the dependencies between class-labels? and how can we learn them efficiently without losing tractability on large-scale structures? A thorough exploration of these questions will be the focus of the first part of the thesis. When such human-generated structures are not available, unsupervised discovery of the semantic structures from data is highly desirable for providing organized views of the data to end users. There are a wide variety of methods that already cater to such needs, including some of the most popular topic models such as Latent Dirichlet Allocation [14] , Hierarchical Topic model [12] , Dynamic Topic Models [15] and their several other variants [57],[56], [37]. However, most existing topic models have fundamental limitations in their ways of representing data, i.e., each document must be represented as a bag of discrete feature counts. On the other hand, in several supervised tasks such as classification and retrieval, continuous data representation with Tf-Idf term weighting (the product of the within-document term frequency and the inverse of the within-corpus frequency) and L2-normalization to a unit-sphere often yield significantly better performance than discrete count based representations. On one hand we have empirical success of such normalized representation and on the other hand we have a wide variety of graphical models that model data using a different representation. Can we bridge the gap between the two approaches ? Can we develop models that combine the strengths of both by developing new graphical models for the data on unit-sphere manifolds ? We answer these questions in the second part of the thesis. In the third part of the thesis, we investigate semi-supervised structure learning i.e. a middle ground between supervised learning and unsupervised learning of structures in the first and second part of the thesis. In reality, end users often have some partial information or incomplete structure of topics by which they would like to organize the data. For e.g. in a corpus of news-stories, some users might prefer to organize the data