Semi-supervised learning with side information: graph-based approaches
Authors: Rong Jin
Venue:  (2007)

In many real-world learning tasks, data examples are known not only by their input patterns, but also in other forms that are often refered to as "side information". Side information provides additional knowledge about the data, leaving hope for machine learning algorithms to gain more insight into the structure of data and thus perform better. However, the usual incomplete, sparse, and noisy nature of side information also poses challenges. This dissertation will present research work in semi-supervised learning with side information. Semi-supervised learning uses both labeled and unlabeled data in training. In this work, we follow the graph-based approaches that are able to explore the underlying structure of data by constructing a graph over data examples. Taking such an advantage of graph representation for semi-supervised learning, we propose to construct a graph over all labeled and unlabeled data and further incorporate side information into the graph. To effectively utilize side information against its incompleteness, sparseness, and noise, this work adopts a common theme in formalizing graph-based learning models, i.e., enforcing consistency over the graph. More specifically, consistency is maximized among data input patterns, supervised information (if any), side information, and predictions on the unlabeled data. Optimization approaches are taken to carry out the consistency enforcement, with objective functions that collect consistency measurement defined on all possible data pairs. Solutions to three generic learning tasks are presented to illustrate the proposed method of utilizing side information in a semi-supervised learning setting. Specifically, we study multi-label learning with class correlation information, to meet the challenge of a large number of classes and a small training set. To improve classification with link information, we propose a boosting framework that is able to improve the classification accuracy of any given supervised classifiers. Another boosting framework is also designed to boost any given clustering algorithm, with the help of pairwise constraints over the data. In addition, two applications in the area of information retrieval will be discussed. In the first application, we develop a maximum coherence framework to tackle the difficulty of query translation disambiguation in cross-language information retrieval, with a bilingual dictionary as side information. The proposed framework will also be explained as two-way graph partitioning. The second application is automatic extraction of question-answer pairs from Web FAQs, where the side information comes from human knowledge on the presentation regularity on Web FAQ pages. Correlated label propagations over a graph constructed for each FAQ page is shown to be an interpretation of the corresponding model. All the semi-supervised learning models proposed for the tasks and applications demonstrate the effectiveness of the consistency enforcement theme in exploiting side information for semi-supervised learning. Analysis shows that the proposed models are robust against the incompleteness, sparseness, and noise of side information, and retain the power of utilizing both labeled and unlabeled data for training as semi-supervised learning.