Regularized Learning in Multiple Tasks with Relationship

REGULARIZED LEARNING IN MULTIPLE TASKS WITH RELATIONSHIPS Anveshi Charuvaka, PhD George Mason University, 2015 Dissertation Director: Dr. Huzefa Rangwala We often encounter classification problems in real world as groups of tasks with complex interactions. In order to fully take advantage of the underlying information and deal with the unique aspects of these problems, we need methods than can utilize the available information in the learning process. In this thesis I present several methods to deal with the learning problems in domains with multiple tasks, with primary focus on large scale hierarchical classification and multi-task learning. Hierarchies form an integral part in information organization in several application areas ranging from protein function classification to document classification. Automated methods for classification into hierarchies can lessen the burden of manual curation or completely obviate the need for it. Research in hierarchical classification is quite mature and several methods have been developed for small scale classification tasks, however, scalability of these methods to large scale problems is one aspect that is being actively researched. In large scale classification, the number of classes, number of training examples as well and number of features can be so large that learning on commodity hardware can be prohibitively expensive. With growing popularity of cluster computing paradigms, it is becoming possible to scaleout computation using intelligent distributed methods. In many large scale classification problems, a significant number of classes are also plagued by problems such as insufficient training data or huge skew in class distributions. Large scale classification techniques should also try to mitigate these problem by leveraging data from related classes, for example, by using techniques inspired by the research in multi-task and transfer learning. In this regard, we have developed a distributed implementation of hierarchical regularization formulation which is scalable to very large scale hierarchical classification problems. Even though the hierarchical regularization is able to effectively tackle some of the challenges associated with large scale classification, it can be computationally expensive to train, compared to its binary alternatives in one-vs-rest settings. To address this shortcoming of recursive regularization, we also propose an alternative formulation using cost sensitive learning where the cost of misclassification are defined with respect to the hierarchy. This formulation allows us to tackle the class imbalance in training as well. Cost sensitive hierarchical learning scheme is able to improve the learning efficiency while at the same time improving the classification performance. In several domains we encounter multiple classification schemes over the same or similar data elements. These classification schema may provide different views and it is logical to exploit these different view in order to improve classification performance. One such prominent domain is protein structure classification, where multiple hierarchical classification schemes exist for protein structure classification. In this thesis, we also present techniques to exploit multi-task learning for learning across disparate classification hierarchies and show that the combined scheme is able to outperform learning on a single hierarchy. Several multi-task learning formulations make the assumption that all tasks under consideration are symmetrically related. However, this assumption may not hold true for a large number of cases. Even the information about task relatedness is often not known. When symmetric task relationship assumptions are naively used in learning, the classification performance can be negatively affected, a phenomenon known as negative transfer. Therefore, effective methods should only selectively transfer information between tasks that can improve performance. Several methods have been developed to infer task relationship structure to deal with this issue. In this thesis, we also present a method that extends task relationship inference techniques to max margin formulation of support vector machines. Chapter 1: Introduction