Learning Kernels for Multiple Predictive Tasks

In this thesis, we consider the paradigm of training several related learning problems (tasks) jointly. The goal is to synergize the related tasks by appropriate sharing of feature information within them and obtain a better generalization across all the tasks (as compared to learning them independently). A key concern while training multiple related tasks jointly is to learn which feature information should be shared among which tasks. Either sharing unhelpful feature information among related tasks or sharing any information with unrelated tasks may result in worsening of performance. The focus of this thesis revolves around this issue. Overall, we present a family of regularized risk minimization based convex formulations, of increasing generality, for learning features (kernels) in various settings involving multiple tasks. We begin with the question of which features to share among all the tasks. This popular setting assumes that all the tasks are related and the aim here is to learn a common feature representation shared by all the tasks. We pose this problem as that of learning a shared kernel. The kernel based framework allows us to implicitly learn task correlations in generic inner product feature spaces. The shared kernel is modeled as a linear combination of a given set of base kernels, the Multiple Kernel Learning setting. We propose a mixed-norm based formulation for learning the shared kernel as well as the prediction functions of all the tasks. We build on this approach and proceed to learn the shared kernel in a couple of richer feature spaces. Firstly, we consider a case where the tasks share low-dimensional subspaces lying in the induced feature spaces of the base kernels. We pose this problem as a mixed Schatten-norm regularized formulation that learns an optimal linear combination of such subspaces. Secondly, we consider learning the shared kernel as a polynomial combination of base kernels. In this setup, the space of possible combinations is exponential in the number of base kernels and we propose a graph based regularizer to perform a clever search in this huge space. i We next study the setting where all the tasks need not share a common feature representation. We develop a formulation that learns groups of related tasks in a given feature space. Each group of tasks is represented by a kernel and the problem of learning groups of related tasks is posed as a kernel learning problem. We generalize this setup and propose a formulation that learns which feature spaces are shared among which groups of tasks. We show that the overall performance improves considerably by learning this information. We end this thesis by discussing our exploration on the problem of computing the kernel selection path in the Multiple Kernel Learning (MKL) framework. Kernel selection path is the variation in the classification accuracy as the fraction of selected kernels is varied from null to unity. We study a generic family of MKL formulations and propose a methodology for computing their kernel selection path efficiently. In addition to the models developed for learning kernels for multiple tasks, derivations of specialized dual formulations as well as the algorithms proposed for solving them efficiently are also an important contribution of this thesis. In particular, solving mixed Schatten-norm regularized problems as well as searching for optimal polynomial kernel combination in exponentially large search space call for novel optimization methodologies. We show utility of the proposed formulations in a few learning applications such as Rule Ensemble Learning where the goal is to construct an ensemble of conjunctive propositional rules.