Information Theoretic Kernel Integration

In this paper we consider a novel information-theoretic app roach to multiple kernel learning based on minimising a Kullback-Leibler (KL) di vergence between the output kernel matrix and the input kernel matrix. There a r two formulations which we refer to asMKLdiv-dc andMKLdiv-conv. We propose to solve MKLdiv-dc by a difference of convex (DC) programming method and MKLdivconv by a projected gradient descent algorithm. The effecti v ness of the proposed approaches is evaluated on a benchmark dataset for protein f ld recognition and a yeast protein function prediction problem. 1 Information-theoretic Data Integration In this paper we consider the problem of integrating multipl e data sources using a kernel-based approach. Recent trends in learning kernel combination are us ually based on the margin maximization criterion used by Support Vector Machines (SVMs) or variant s [5, 8, 9, 10, 14, 16, 17]. There, each data source can be represented by x = {xi : i ∈ Nn} for ` ∈ Nm and the outputs are similarly denoted byy = {yi : i ∈ Nn}. With kernel methods, for anỳ∈ Nm, each̀ -th data source can be encoded into a candidate kernel matrix denoted by K` = (K`(xi , x ` j))ij . Depending on the type of data source used, the candidate kernel function K` would be specified a priori, as a graph kernel for graph data or a string kernel for sequence data, for example. Th composite kernel matrix is given byKλ = ∑ `∈Nm λ`K`. Hence, in this context the problem of data integration is red uc to learning a convex combination of candidate kernel matrices with kernel coefficients or weightsdenoted byλ. We can quantify the similarity between Kλ and the output kernel Ky through a Kullback-Leibler (KL) divergence or relative entropy term [3, 6, 7, 12, 13]. Th ere is a simple bijection between the set of distance measures in these data spaces and the set o f zero-mean multivariate Gaussian distributions [3]. Using this bijection, the difference be tw en two distance measures, parameterized by Kλ and Ky, can be quantified by the relative entropy or Kullback-Leibl r (KL) divergence between the corresponding multivariate Gaussians. Kernel matrices are generally positive semidefinite and thus can be regarded as the covariance matrices o f the Gaussian distributions. Matching kernel matricesKλ andKy, can therefore be realized by minimizing a KL divergence bet we n these two distributions. As described in [3, 6, 13], the Kull back-Leibler (KL) divergence (relative entropy) between a Gaussian distribution N (0,Ky) with the output covariance matrix Ky and a Gaussian distributionN (0,Kx) with the input kernel covariance matrix Kx is defined by KL ( N (0,Ky)||N (0,Kx) ) := 1 2 Tr(KyK −1 x ) + 1 2 log |Kx| − 1 2 log |Ky| − n 2 . (1) Here, the notation Tr (B) denotes its trace. Though KL ( N (0,Ky)||N (0,Kx) ) is non-convex w.r. t. Kx, it has a unique minimum at Kx = Ky if Ky is positive definite, suggesting that minimizing the above KL-divergence encourages Kx to approachKy. If the input kernel matrixKx is represented by a linear combination of m candidate kernel matrices, i.e. Kx = Kλ = ∑ `∈Nm λ`K`, the above 1 MKLdiv-dc MKLdiv-conv SimpleMKL VBKC MKL-RKDA All data sources 73.36 71.01 66.57 68.1± 1.2 68.40 Uniform weighted 68.40 68.40 68.14 − 66.06 Table 1: Performance with individual and all data sources. T he results of VBKC are cited from [2]. The results not employed there are denoted by ‘ −’. The best result for each kernel learning method is marked in bold. KL-divergence based kernel learning is reduced to the follo wing formulation: arg minλ∈4 KL (N (0,Ky)||N (0,Kλ)) = argminλ∈4 Tr (