# Improving Localized Multiple Kernel Learning via Radius-Margin Bound

Authors: Xiaoming Wang; Zengxi Huang; Yajun Du
Journal: Mathematical Problems in Engineering
 (2017)

---

## Abstract

Localized multiple kernel learning (LMKL) is an effective method of multiple kernel learning (MKL). It tries to learn the optimal kernel from a set of predefined basic kernels by directly using the maximum margin principle, which is embodied in support vector machine (SVM). However, LMKL does not consider the radius of minimum enclosing ball (MEB) which actually impacts the error bound of SVM as well as the separating margin. In the paper, we propose an improved version of LMKL, which is named ILMKL. The proposed method explicitly takes into consideration both the margin and the radius and so achieves better performance over its counterpart. Moreover, the proposed method can automatically tune the regularization parameter when learning the optimal kernel. Consequently, it avoids using the time-consuming cross-validation process to choose the parameter. Comprehensive experiments are conducted and the results well demonstrate the effectiveness and efficiency of the proposed method.


---

## Body

## 1. Introduction

Over the past decade, kernel methods [1] have drawn a lot of attention of researchers in the machine learning community and have been widely applied. A kernel characterizes the similarity between two samples [2]. Actually, the performance of a kernel-based algorithm often strongly depends on the selection of the kernel. Generally, an unsuitable kernel would lead to a poor performance. Therefore, it is very critical to choose a suitable kernel for a kernel-based algorithm.Recent researches on kernel methods have highlighted the requirement to learn a suitable kernel matrix or function from the training data. A generic technique is known as multiple kernel learning (MKL) [3]. Given a set of predefined basic kernel functions, MKL tries to find their combination by employing a criterion which maximizes a generalization performance measure or minimizes an error bound. Actually, the practical problems frequently involve multiple heterogeneous data sources [4]. Thus, MKL is in accord with this fact. Many studies [5–13] have shown that MKL can generally find the suitable combination of basic kernel functions and so can usually achieve better performance in contrast with single kernel. The idea of MKL has been applied in all sorts of kernel-based algorithms, for example, support vector machine (SVM), which is a powerful and excellent machine learning method based on Vapnik’s statistical learning theory [14]. In the paper, we will only focus on the SVM-based MKL.Localized multiple kernel learning (LMKL) [15–17], as a method of MKL, is an attractive method which combines multiple heterogeneous attributes according to their discriminative ability for each individual instance. Generally, other MKL methods try to learn a global combination in the whole input space [2, 5, 6, 18, 19], whereas LMKL believes that a sample-specific local combination should most likely better reflect the distinctive characteristics of each instance and so embodies the idea. This is the key difference between other MKL methods and LMKL. Overall, LMKL consists of an SVM learning problem and a parametric gating model. The gating model is used to assign local weights to predefined basic kernels. In LMKL, a two-step alternate optimization method is employed to train the two components. In contrast with other MKL methods, LMKL generally provides fewer support vectors but can achieve statistically similar accuracy results. The idea of LMKL has been extended to other kernel-based methods and successfully used in some practical applications [20–23].However, LMKL learns the kernel function (essentially the parameters of the gating model) only by maximizing the margin which is embodied in single-kernel-based SVM. A key fact is that the generalization performance of SVM depends not only on the separating margin, but also on the radius of the smallest ball that encloses the data [24–28]. In fact, it is not necessary that standard SVM (or single-kernel SVM) exploits the radius. The reason is that the radius of the minimum enclosing ball (MEB) is fixed once the kernel including its parameters is selected. However, in the context of LMKL, the radius is not fixed but is a function of the parameters of the gating model.Actually, several attempts have recently been directed at incorporating the radius into SVM-based MKL [29]. However, most of these works direct SVM with l2 soft margin (l2-SVM) because the problem of l2-SVM can be transformed into a form of SVM with hard margin, in which the radius-margin bound holds and can be used to conduct model selection [25]. Unfortunately, for SVM with l1 soft margin (l1-SVM), the radius-margin bound does not hold, as we have no way of reducing the formulation of l1-SVM to a form of SVM with hard margin. So, one cannot directly utilize the radius-margin bound in LMKL since its formulation is rooted in l1-SVM. However, in [27], Chung et al. investigated several heuristic bounds for SVM and developed a modified radius-margin bound to conduct model selection for l1-SVM. The experimental results have indicated its effectiveness.Inspired by the work of Chung et al. in [27] and aiming at the drawback of LMKL, in this paper, we propose an improved version of LMKL, which is named ILMKL. A noticeable characteristic of the proposed method is that it takes account of both the separating margin and the radius of MEB; that is, it integrates the information of the margin and the radius to measure the goodness of the kernel and learn the parameters of the gating model. Actually, a key insight of our work is that learning the parameters of the gating model in LMKL is similar to conducting model selection. Analogous to LMKL, the problem of the proposed method can be efficiently resolved in a coupled manner through employing the two-step alternate optimization method. Moreover, the proposed method treats the regularization parameter as an extra parameter that can be automatically learned. Consequently, we can jointly tune it with the parameters of the gating during the learning kernel function process. This improves the computational efficiency of the proposed method to some extent since it avoids the time-consuming cross-validation process. Comprehensive experiments are conducted and the experimental results well demonstrate the efficiency and effectiveness of the proposed method.The rest of the paper is organized as follows. Section2 reviews the related work. In Section 3, we first present the formulation of the proposed method and then detail how to solve the optimization problem. After that, we conduct some preliminary discussion on the proposed method and outline the algorithm step. Section 4 reports the experimental results, and the conclusions are drawn in Section 5.


## 2. Preliminaries

In the paper, we suppose a training dataset which consists ofN samples and is represented by D={(x1,y1),…,(xN,yN)}, where the samples xi∈Rd and its corresponding labels yi∈{1,-1}. Here, i=1,…,N and d is the dimension of the sample space. Denote, for convenience, by I the set of all indices; that is, I={1,…,N}.


### 2.1. Radius-Margin Bound for SVM

SVM embodies the structural risk minimization principle, which is related to the probability of incorrectly classifying an unknown sample. Geometrically, the key idea of SVM is to construct a separating hyperplane in the data space through employing the maximal margin principle among two different classes of samples [14]. In the nonlinear case, l1-SVM defines the following optimization problem:(1)minw,b,ξ12w2+C∑i∈Iξis.t.yiw,φxi+b≥1-ξi,ξi≥0,i∈I,where φ(x):Rd→H, ·,· is the inner product of two vectors, ξi represents the training error, and C is the regularization parameter that adjusts the training error and the regularization term w2. Problem (1) can be efficiently solved by transforming it to its corresponding dual problem [30], which is formulated as(2)maxα∑i=1Nαi-12∑i=1N∑J=1Nαiαjyiyjkxi,xjs.t.∑i=1Nyiαi=0,C≥αi≥0,i∈I.Here, k(xi,xj)=φ(xi),φ(xj) is called kernel function. Suppose that α∗=[α1∗,…,αN∗]T solves the above optimization problem and b∗ is the optimal threshold which can be computed by using the KKT condition of (1); the decision function of SVM is formulated as(3)fx=∑i∈Iαi∗yikxi,x+b∗.In order to obtain a better performance in the practical applications, it is very important to choose suitable hyperparameters which include the regularization parameterC and the kernel parameter of the kernel function k(·,·) for SVM. This is the so-called model selection [24]. Generally, one can empirically set these hyperparameters. But this is very hard work because one cannot know in advance the suitable hyperparameters when facing all kinds of practical applications. Many works have tried to find a good criterion to automatically learn the related hyperparameters [24–26]. In [27], Chung et al. proposed the following bound for l1-SVM:(4)R2+ΔC12w2+C∑i∈Iξi,where Δ is a parameter and can generally be set as Δ=1 and R refers to the radius of MEB. This radius-margin bound is differentiable and is successfully used to conduct model selection for l1-SVM in [27].


### 2.2. Localized Multiple Kernel Learning

In the context of MKL, we assume that there existM different mappings φm(x):Rd→Hm (m=1,…,M), the mth mapping of which is endowed with the base kernel km(·,·) of associated reproducing kernel Hilbert space (RKHS) Hm.As a method of MKL, LMKL is based onl1-SVM and defines its optimization problem as follows:(5)minw,b,ξ,vm12w2+C∑i=1Nξis.t.yiw⌢,φ⌢xi+b≥1-ξi,ξi≥0,i∈I,where w⌢=w1;…;wm, φ⌢(xi)=η1(xi)φ1(xi);…;ηM(xi)φM(xi), b∈R, C is a regularization parameter that adjusts the training error and the regularization term w⌢2, and ξi represents the training error. Here, wm∈Hm, φm(x)∈Hm, and ηm(x) is a gating function defined up to a set of parameters which need to be learned from the training data. Further, by using duality, we have the dual formulation of the primal problem in (5) as follows:(6)maxα∑i=1Nαi-12∑i=1N∑J=1Nαiαjyiyjk⌢xi,xjs.t.∑i=1Nyiαi=0,C≥αi≥0,i∈I,where the locally combined kernel function is defined as(7)k⌢xi,xj=φ⌢xi,φ⌢xj=∑m=1Mηmxiηmxjkmxi,xj.If the used gating model ηm(x) is constant (not a function of x), LMKL finds a fixed combination over the whole input space and is similar to the original MKL formulation. The main advantage of LMKL is that it can achieve statistically similar accuracy results by storing fewer support vectors compared with the original MKL.


## 2.1. Radius-Margin Bound for SVM

SVM embodies the structural risk minimization principle, which is related to the probability of incorrectly classifying an unknown sample. Geometrically, the key idea of SVM is to construct a separating hyperplane in the data space through employing the maximal margin principle among two different classes of samples [14]. In the nonlinear case, l1-SVM defines the following optimization problem:(1)minw,b,ξ12w2+C∑i∈Iξis.t.yiw,φxi+b≥1-ξi,ξi≥0,i∈I,where φ(x):Rd→H, ·,· is the inner product of two vectors, ξi represents the training error, and C is the regularization parameter that adjusts the training error and the regularization term w2. Problem (1) can be efficiently solved by transforming it to its corresponding dual problem [30], which is formulated as(2)maxα∑i=1Nαi-12∑i=1N∑J=1Nαiαjyiyjkxi,xjs.t.∑i=1Nyiαi=0,C≥αi≥0,i∈I.Here, k(xi,xj)=φ(xi),φ(xj) is called kernel function. Suppose that α∗=[α1∗,…,αN∗]T solves the above optimization problem and b∗ is the optimal threshold which can be computed by using the KKT condition of (1); the decision function of SVM is formulated as(3)fx=∑i∈Iαi∗yikxi,x+b∗.In order to obtain a better performance in the practical applications, it is very important to choose suitable hyperparameters which include the regularization parameterC and the kernel parameter of the kernel function k(·,·) for SVM. This is the so-called model selection [24]. Generally, one can empirically set these hyperparameters. But this is very hard work because one cannot know in advance the suitable hyperparameters when facing all kinds of practical applications. Many works have tried to find a good criterion to automatically learn the related hyperparameters [24–26]. In [27], Chung et al. proposed the following bound for l1-SVM:(4)R2+ΔC12w2+C∑i∈Iξi,where Δ is a parameter and can generally be set as Δ=1 and R refers to the radius of MEB. This radius-margin bound is differentiable and is successfully used to conduct model selection for l1-SVM in [27].


## 2.2. Localized Multiple Kernel Learning

In the context of MKL, we assume that there existM different mappings φm(x):Rd→Hm (m=1,…,M), the mth mapping of which is endowed with the base kernel km(·,·) of associated reproducing kernel Hilbert space (RKHS) Hm.As a method of MKL, LMKL is based onl1-SVM and defines its optimization problem as follows:(5)minw,b,ξ,vm12w2+C∑i=1Nξis.t.yiw⌢,φ⌢xi+b≥1-ξi,ξi≥0,i∈I,where w⌢=w1;…;wm, φ⌢(xi)=η1(xi)φ1(xi);…;ηM(xi)φM(xi), b∈R, C is a regularization parameter that adjusts the training error and the regularization term w⌢2, and ξi represents the training error. Here, wm∈Hm, φm(x)∈Hm, and ηm(x) is a gating function defined up to a set of parameters which need to be learned from the training data. Further, by using duality, we have the dual formulation of the primal problem in (5) as follows:(6)maxα∑i=1Nαi-12∑i=1N∑J=1Nαiαjyiyjk⌢xi,xjs.t.∑i=1Nyiαi=0,C≥αi≥0,i∈I,where the locally combined kernel function is defined as(7)k⌢xi,xj=φ⌢xi,φ⌢xj=∑m=1Mηmxiηmxjkmxi,xj.If the used gating model ηm(x) is constant (not a function of x), LMKL finds a fixed combination over the whole input space and is similar to the original MKL formulation. The main advantage of LMKL is that it can achieve statistically similar accuracy results by storing fewer support vectors compared with the original MKL.


## 3. The Proposed LMKL Framework

In this section, we first present the primal optimization problem of the proposed method ILMKL and then detail how to solve it, and finally some preliminary discussion on the proposed method is given and the algorithm is outlined.


### 3.1. Primal Optimization Model of the Proposed Method

In the context of LMKL, it is easy to find that the radius of MEB is not fixed but is a function of the parameters of the gating model. Nevertheless, LMKL learns the parameters of the gating model only through using the separating margin. Therefore, LMKL ignores the fact that the generalization performance of SVM depends not only on the separating margin but also on the radius.Actually, the purpose of learning the parameters of the gating model is in essence to yield an appropriate kernel matrix for good performance. In our opinion, this process is similar to model selection by which SVM chooses the appropriate parameters to achieve good performance. Therefore, following the basic idea of the work in [27], we define the primal optimization problem of ILMKL as follows:(8)minw^,b,ξ,a,R,vm,CR2+ΔC12w⌢2+C∑i=1Nξis.t.yiw⌢,φ⌢xi+b≥1-ξi,ξi≥0,i∈Iφ⌢xi-a2≤R2,i∈I.As in Section 2.2, here, w⌢=w1;…;wm, φ⌢(xi)=η1(xi)φ1(xi);…;ηM(xi)φM(xi), wm∈Hm, b∈R, φm(x)∈Hm, ξi represents the training error, and C is a regularization parameter that adjusts the training error and the regularization term w⌢2. Note that w⌢2=∑m=1Mwm2 and w⌢,φ⌢(xi)=∑m=1Mwm,φm(xi), and thus we can reformulate (8) into(9)minwm,b,ξ,a,R,vm,CR2+ΔC12∑m=1Mwm2+C∑i=1Nξis.t.yi∑m=1Mηmxiwm,φmxi+b≥1-ξi,ξi≥0,i∈I∑m=1Mηmxiwm,φmxi-a2≤R2,i∈I.Here, ηm(x) is a gating function. As in [15], we employ the softmax gating model determining the parameters vm (m=1,…,M) and it can be expressed as(10)ηmxi=exp⁡vm0+∑d=1Dvmdxid∑p=1Mexp⁡vp0+∑d=1Dvpdxid=exp⁡vm,x~i∑p=1Mexp⁡vp,x~i,where xid is the dth feature of the ith sample, x~i=[1,xi], and vm=[vm0,vm1,vm2,…,vmM]∈RM+1 (m=1,…,M) are the parameters of the gating model associated with themth kernel and the softmax guarantees nonnegativity. As pointed out in [17], one can use more complex gating models.Obviously, in contrast with LMKL (5), ILMKL has two noticeable characteristics. One is that it takes into consideration the information of the radius and margin. Another characteristic of the proposed method is that it treats the regularization parameter C as a variable that can be automatically learned during the procedure of learning the parameters of the gating model. To sum up, the key insight of our method is that, in the context of LMKL, learning the parameters of the gating model is similar to conducting model selection for SVM.


### 3.2. Training with Alternating Optimization

Generally, it is very difficult to directly solve problem (9). In LMKL, a two-step alternate optimization method is employed to find the parameters of the gating model and the discriminant function. In our method ILMKL, we use the same strategy.The first step is to fix{vm,C} and solve (9) with respect to {wm,b,ξ,a,R}, and the second step is to optimize the parameters of {vm,C} by using a gradient-descent method. The objective value obtained for a fixed {vm,C} is an upper bound for (9) and the parameters of {vm,C} are optimized according to the current solution. The objective value obtained at the next iteration cannot be greater than the current one due to the use of gradient-descent procedure. And as iterations progress with a proper step size selection procedure, the objective value of (9) never increases. Note that this way does not guarantee convergence to the global optimum and so the initial parameters of {vm,C} may affect the solution quality.In this subsection, we will discuss how to solve problem (9) when fixing {vm,C}. In the following two subsections, we will, respectively, discuss how to optimize vm and C.For a fixed{vm,C}, we have(11)minwm,b,ξ,a,R,vm,CR2+ΔC12∑m=1Mwm2+C∑i=1Nξi=mina,R⁡R2+ΔC×minwm,b,ξ⁡12∑m=1Mwm2+C∑i=1Nξi.Here, we set(12)Ivm,C=mina,R⁡R2+ΔC×minwm,b,ξ⁡12∑m=1Mwm2+C∑i=1NξiJ1vm,C=mina,R⁡R2+ΔCJ2vm,C=minwm,b⁡12∑m=1Mwm2+C∑i=1Nξi.Therefore, for a fixed {vm,C}, problem (9) of the proposed method can be expressed as(13)minwm,b,ξ,a,RIvm,C=J1vm,C×J2vm,C,whereJ1vm,C=minwm,b,a,RR2+ΔCs.t.∑m=1Mηmxiwm,φmxi-a2≤R2,i∈I,J2vm,C=minwm,b12∑m=1Mwm2+C∑i=1Nξis.t.yi∑m=1Mηmxiwm,φmxi+b≥1-ξi,ξi≥0,i∈I.Note that if we fix the gating model parameters and the regularization parameter, the optimization problem (13) becomes convex. In order to find its solution, we can switch it into the dual optimization problem. By using duality, the dual problem of the primal problem in (13) can be formulated as(14)minwm,b,ξ,a,RIvm,C=J1vm,C×J2vm,C,whereJ1vm,C=ΔC+maxβ∑i=1Nβikxi,xi-∑i=1N∑J=1Nβiβjk⌢xi,xjs.t.∑i=1Nβi=1,βi≥0,i∈I,J2vm,C=maxα∑i=1Nαi-12∑i=1N∑J=1Nαiαjyiyjk⌢xi,xjs.t.∑i=1Nyiαi=0,C≥αi≥0,i∈I,where the locally combined kernel function k⌢(xi,xj) is defined as (7). Obviously, this formulation corresponds to, respectively, solving a canonical SVM dual problem and a canonical support vector domain description (SVDD) [31] dual problem with the kernel matrix K⌢(η)=k⌢(xi,xj)N×N, which should be positive semidefinite.Finally, once the final gating modelηm(x) has been learned and problem (14) is solved, the resulting discriminant function of the proposed method ILMKL can be expressed as follows:(15)fx=sign∑i=1N∑m=1Mαiηmxiηmxkmxi,x+b.


### 3.3. Optimizing the Parameters of the Gating Model

In order to optimize the parametersvm of the gating model ηm(x) by using a gradient-descent method, one needs to calculate the derivatives of the primal objective with respect to the parameters vm. Next, we will discuss how to calculate the derivatives of the parameters.First, note that(16)∂expvk,x~∂vm=0,k≠mexp⁡vk,x~x~,k=m=δmkexpvk,x~x~.So, we have(17)∂ηkx∂vm=δmkηkxx~-ηkxηmxx~.Thus, we can calculate the derivatives of k(xi,xj) with respect to the gating model parameters vm as follows:(18)∂kxi,xj∂vm=∑k=1Mkkxi,xjηkxj∂ηkxi∂vm+ηkxi∂ηkxj∂vm.Further, according to the above formula, the following can be obtained:(19)∂J1vm,C∂vm=∑i=1Nβi∂kxi,xi∂vm-∑i=1N∑J=1Nβiβj∂kxi,xj∂vm∂J2vm,C∂vm=-12∑i=1N∑J=1Nαiαjyiyj∂kxi,xj∂vm.Finally, the derivatives of I(vm,C) with respect to the parameters vm of the gating model ηm(x) can be formulated as(20)∂Ivm,C∂vm=J2vm,C∂J1vm,C∂vm+J1vm,C∂J2vm,C∂vm.


### 3.4. Optimizing the Regularization Parameter

In our method, the regularization parameterC is treated as a variable that can be learned when learning the gating model. Similar to the process of optimizing the parameter of the gating model, we employ a gradient-descent method to optimize the regularization parameter C and so the derivative of (14) with respect to C is needed. In the following, we will discuss how to compute the derivative.Actually, the derivative of (14) I(vm,C) with respect to C can be expressed as(21)∂Ivm,C∂C=J2vm,C∂J1vm,C∂C+J1vm,C∂J2vm,C∂C.Obviously, one obtains in advance ∂(J1(vm,C))/∂C and ∂(J2(vm,C))/∂C, which are, respectively, computed as(22)∂J1vm,C∂C=∂R2+Δ/C∂C=∂Δ/C∂C=-ΔC2,∂J2vm,C∂C=1C∑i=1Nαi-∑i=1N∑J=1Nαiαjyiyjkxi,xj=1CJ2vm,C.


### 3.5. Discussion

It should be noted thatC>0 must hold in the whole procedure. However, actually in the iterations, this condition may be broken. In order to deal with this problem, following [27], we can use ln⁡(C) instead of C in the solving procedure. The reason is that ln⁡(C) can be any real number when C>0. Thus, the positivity of C is dodged. Here, we need to rewrite the above partial derivatives. According to the chain rules, we can modify (21) as the following:(23)∂Ivm,C∂ln⁡C=J2vm,C∂J1vm,C∂ln⁡C+J1∂J2vm,C∂ln⁡C,where(24)∂J1vm,C∂lnC=C∂J1vm,C∂C,∂J2vm,C∂ln⁡C=C∂J2vm,C∂C.Finally, according to the above discussion, we outline the complete algorithm of ILMKL in Algorithm1.Algorithm 1:ILMKL.
(1) Initialize lnCinit and initialize vm to small random numbers for m=1,…,M; (2) whilestopping criterion not met do (3)    Compute C with C=exp⁡(ln⁡(C)); (4)    Calculate k⌢(xi,xj) with the gating model according to (7) when fixing vm; (5)    Compute I(vm,C) by using an canonical SVM solver and an canonical SVDD solver with k⌢(xi,xj) according to (14); (6)    Compute ∂(I(vm,C))/∂vm for m=1,…,M with (20); (7)    Compute ∂I(vm,C)/∂ln⁡(C) with (23); (8)    Update vm and ln⁡(C) by the gradient-descent method; (9) end while


## 3.1. Primal Optimization Model of the Proposed Method

In the context of LMKL, it is easy to find that the radius of MEB is not fixed but is a function of the parameters of the gating model. Nevertheless, LMKL learns the parameters of the gating model only through using the separating margin. Therefore, LMKL ignores the fact that the generalization performance of SVM depends not only on the separating margin but also on the radius.Actually, the purpose of learning the parameters of the gating model is in essence to yield an appropriate kernel matrix for good performance. In our opinion, this process is similar to model selection by which SVM chooses the appropriate parameters to achieve good performance. Therefore, following the basic idea of the work in [27], we define the primal optimization problem of ILMKL as follows:(8)minw^,b,ξ,a,R,vm,CR2+ΔC12w⌢2+C∑i=1Nξis.t.yiw⌢,φ⌢xi+b≥1-ξi,ξi≥0,i∈Iφ⌢xi-a2≤R2,i∈I.As in Section 2.2, here, w⌢=w1;…;wm, φ⌢(xi)=η1(xi)φ1(xi);…;ηM(xi)φM(xi), wm∈Hm, b∈R, φm(x)∈Hm, ξi represents the training error, and C is a regularization parameter that adjusts the training error and the regularization term w⌢2. Note that w⌢2=∑m=1Mwm2 and w⌢,φ⌢(xi)=∑m=1Mwm,φm(xi), and thus we can reformulate (8) into(9)minwm,b,ξ,a,R,vm,CR2+ΔC12∑m=1Mwm2+C∑i=1Nξis.t.yi∑m=1Mηmxiwm,φmxi+b≥1-ξi,ξi≥0,i∈I∑m=1Mηmxiwm,φmxi-a2≤R2,i∈I.Here, ηm(x) is a gating function. As in [15], we employ the softmax gating model determining the parameters vm (m=1,…,M) and it can be expressed as(10)ηmxi=exp⁡vm0+∑d=1Dvmdxid∑p=1Mexp⁡vp0+∑d=1Dvpdxid=exp⁡vm,x~i∑p=1Mexp⁡vp,x~i,where xid is the dth feature of the ith sample, x~i=[1,xi], and vm=[vm0,vm1,vm2,…,vmM]∈RM+1 (m=1,…,M) are the parameters of the gating model associated with themth kernel and the softmax guarantees nonnegativity. As pointed out in [17], one can use more complex gating models.Obviously, in contrast with LMKL (5), ILMKL has two noticeable characteristics. One is that it takes into consideration the information of the radius and margin. Another characteristic of the proposed method is that it treats the regularization parameter C as a variable that can be automatically learned during the procedure of learning the parameters of the gating model. To sum up, the key insight of our method is that, in the context of LMKL, learning the parameters of the gating model is similar to conducting model selection for SVM.


## 3.2. Training with Alternating Optimization

Generally, it is very difficult to directly solve problem (9). In LMKL, a two-step alternate optimization method is employed to find the parameters of the gating model and the discriminant function. In our method ILMKL, we use the same strategy.The first step is to fix{vm,C} and solve (9) with respect to {wm,b,ξ,a,R}, and the second step is to optimize the parameters of {vm,C} by using a gradient-descent method. The objective value obtained for a fixed {vm,C} is an upper bound for (9) and the parameters of {vm,C} are optimized according to the current solution. The objective value obtained at the next iteration cannot be greater than the current one due to the use of gradient-descent procedure. And as iterations progress with a proper step size selection procedure, the objective value of (9) never increases. Note that this way does not guarantee convergence to the global optimum and so the initial parameters of {vm,C} may affect the solution quality.In this subsection, we will discuss how to solve problem (9) when fixing {vm,C}. In the following two subsections, we will, respectively, discuss how to optimize vm and C.For a fixed{vm,C}, we have(11)minwm,b,ξ,a,R,vm,CR2+ΔC12∑m=1Mwm2+C∑i=1Nξi=mina,R⁡R2+ΔC×minwm,b,ξ⁡12∑m=1Mwm2+C∑i=1Nξi.Here, we set(12)Ivm,C=mina,R⁡R2+ΔC×minwm,b,ξ⁡12∑m=1Mwm2+C∑i=1NξiJ1vm,C=mina,R⁡R2+ΔCJ2vm,C=minwm,b⁡12∑m=1Mwm2+C∑i=1Nξi.Therefore, for a fixed {vm,C}, problem (9) of the proposed method can be expressed as(13)minwm,b,ξ,a,RIvm,C=J1vm,C×J2vm,C,whereJ1vm,C=minwm,b,a,RR2+ΔCs.t.∑m=1Mηmxiwm,φmxi-a2≤R2,i∈I,J2vm,C=minwm,b12∑m=1Mwm2+C∑i=1Nξis.t.yi∑m=1Mηmxiwm,φmxi+b≥1-ξi,ξi≥0,i∈I.Note that if we fix the gating model parameters and the regularization parameter, the optimization problem (13) becomes convex. In order to find its solution, we can switch it into the dual optimization problem. By using duality, the dual problem of the primal problem in (13) can be formulated as(14)minwm,b,ξ,a,RIvm,C=J1vm,C×J2vm,C,whereJ1vm,C=ΔC+maxβ∑i=1Nβikxi,xi-∑i=1N∑J=1Nβiβjk⌢xi,xjs.t.∑i=1Nβi=1,βi≥0,i∈I,J2vm,C=maxα∑i=1Nαi-12∑i=1N∑J=1Nαiαjyiyjk⌢xi,xjs.t.∑i=1Nyiαi=0,C≥αi≥0,i∈I,where the locally combined kernel function k⌢(xi,xj) is defined as (7). Obviously, this formulation corresponds to, respectively, solving a canonical SVM dual problem and a canonical support vector domain description (SVDD) [31] dual problem with the kernel matrix K⌢(η)=k⌢(xi,xj)N×N, which should be positive semidefinite.Finally, once the final gating modelηm(x) has been learned and problem (14) is solved, the resulting discriminant function of the proposed method ILMKL can be expressed as follows:(15)fx=sign∑i=1N∑m=1Mαiηmxiηmxkmxi,x+b.


## 3.3. Optimizing the Parameters of the Gating Model

In order to optimize the parametersvm of the gating model ηm(x) by using a gradient-descent method, one needs to calculate the derivatives of the primal objective with respect to the parameters vm. Next, we will discuss how to calculate the derivatives of the parameters.First, note that(16)∂expvk,x~∂vm=0,k≠mexp⁡vk,x~x~,k=m=δmkexpvk,x~x~.So, we have(17)∂ηkx∂vm=δmkηkxx~-ηkxηmxx~.Thus, we can calculate the derivatives of k(xi,xj) with respect to the gating model parameters vm as follows:(18)∂kxi,xj∂vm=∑k=1Mkkxi,xjηkxj∂ηkxi∂vm+ηkxi∂ηkxj∂vm.Further, according to the above formula, the following can be obtained:(19)∂J1vm,C∂vm=∑i=1Nβi∂kxi,xi∂vm-∑i=1N∑J=1Nβiβj∂kxi,xj∂vm∂J2vm,C∂vm=-12∑i=1N∑J=1Nαiαjyiyj∂kxi,xj∂vm.Finally, the derivatives of I(vm,C) with respect to the parameters vm of the gating model ηm(x) can be formulated as(20)∂Ivm,C∂vm=J2vm,C∂J1vm,C∂vm+J1vm,C∂J2vm,C∂vm.


## 3.4. Optimizing the Regularization Parameter

In our method, the regularization parameterC is treated as a variable that can be learned when learning the gating model. Similar to the process of optimizing the parameter of the gating model, we employ a gradient-descent method to optimize the regularization parameter C and so the derivative of (14) with respect to C is needed. In the following, we will discuss how to compute the derivative.Actually, the derivative of (14) I(vm,C) with respect to C can be expressed as(21)∂Ivm,C∂C=J2vm,C∂J1vm,C∂C+J1vm,C∂J2vm,C∂C.Obviously, one obtains in advance ∂(J1(vm,C))/∂C and ∂(J2(vm,C))/∂C, which are, respectively, computed as(22)∂J1vm,C∂C=∂R2+Δ/C∂C=∂Δ/C∂C=-ΔC2,∂J2vm,C∂C=1C∑i=1Nαi-∑i=1N∑J=1Nαiαjyiyjkxi,xj=1CJ2vm,C.


## 3.5. Discussion

It should be noted thatC>0 must hold in the whole procedure. However, actually in the iterations, this condition may be broken. In order to deal with this problem, following [27], we can use ln⁡(C) instead of C in the solving procedure. The reason is that ln⁡(C) can be any real number when C>0. Thus, the positivity of C is dodged. Here, we need to rewrite the above partial derivatives. According to the chain rules, we can modify (21) as the following:(23)∂Ivm,C∂ln⁡C=J2vm,C∂J1vm,C∂ln⁡C+J1∂J2vm,C∂ln⁡C,where(24)∂J1vm,C∂lnC=C∂J1vm,C∂C,∂J2vm,C∂ln⁡C=C∂J2vm,C∂C.Finally, according to the above discussion, we outline the complete algorithm of ILMKL in Algorithm1.Algorithm 1:ILMKL.
(1) Initialize lnCinit and initialize vm to small random numbers for m=1,…,M; (2) whilestopping criterion not met do (3)    Compute C with C=exp⁡(ln⁡(C)); (4)    Calculate k⌢(xi,xj) with the gating model according to (7) when fixing vm; (5)    Compute I(vm,C) by using an canonical SVM solver and an canonical SVDD solver with k⌢(xi,xj) according to (14); (6)    Compute ∂(I(vm,C))/∂vm for m=1,…,M with (20); (7)    Compute ∂I(vm,C)/∂ln⁡(C) with (23); (8)    Update vm and ln⁡(C) by the gradient-descent method; (9) end while


## 4. Experiments

In this section, the experimental results will be reported. In the first experiment, we investigate the influence of parameters on ILMKL performance. In the second experiment, we further explore the possibility of learning the regularization parameterC under different initial value on a synthetic dataset. In the third experiment, we conduct the experiments on several UCI datasets and compare the proposed method with traditional MKL methods.


### 4.1. Parameter Influence on Performance of the Proposed Method

In the training procedure of LMKL, the regularization parameterC must be predefined. In our method ILMKL, the parameter C can be automatically tuned during the learning process of the parameters of the gating model. However, we need to first set the parameter Δ and the initial value (denoted by Cinit) of C. In [27], the authors suggested Δ=1 and ln⁡(Cinit)=0. In this subsection, we will investigate the influence of these parameters on the final learned regularization parameter C and the classification performance of the proposed method.We used the Sonar dataset, which was selected from the UCI repository [32]. In the experiment, 50% of the datasets were randomly selected for training and the rest for testing. The data were preprocessed in the following way: first, the mean and the standard deviation of each feature were computed according to training data; then, training examples were normalized to have mean 0 by subtracting the mean and unit variance; finally, testing examples were correspondingly preprocessed using the mean and the standard deviation. The base kernels include one linear kernel and one polynomial kernel with degree of two. All kernel matrices are calculated and normalized to unit trace before training.Figure1 shows the experimental results under different Δ. From Figure 1(a), we can find that the value of Δ actually influences the final value of learned C. The reason, in our opinion, is that the algorithm may fall into a local minimum since we adopt the gradient-descent method which cannot guarantee finding the global minimum. Actually, the final values of learned C under different Δ are close to each other on the whole. Moreover, the classification accuracies under different Δ have almost no difference according to Figure 1(b). Figure 2 shows the experimental results under different initial value of C. From Figure 2(a), similar to the case which is shown in Figure 1(a), we can find that the initial value Cinit of C also impacts the final learned C. However, according to Figure 2(b), the initial value scarcely influences the classification accuracy.Figure 1
The experimental results of ILMKL under differentΔ on the Sonar dataset.
(a)
The final learnedC of ILMKL under different Δ (b)
The classification accuracy of ILMKL under differentΔFigure 2
The experimental results of ILMKL under the initial valueCinit on the Sonar dataset.
(a)
The final learnedC of ILMKL under different initial value Cinit (b)
The classification accuracy of ILMKL under different initial valueCinitTherefore, it can be concluded that the proposed method is effective to learn a suitableC in the SVM-based LMKL scenario under different Δ and initial value Cinit of C.


### 4.2. Experimental Results on Synthetic Dataset

In the above experiments, the final regularization parameter learnedC is always much larger than the initial value Cinit of C and so C almost always increases in the learning progress. In these experiments, we will show that the final learned C actually can be larger or smaller than the initial value Cinit of C.Following [15], we create a synthetic dataset, which consists of two classes, and each class contains 200 samples. The samples come from four Gaussian components (two for each class), and each component, respectively, has the following mean vector and covariance matrix:(25)μ1=-3.0,+1.0,μ2=+1.0,+1.0,μ3=-1.0,-2.2,μ4=+3.0,-2.2Σ1=0.80.00.02.0,Σ2=0.80.00.02.0,Σ3=0.80.00.04.0,Σ4=0.80.00.04.0,where the samples in class 1 are from the first two components (denoted by red ×) and others belong to class 2 (denoted by blue +). Here, we adopt the same base kernels as in Section 4.1, that is, linear kernels and one polynomial kernel with degree of two. Before training, all kernel matrices are computed and preprocessed to unit trace in advance.Figure3 illustrates the experimental results of LMKL under different regularization parameter C. It can be easily found that the values of the regularization parameter C impact severely the result of LMKL.Obviously, the experimental result illustrated in Figure 3(c) is better. Here, the regularization parameter C is set as C=10. The experimental results of the proposed method are illustrated in Figure 4. It can be found that we almost obtain the same result under different initial value Cinit of C. Moreover, the final learned C is sometimes smaller and sometimes larger than the initial value. Note that the final values of learned C are different under the different initial value Cinit of C. The reason, as pointed out in Section 4.1, is that the gradient-descent method is employed to optimize the regularization parameter C. However, the final learned regularization parameters are close to each other. The experiments further validate the fact that our method can effectively automatically learn the regularization parameter C. This is an advantage over traditional LMKL.Figure 3
The experimental results of LMKL on the synthetic dataset.
(a)
The obtained separate hyperplane underCsvm = 10−1 (b)
The obtained separate hyperplane underCsvm = 100 (c)
The obtained separate hyperplane underCsvm = 101 (d)
The obtained separate hyperplane underCsvm = 102 (e)
The obtained separate hyperplane underCsvm = 103Figure 4
The experimental results of ILMKL on the synthetic dataset.
(a)
The obtained separate hyperplane underCinit = 10−1 (b)
The obtained separate hyperplane underCinit = 100 (c)
The obtained separate hyperplane underCinit = 101 (d)
The obtained separate hyperplane underCinit = 102 (e)
The obtained separate hyperplane underCinit = 103


### 4.3. Experimental Results on UCI Datasets

In this subsection, we report the performance comparison about SimpleMKL [2], LMKL [15], and the proposed method on several UCI datasets [32].In the experiment, we use 50% of each dataset as a training set and the rest as the test set. As in Section4.1, the data were normalized (i.e., 0 mean and 1 standard deviation). The base kernels include seven Gaussian kernels with the widths of [3-3,3-2,3-1,30,31,32,33] and four polynomial kernels with degrees of one to four. Before training, all kernel matrices are calculated in advance and preprocessed to unit trace. Each experiment is repeated 50 times, and the mean accuracy and standard deviation were computed. In the experiments, SimpleMKL and LMKL employ the cross-validation technique to choose the regularization parameter C from the set {10-4, 10-3, 10-2, 10-1, 100, 101, 102, 103, 104}. For our method ILMKL, it is not necessary to use cross-validation to select the parameter C because it can learn an appropriate value.Table1 reports the classification accuracies of several SVM-based MKL methods on the selected datasets. From Table 1, it can be found that LMKL has comparable performance to SimpleMKL. However, on the whole, it can be found that ILMKL has a clear improvement in the classification performance in contrast with SimpleMKL and LMKL. These experimental results indicate that the generalized performance in SVM-based MKL can be improved when the information of radius of MEB is considered. The proposed method ILMKL embodies the idea.Table 1
Classification accuracy (mean ± standard derivation) on the selected datasets.
Dataset SimpleMKL LMKL ILMKL Ionosphere 92.07 ± 2.02 91.64 ± 2.77 93.89 ± 1.21 Heart 84.01 ± 1.38 83.92 ± 1.65 85.91 ± 2.84 Wdbc 96.19 ± 0.44 95.54 ± 2.35 96.97 ± 1.96 Wpbc 76.59 ± 1.02 77.40 ± 2.21 78.42 ± 1.14 Liver 64.45 ± 3.64 62.13 ± 2.93 64.87 ± 3.46 Musk1 89.79 ± 2.02 86.44 ± 3.37 90.27 ± 4.68 Sonar 83.42 ± 3.30 79.95 ± 3.15 82.98 ± 5.02 Fourclass 99.95 ± 0.06 99.93 ± 0.14 99.97 ± 0.09 Coloncancer 74.98 ± 8.18 74.83 ± 3.93 76.11 ± 1.16 Germannumber 72.74 ± 1.55 73.32 ± 1.78 75.24 ± 1.15 Pima 74.37 ± 2.05 75.23 ± 3.12 77.01 ± 1.52For a rigorous comparison, simultaneously, we further conducted the paired two-tailedt-tests [33] on these methods. In t-test, the p value depicts the probability that two sets generate from distributions with equal means. If the p value is smaller, then the difference of the two mean values is more significant. Generally, 0.05 is viewed as a typical threshold of the p value; that is, it is considered statistically significant when the p value is smaller than 0.05. Table 2 reports the experimental results of the t-tests. For example, the p value of the t-test when comparing LMKL and SimpleMKL on the Ionosphere dataset is 0.0862 (>0.05), meaning that SimpleMKL does not perform significantly better than LMKL on this dataset at the 0.05 significant level though SimpleMKL has better classification accuracy according to Table 1. However, ILMKL performs significantly better than SimpleMKL since the p value of the t-test is 0.0296 (<0.05) on this dataset at the 0.05 significant level. From Table 2, ILMKL has on the whole significant improvement in the generalized performance in contrast with SimpleMKL and LMKL.Table 2
p value of t-test on the selected datasets.
Dataset LMKL/SimpleMKL ILMKL/SimpleMKL ILMKL/LMKL Ionosphere 0.0862 0.0296 0.0451 Heart 0.0639 0.0371 0.0339 Wdbc 0.0418 0.0641 0.0207 Wpbc 0.0325 0.0153 0.4759 Liver 0.0393 0.0426 0.0931 Musk1 0.0244 0.0915 0.0185 Sonar 0.0167 0.0733 0.0239 Fourclass 0.8514 0.0435 0.1384 Coloncancer 0.7426 0.0158 0.0256 Germannumber 0.0462 0.0192 0.0374 Pima 0.0437 0.0382 0.0427Finally, we investigated the support vector percentages of several methods on the selected datasets, which are reported in Table3. Generally, fewer support vectors mean less test time. From Table 3, LMKL tends to have more support vectors in contrast with SimpleMKL. The proposed method ILMKL has on the whole similar support vector percentages to LMKL. So, our method inherits the advantage of LMKL that it stores fewer support vectors but can achieve statistically similar accuracy results compared with other MKL methods.Table 3
Rate of support vectors comparison on the selected datasets.
Dataset SimpleMKL LMKL ILMKL Ionosphere 0.6982 ± 0.0926 0.4862 ± 0.0504 0.4565 ± 0.0674 Heart 0.7681 ± 0.0671 0.5948 ± 0.1486 0.4333 ± 0.1102 Wdbc 0.3640 ± 0.0213 0.2191 ± 0.0182 0.2010 ± 0.0218 Wpbc 0.9948 ± 0.0153 0.7928 ± 0.0952 0.7857 ± 0.0579 Liver 0.9982 ± 0.0026 0.9203 ± 0.0550 0.9302 ± 0.0111 Musk1 0.9316 ± 0.0309 0.6793 ± 0.0544 0.6185 ± 0.1155 Sonar 0.9126 ± 0.0330 0.6456 ± 0.0367 0.5398 ± 0.1079 Fourclass 0.3790 ± 0.0177 0.1804 ± 0.0514 0.1388 ± 0.0840 Coloncancer 1.0000 ± 0.0000 0.9274 ± 0.0265 0.9258 ± 0.0816 Germannumber 0.8972 ± 0.0373 0.7006 ± 0.0356 0.7378 ± 0.0407 Pima 0.7937 ± 0.0872 0.5893 ± 0.0876 0.5759 ± 0.0209


## 4.1. Parameter Influence on Performance of the Proposed Method

In the training procedure of LMKL, the regularization parameterC must be predefined. In our method ILMKL, the parameter C can be automatically tuned during the learning process of the parameters of the gating model. However, we need to first set the parameter Δ and the initial value (denoted by Cinit) of C. In [27], the authors suggested Δ=1 and ln⁡(Cinit)=0. In this subsection, we will investigate the influence of these parameters on the final learned regularization parameter C and the classification performance of the proposed method.We used the Sonar dataset, which was selected from the UCI repository [32]. In the experiment, 50% of the datasets were randomly selected for training and the rest for testing. The data were preprocessed in the following way: first, the mean and the standard deviation of each feature were computed according to training data; then, training examples were normalized to have mean 0 by subtracting the mean and unit variance; finally, testing examples were correspondingly preprocessed using the mean and the standard deviation. The base kernels include one linear kernel and one polynomial kernel with degree of two. All kernel matrices are calculated and normalized to unit trace before training.Figure1 shows the experimental results under different Δ. From Figure 1(a), we can find that the value of Δ actually influences the final value of learned C. The reason, in our opinion, is that the algorithm may fall into a local minimum since we adopt the gradient-descent method which cannot guarantee finding the global minimum. Actually, the final values of learned C under different Δ are close to each other on the whole. Moreover, the classification accuracies under different Δ have almost no difference according to Figure 1(b). Figure 2 shows the experimental results under different initial value of C. From Figure 2(a), similar to the case which is shown in Figure 1(a), we can find that the initial value Cinit of C also impacts the final learned C. However, according to Figure 2(b), the initial value scarcely influences the classification accuracy.Figure 1
The experimental results of ILMKL under differentΔ on the Sonar dataset.
(a)
The final learnedC of ILMKL under different Δ (b)
The classification accuracy of ILMKL under differentΔFigure 2
The experimental results of ILMKL under the initial valueCinit on the Sonar dataset.
(a)
The final learnedC of ILMKL under different initial value Cinit (b)
The classification accuracy of ILMKL under different initial valueCinitTherefore, it can be concluded that the proposed method is effective to learn a suitableC in the SVM-based LMKL scenario under different Δ and initial value Cinit of C.


## 4.2. Experimental Results on Synthetic Dataset

In the above experiments, the final regularization parameter learnedC is always much larger than the initial value Cinit of C and so C almost always increases in the learning progress. In these experiments, we will show that the final learned C actually can be larger or smaller than the initial value Cinit of C.Following [15], we create a synthetic dataset, which consists of two classes, and each class contains 200 samples. The samples come from four Gaussian components (two for each class), and each component, respectively, has the following mean vector and covariance matrix:(25)μ1=-3.0,+1.0,μ2=+1.0,+1.0,μ3=-1.0,-2.2,μ4=+3.0,-2.2Σ1=0.80.00.02.0,Σ2=0.80.00.02.0,Σ3=0.80.00.04.0,Σ4=0.80.00.04.0,where the samples in class 1 are from the first two components (denoted by red ×) and others belong to class 2 (denoted by blue +). Here, we adopt the same base kernels as in Section 4.1, that is, linear kernels and one polynomial kernel with degree of two. Before training, all kernel matrices are computed and preprocessed to unit trace in advance.Figure3 illustrates the experimental results of LMKL under different regularization parameter C. It can be easily found that the values of the regularization parameter C impact severely the result of LMKL.Obviously, the experimental result illustrated in Figure 3(c) is better. Here, the regularization parameter C is set as C=10. The experimental results of the proposed method are illustrated in Figure 4. It can be found that we almost obtain the same result under different initial value Cinit of C. Moreover, the final learned C is sometimes smaller and sometimes larger than the initial value. Note that the final values of learned C are different under the different initial value Cinit of C. The reason, as pointed out in Section 4.1, is that the gradient-descent method is employed to optimize the regularization parameter C. However, the final learned regularization parameters are close to each other. The experiments further validate the fact that our method can effectively automatically learn the regularization parameter C. This is an advantage over traditional LMKL.Figure 3
The experimental results of LMKL on the synthetic dataset.
(a)
The obtained separate hyperplane underCsvm = 10−1 (b)
The obtained separate hyperplane underCsvm = 100 (c)
The obtained separate hyperplane underCsvm = 101 (d)
The obtained separate hyperplane underCsvm = 102 (e)
The obtained separate hyperplane underCsvm = 103Figure 4
The experimental results of ILMKL on the synthetic dataset.
(a)
The obtained separate hyperplane underCinit = 10−1 (b)
The obtained separate hyperplane underCinit = 100 (c)
The obtained separate hyperplane underCinit = 101 (d)
The obtained separate hyperplane underCinit = 102 (e)
The obtained separate hyperplane underCinit = 103


## 4.3. Experimental Results on UCI Datasets

In this subsection, we report the performance comparison about SimpleMKL [2], LMKL [15], and the proposed method on several UCI datasets [32].In the experiment, we use 50% of each dataset as a training set and the rest as the test set. As in Section4.1, the data were normalized (i.e., 0 mean and 1 standard deviation). The base kernels include seven Gaussian kernels with the widths of [3-3,3-2,3-1,30,31,32,33] and four polynomial kernels with degrees of one to four. Before training, all kernel matrices are calculated in advance and preprocessed to unit trace. Each experiment is repeated 50 times, and the mean accuracy and standard deviation were computed. In the experiments, SimpleMKL and LMKL employ the cross-validation technique to choose the regularization parameter C from the set {10-4, 10-3, 10-2, 10-1, 100, 101, 102, 103, 104}. For our method ILMKL, it is not necessary to use cross-validation to select the parameter C because it can learn an appropriate value.Table1 reports the classification accuracies of several SVM-based MKL methods on the selected datasets. From Table 1, it can be found that LMKL has comparable performance to SimpleMKL. However, on the whole, it can be found that ILMKL has a clear improvement in the classification performance in contrast with SimpleMKL and LMKL. These experimental results indicate that the generalized performance in SVM-based MKL can be improved when the information of radius of MEB is considered. The proposed method ILMKL embodies the idea.Table 1
Classification accuracy (mean ± standard derivation) on the selected datasets.
Dataset SimpleMKL LMKL ILMKL Ionosphere 92.07 ± 2.02 91.64 ± 2.77 93.89 ± 1.21 Heart 84.01 ± 1.38 83.92 ± 1.65 85.91 ± 2.84 Wdbc 96.19 ± 0.44 95.54 ± 2.35 96.97 ± 1.96 Wpbc 76.59 ± 1.02 77.40 ± 2.21 78.42 ± 1.14 Liver 64.45 ± 3.64 62.13 ± 2.93 64.87 ± 3.46 Musk1 89.79 ± 2.02 86.44 ± 3.37 90.27 ± 4.68 Sonar 83.42 ± 3.30 79.95 ± 3.15 82.98 ± 5.02 Fourclass 99.95 ± 0.06 99.93 ± 0.14 99.97 ± 0.09 Coloncancer 74.98 ± 8.18 74.83 ± 3.93 76.11 ± 1.16 Germannumber 72.74 ± 1.55 73.32 ± 1.78 75.24 ± 1.15 Pima 74.37 ± 2.05 75.23 ± 3.12 77.01 ± 1.52For a rigorous comparison, simultaneously, we further conducted the paired two-tailedt-tests [33] on these methods. In t-test, the p value depicts the probability that two sets generate from distributions with equal means. If the p value is smaller, then the difference of the two mean values is more significant. Generally, 0.05 is viewed as a typical threshold of the p value; that is, it is considered statistically significant when the p value is smaller than 0.05. Table 2 reports the experimental results of the t-tests. For example, the p value of the t-test when comparing LMKL and SimpleMKL on the Ionosphere dataset is 0.0862 (>0.05), meaning that SimpleMKL does not perform significantly better than LMKL on this dataset at the 0.05 significant level though SimpleMKL has better classification accuracy according to Table 1. However, ILMKL performs significantly better than SimpleMKL since the p value of the t-test is 0.0296 (<0.05) on this dataset at the 0.05 significant level. From Table 2, ILMKL has on the whole significant improvement in the generalized performance in contrast with SimpleMKL and LMKL.Table 2
p value of t-test on the selected datasets.
Dataset LMKL/SimpleMKL ILMKL/SimpleMKL ILMKL/LMKL Ionosphere 0.0862 0.0296 0.0451 Heart 0.0639 0.0371 0.0339 Wdbc 0.0418 0.0641 0.0207 Wpbc 0.0325 0.0153 0.4759 Liver 0.0393 0.0426 0.0931 Musk1 0.0244 0.0915 0.0185 Sonar 0.0167 0.0733 0.0239 Fourclass 0.8514 0.0435 0.1384 Coloncancer 0.7426 0.0158 0.0256 Germannumber 0.0462 0.0192 0.0374 Pima 0.0437 0.0382 0.0427Finally, we investigated the support vector percentages of several methods on the selected datasets, which are reported in Table3. Generally, fewer support vectors mean less test time. From Table 3, LMKL tends to have more support vectors in contrast with SimpleMKL. The proposed method ILMKL has on the whole similar support vector percentages to LMKL. So, our method inherits the advantage of LMKL that it stores fewer support vectors but can achieve statistically similar accuracy results compared with other MKL methods.Table 3
Rate of support vectors comparison on the selected datasets.
Dataset SimpleMKL LMKL ILMKL Ionosphere 0.6982 ± 0.0926 0.4862 ± 0.0504 0.4565 ± 0.0674 Heart 0.7681 ± 0.0671 0.5948 ± 0.1486 0.4333 ± 0.1102 Wdbc 0.3640 ± 0.0213 0.2191 ± 0.0182 0.2010 ± 0.0218 Wpbc 0.9948 ± 0.0153 0.7928 ± 0.0952 0.7857 ± 0.0579 Liver 0.9982 ± 0.0026 0.9203 ± 0.0550 0.9302 ± 0.0111 Musk1 0.9316 ± 0.0309 0.6793 ± 0.0544 0.6185 ± 0.1155 Sonar 0.9126 ± 0.0330 0.6456 ± 0.0367 0.5398 ± 0.1079 Fourclass 0.3790 ± 0.0177 0.1804 ± 0.0514 0.1388 ± 0.0840 Coloncancer 1.0000 ± 0.0000 0.9274 ± 0.0265 0.9258 ± 0.0816 Germannumber 0.8972 ± 0.0373 0.7006 ± 0.0356 0.7378 ± 0.0407 Pima 0.7937 ± 0.0872 0.5893 ± 0.0876 0.5759 ± 0.0209


## 5. Conclusions

In this paper, by following the work in [27], we presented a novel LMKL method. Different from traditional LMKL, our method takes into consideration the information of both the radius and the margin when learning the parameters of the gating model. As a result, our method can achieve better accuracy. Simultaneously, our method can automatically tune the regularization parameter C during the process of learning the parameters of the gating model. Therefore, this can improve the computational efficiency of our method by avoiding using the time-consuming cross-validation technique to find a suitable regularization parameter. Comprehensive experiments are conducted on several toy and benchmark datasets and the results well demonstrate the efficiency and effectiveness of the proposed method.



---

*Source: 4579214-2017-01-09.xml*