Improved Learning Rates of a Functional Lasso-type SVM with Sparse Multi-Kernel Representation

1 Introduction . SVM was first introduced in [ 45 ] and has became one of the most popular machine learning algorithms in the past two decades . The standard SVM classification consists of two main ingredients , namely the hinge loss and the kernel embedding . The hinge loss is used to model the learning target and can often generate sparse solutions [ 38 ] , while the kernel embedding is used to model nonlinear relationship between input features and response [ 39 ] . In [ 38 ] , they provide a detailed overview of SVMs and related learning theory . More work on theoretical perspective of SVM have also been developed in recent years , such as [ 12 , 53 , 36 , 33 , 30 , 24 ] , among many others . It is known that the performance of kernel machines largely depends on the data representation via the choice of kernel function [ 34 , 16 , 19 , 28 , 26 , 48 , 49 ] . Towards this direction , many approaches have been proposed for kernel selection under different frameworks . For example , Micchelli et al . [ 34 ] attempt to find an optimal kernel from a prescribed convex set of basis kernels ; Wu et al . [ 46 ] optimize the scale parameter among various Gaussian kernels ; and Ong et al . [ 35 ] study hyperkernels on the space of kernels and alternative approaches , including kernels selection by DC programming and semi-infinite programming . In particular , the seminal work of [ 22 ] proposes the so-called multiple kernel learning ( MKL ) method , learning SVM and a linear combination of kernels simultaneously , which has received a lot of attention in recent years [ 3 , 50 , 20 , 19 , 16 , 23 , 17 , 27 , 25 , 29 , 51 ] . To be precise , given a finite ( possibly large ) dictionary of symmetric positive semidefinite kernels { Km : m = 1 , 2 , ... , M } , one can try to find an ‘ ideal ’ kernel K as a convex combination of the kernels from a predefined family of potential candidates : K ∈ K : = { ∑M m=1 θmKm : ∑M m=1 θm = 1 , θm ≥ 0 } . This combination of multiple kernels corresponds to a ∗Corresponding Author 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . multi-kernel hypothesis space for learning : HM : = { M∑ m=1 fm ( x ) : fm ∈ HKm , x ∈ X } , whereHKm is a reproducing kernel Hilbert space ( RKHS ) induced by the kernel Km , as defined in Section 2 . Given the learning rule , θm ’ s also need to be estimated automatically from the training data . Besides flexibility enhancement , other justifications of MKL have also been proposed ; such as each kernel function corresponds to different information source ( e.g . text , image , gene ) , or classifiers from different spaces ( corresponding to different kernels ) are averaged to achieve better classification accuracy [ 8 ] . Under many of these situations , the number of candidate kernels M is often very large , even larger than the sample size n in some extreme cases . Interestingly , when one-dimensional linear ( or additive ) kernels are used and M is the dimensionality of covariates with M ≥ n , MKL reduces to high-dimensional linear ( or additive ) models , which have been widely studied in the statistics and machine learning literature [ 15 , 6 , 32 ] . Given the large number of kernels , it is common that not all the kernels are significantly relevant to the response . To avoid impairing generalization performance [ 54 ] and enhance interpretability , many redundant kernels should be removed accordingly . In view of generalization ability , interpretability and computational feasibility , a more parsimonious and flexible algorithm for MKL is to generate a sparse linear combination of kernels based on the training data , namely sparse MKL . From a statistical point of view , sparse MKL can be interpreted as a model selection task , and can also be viewed as an extension of sparse linear models or sparse additive models . In other words , sparse MKL provides an appropriate route to tackle the kernel learning issues in machine learning and the high-dimensional issues in statistics . In literature , many regularized learning algorithms based on different regularization terms have been investigated in multi-kernel regression [ 3 , 20 , 18 , 37 , 41 ] . Particularly , in [ 37 , 41 ] , they established the optimal rates of convergence for the least square approaches in a related setting . In multi-kernel classification , Christmann and Hable [ 12 ] gave consistency and robustness property of non-sparse additive SVM when M is fixed ; In [ 36 , 42 ] , they provided upper bounds on the estimation error and the excess risk in high-dimensional linear case or with finite number of basis functions . In summary , all the aforementioned work either focus on the regression problem with the quadratic losses , or only consider the parametric case for high-dimensional SVM . The main focus of this paper is on a more challenging case with sparse multi-kernel approximation for SVM . That is , while the total number of kernels may be larger than the sample size , only a small number of kernels are needed to represent/approximate the target function , so that such a learning problem in the sense of approximation is sparse . Let X be a random variable of the input space X , and Y ∈ Y = { −1 , +1 } be the response . Define the theoretical oracle predictor of the classical SVM withinHM as f∗ = arg min f∈HM E ( f ) , with E ( f ) : = E [ φ ( Y f ( X ) ) ] . ( 1 ) Here φ ( t ) : = ( 1−t ) + is the hinge loss and the expectation is taken with respect to the joint distribution ρ defined on X × Y . In the sparse setting , one often assumes that f∗ has an additive and sparse representation withinHM , namely , f∗ = ∑ m∈S f ∗ m with an unknown subset S ⊂ { 1 , 2 , ... , M } and f∗m ∈ Hm with ‖f∗m‖Km ≤ 1 for all m ∈ S. Of primary interest is the case with s : = |S| M . Note that f∗ here is an optimal estimator withinHM , as opposed to the classical Bayes decision rule defined over all the measurable functions . Actually , the minimizer of the expected risk corresponding to the hinge loss is not continuous against the conditional probability [ 5 ] . The main contribution of this paper is to develop a set of theoretical results on multi-kernel SVM and the rates of convergence for nonparametric classification within the multi-kernel setting , including high-dimensional linear or additive classification as special cases . This paper provides refined error bounds for the excess risk , defined as E ( f̂ ) − E ( f∗ ) , of the proposed estimator f̂ , as well as its estimation error ‖f̂ − f∗‖2 . The proposed estimator in ( 5 ) is based on a regularization of the hinge loss with two fold ` 1 penalty terms , where one is used to control nonparametric sparsity and the other is to control functional smoothness . Particularly , Corollary 1 shows that with high probability , the excess risk of the proposed estimator is upper bounded by s Γ2 ( S , ρX ) O ( n− 1 1+τ + logM n ) κ 2κ−1 under some regularity conditions , where τ corresponds to the spectral decay of each RKHS and is used to characterize functional complexity of kernels , κ is called the Bernstein parameter that reflects the low-density level of decision boundary , and Γ ( S , ρX ) is used to characterize correlation structure among multiple RKHS ’ s . The same rate also holds true for ‖f̂ − f∗‖2κ2 following the Bernstein condition and the established results on the excess risk . Furthermore , we establish the oracle rate of the relative misclassification error of f̂ in Theorem 2 , which consists of the same rate as the excess risk of the estimator plus some additional sparse approximation error . By using this specific form of RKHS ’ s , we can incorporate many existing results as special cases of MKL for SVMs , and more importantly , several existing results can be improved by our derived ones . The detailed comparison between our general results with existing results is conducted in Section 3.3 . Notations . Define the L2 norm of a function f by ‖f‖22 = ∫ X |f ( x ) |2dρX ( x ) with marginal distribution ρX . Given sample points { xi } ni=1 , ‖fm‖2n : = 1n ∑n i=1 fm ( xi ) 2 and ‖fm‖n is viewed as the empirical L2-norm . a ' bmeans that there are two positive constants c , C such that ca ≤ b ≤ Ca . Also , a = O ( b ) means that there is some positive constant C such that a ≤ Cb in probability , and a = o ( b ) means that a/b→ 0 in probability . To ease the notations , we write [ M ] : = { 1 , ... , M } . 2 Preliminaries and Algorithms . Given a compact set X , we denote a positive semidefinite kernel on X ×X , i.e. , a symmetric function K : X × X → R satisfying ∑n i , j=1 cicjK ( xi , xj ) ≥ 0 for any x1 , ... , xn in X and c1 , ... , cn ∈ R. As showed in [ 2 ] , a positive semidefinite kernel on X is associated with a unique Hilbert space HK consisting of functions on X , andHK is called a RKHS associated with the kernel K. RKHS is known for its reproducing property , i.e. , for any f ∈ HK and x ∈ X , f ( x ) = 〈f , Kx〉K with Kx = K ( x , · ) . Another key property of RKHS is the spectral theorem , which assures that K admits the following eigen-decomposition : K ( x , x′ ) = ∑ ` ≥1 µ ` ψ ` ( x ) ψ ` ( x ′ ) , for any x , x′ ∈ X , ( 2 ) where µ1 ≥ µ2 ≥ ... ≥ 0 are its eigenvalues and { ψ ` : ` ≥ 1 } are the corresponding eigenfunctions , leading to an orthogonal basis in L2 ( ρX ) . These two fundamental properties of RKHS will be the foundation of our theoretical analysis . To characterize functional complexities , we first introduce several basic facts of the empirical processes defined in RKHS . On basis of ( 2 ) , for any δ ∈ ( 0 , 1 ] , we define ωn ( δ ) : = ( 1 n n∑ ` =1 ( µ ` ∧ δ2 ) ) 1/2 , where a ∧ b means min { a , b } . For some constant A > 0 , we define a quantity associated with ωn ( δ ) as ( K ) : = inf { ≥ √ A logM n : ωn ( δ ) ≤ δ + 2 , ∀ δ ∈ ( 0 , 1 ] } . The quantity ( K ) plays a key role in bounding the excess risk in RKHS [ 40 , 20 ] . Given appropriate decay of µ ` , the upper bound of ( K ) can be derived explicitly in Section 3.1 . Given the training sample { Xi , Yi } ni=1 i.i.d . from X ×Y , the standard SVM with a single kernel [ 45 ] solves the following optimization task min f∈HK { 1 n n∑ i=1 φ ( Yif ( Xi ) ) + λ‖f‖2K } , ( 3 ) where λ is a regularization parameter that balances the tradeoff between the empirical risk and the function complexity . The theoretical properties of the standard SVM has been extensively studied in literature ; see the earlier work of [ 11 , 39 , 7 ] . In the context of MKL , the sparse multi-kernel SVM [ 34 ] equips the empirical hinge loss with the ` 1 penalty , min f= ∑M m=1 fm , fm∈HKm { 1 n n∑ i=1 φ ( Yif ( Xi ) ) + λ M∑ m=1 ‖fm‖Km } . ( 4 ) This type of sparse MKL method has been studied extensively in [ 34 , 3 , 20 , 16 ] , and among others . Essentially , it can be regarded as an infinite-dimensional version of the Lasso-type penalization [ 43 ] . In this paper , we present a different approach from ( 4 ) to formulate the sparse multi-kernel SVM . For simplicity , we assume that supx∈X |Km ( x , x ) | ≤ 1 for all m ∈ [ M ] . We use a new L1-type regularization term partially inspired by the additive mean models [ 32 ] . Denote the bounded ball ofHM by BM : = { f = ∑M j=1 fm : fm ∈ HKm , ‖fm‖Km ≤ 1 } , the regularization term we adopt for the multiple-kernel SVM combines the empirical L2-norms and RKHS-norms . Specifically , the proposed sparse multi-kernel SVM is formulated as f̂ = M∑ m=1 f̂m : = arg min f∈BM { 1 n n∑ i=1 φ ( Yif ( Xi ) ) + M∑ m=1 λm √ ‖fm‖2n + γm‖fm‖2Km } , ( 5 ) where ‖fm‖n is used to enforce the sparsity in f̂ , whereas ‖fm‖Km is used to enforce the smoothness of each f̂m . Here ( λm , γm ) Mm=1 are the regularization parameters , which will be specified in our theoretical results . By finite representation of reproducing kernel , each additive estimator fm ( · ) = ∑n i=1 α m i Km ( xi , · ) for all m = 1 , ... , M . A direct computation leads to√ ‖fm‖2n + γn‖fm‖2K = √ ( αm ) T K̃mαm where K̃m = K2m n + γnKm . Here Km is the kernel matrix induced by Km at points { xi } n i=1 . Note that K̃m is a semi-definite matrix , which can be written as K̃m = A2 with some matrix A . Hence , our original learning scheme in Eq . ( 5 ) can be transformed into a group Lasso optimization [ 32 ] , and there exists several efferent numerical algorithms for solving it , such as proximal methods and coordinate descent ones . This penalty term significantly differs from other sparsity penalties for nonparametric models in literature , such as ∑M m=1 ‖fm‖Km [ 22 ] and ∑M m=1 ( λ1‖fm‖n +λ2‖fm‖Km ) [ 20 , 37 ] . Although the former penalty often generates sparse solutions , it is difficult to establish its theoretical results , due to the fact that the ‖ · ‖K-norm can not fully reflect the marginal distribution information . The latter penalty has been proved to enjoy some theoretical properties in [ 37 ] , which is equivalent to the mixed L1-norm ( ∑M m=1 λm √ ‖fm‖2n + γm‖fm‖2Km ) in ( 5 ) . Theoretically , based on empirical processes theory , our proposed approach can also achieve improved learning rates , like the penalized method with the penalty ∑M m=1 ( λ1‖fm‖n + λ2‖fm‖Km ) , which has been considered in [ 20 , 31 , 37 ] and others . Remark . In general , ‖f̂‖n = 0 does not imply f = 0 . However , in the case of any kernel-based minimization problem , ‖f̂‖n = 0 always implies f = 0 . Based on the reproducing property of Mercer kernel , f ( x ) = 〈f , Kx〉K for any f ∈ HK , where HK is a reproducing kernel Hilbert space . In fact , if ‖f̂‖n = 0 holds , that is , f̂ ( xi ) = 0 for all i = 1 , . . . , n , by the reproducing property we have 〈f̂ , Kxi〉K = 0 for all i . Hence , f̂ is orthogonal to the subspace Sn : = span { Kx1 , ... , Kxn } . On the other hand , using the reproducing property again , any solution f̂ of kernel-based minimization problems has a finite representation within Sn . So we conclude f̂ = 0 . Remark . The use of the mixed norm regularization is mainly motivated by the following fact : i ) the proposed estimation with the mixed norm can lead to very sharp learning rates ; ii ) the empirical norm ‖ · ‖n used for sparsity is much milder than ‖ · ‖K . There exists several related work on multi-kernel SVM . Under the setting of one-dimensional additive kernels , Christmann and Hable [ 12 ] constructed kernels for additive SVM and provided consistent and statistically robust estimators under the fixed dimensional and non-sparse setting . Zhao and Liu [ 53 ] proposed a group Lasso penalty by means of finite-bases approximation to a RKHS , and particularly they developed an efficient accelerated proximal gradient descent algorithm and established oracle properties of the SVM under sparse ultra-high dimensional setting ( e.g . M = o ( en ) ) . However , the additive SVM model may suffer from the lack of algorithmic flexibility and underfitting especially when the true model involves interaction effects . Similar concerns have been raised towards linear SVM as in [ 55 , 21 , 52 ] . Note that in the past decade , a lot of work on general MKL with logarithmic dependence on M have emerged [ 18 , 41 , 20 ] , yet their analysis requires the loss function to be strongly convex , which rules out the commonly-used hinge loss for SVM . We also note that [ 13 ] provided an upper bound Op ( √ logM/n ) of the Rademacher complexity with the L1-norm constraint , which may lead to the same decay rate of the excess risk of SVM . However , this rate is not tight in general , since it is known that the fast rate of order 1/n can be attained for the linear case [ 11 , 44 ] . This paper primarily focuses on the non-asymptotic analysis of the proposed sparse multi-kernel SVM method in ( 5 ) with an exponential number of kernels . Under the best ideal settings , the relative classification error of the proposed method is of near-oracle rate O ( s log ( M ) /n ) , as if we knew the true sparsity in advance . Moreover , the method is adaptive to the sparsity of the learning problem and the margin parameter . In our proof , we have to face some technical challenges , such as dealing with non-smoothness of the hinge loss , functional complexities and NP-dimensionality .

Summary: This paper derives generalization bounds for the sparse multi-kernel Support Vector Machine (SVM) problem. Recall that previous literature has focused on sparse multi-kernel problems, but only for the Ridge regression (square loss instead of hinge loss), or on multi-kernel SVM but without the sparsity inducing penalization. To do so, the authors introduce a new penalization, see Equation (5). Under some assumptions already used in previous works, excess risk bounds and estimation error bounds are derived (Theorem 1), further explicited in terms of the eigendecay of the kernel (Corollary 1). Bounds on the classification error are also provided (Theorem 2).