Improved Learning Rates of a Functional Lasso-type SVM with Sparse Multi-Kernel Representation

1 Introduction . SVM was first introduced in [ 45 ] and has became one of the most popular machine learning algorithms in the past two decades . The standard SVM classification consists of two main ingredients , namely the hinge loss and the kernel embedding . The hinge loss is used to model the learning target and can often generate sparse solutions [ 38 ] , while the kernel embedding is used to model nonlinear relationship between input features and response [ 39 ] . In [ 38 ] , they provide a detailed overview of SVMs and related learning theory . More work on theoretical perspective of SVM have also been developed in recent years , such as [ 12 , 53 , 36 , 33 , 30 , 24 ] , among many others . It is known that the performance of kernel machines largely depends on the data representation via the choice of kernel function [ 34 , 16 , 19 , 28 , 26 , 48 , 49 ] . Towards this direction , many approaches have been proposed for kernel selection under different frameworks . For example , Micchelli et al . [ 34 ] attempt to find an optimal kernel from a prescribed convex set of basis kernels ; Wu et al . [ 46 ] optimize the scale parameter among various Gaussian kernels ; and Ong et al . [ 35 ] study hyperkernels on the space of kernels and alternative approaches , including kernels selection by DC programming and semi-infinite programming . In particular , the seminal work of [ 22 ] proposes the so-called multiple kernel learning ( MKL ) method , learning SVM and a linear combination of kernels simultaneously , which has received a lot of attention in recent years [ 3 , 50 , 20 , 19 , 16 , 23 , 17 , 27 , 25 , 29 , 51 ] . To be precise , given a finite ( possibly large ) dictionary of symmetric positive semidefinite kernels { Km : m = 1 , 2 , ... , M } , one can try to find an ‘ ideal ’ kernel K as a convex combination of the kernels from a predefined family of potential candidates : K ∈ K : = { ∑M m=1 θmKm : ∑M m=1 θm = 1 , θm ≥ 0 } . This combination of multiple kernels corresponds to a ∗Corresponding Author 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . multi-kernel hypothesis space for learning : HM : = { M∑ m=1 fm ( x ) : fm ∈ HKm , x ∈ X } , whereHKm is a reproducing kernel Hilbert space ( RKHS ) induced by the kernel Km , as defined in Section 2 . Given the learning rule , θm ’ s also need to be estimated automatically from the training data . Besides flexibility enhancement , other justifications of MKL have also been proposed ; such as each kernel function corresponds to different information source ( e.g . text , image , gene ) , or classifiers from different spaces ( corresponding to different kernels ) are averaged to achieve better classification accuracy [ 8 ] . Under many of these situations , the number of candidate kernels M is often very large , even larger than the sample size n in some extreme cases . Interestingly , when one-dimensional linear ( or additive ) kernels are used and M is the dimensionality of covariates with M ≥ n , MKL reduces to high-dimensional linear ( or additive ) models , which have been widely studied in the statistics and machine learning literature [ 15 , 6 , 32 ] . Given the large number of kernels , it is common that not all the kernels are significantly relevant to the response . To avoid impairing generalization performance [ 54 ] and enhance interpretability , many redundant kernels should be removed accordingly . In view of generalization ability , interpretability and computational feasibility , a more parsimonious and flexible algorithm for MKL is to generate a sparse linear combination of kernels based on the training data , namely sparse MKL . From a statistical point of view , sparse MKL can be interpreted as a model selection task , and can also be viewed as an extension of sparse linear models or sparse additive models . In other words , sparse MKL provides an appropriate route to tackle the kernel learning issues in machine learning and the high-dimensional issues in statistics . In literature , many regularized learning algorithms based on different regularization terms have been investigated in multi-kernel regression [ 3 , 20 , 18 , 37 , 41 ] . Particularly , in [ 37 , 41 ] , they established the optimal rates of convergence for the least square approaches in a related setting . In multi-kernel classification , Christmann and Hable [ 12 ] gave consistency and robustness property of non-sparse additive SVM when M is fixed ; In [ 36 , 42 ] , they provided upper bounds on the estimation error and the excess risk in high-dimensional linear case or with finite number of basis functions . In summary , all the aforementioned work either focus on the regression problem with the quadratic losses , or only consider the parametric case for high-dimensional SVM . The main focus of this paper is on a more challenging case with sparse multi-kernel approximation for SVM . That is , while the total number of kernels may be larger than the sample size , only a small number of kernels are needed to represent/approximate the target function , so that such a learning problem in the sense of approximation is sparse . Let X be a random variable of the input space X , and Y ∈ Y = { −1 , +1 } be the response . Define the theoretical oracle predictor of the classical SVM withinHM as f∗ = arg min f∈HM E ( f ) , with E ( f ) : = E [ φ ( Y f ( X ) ) ] . ( 1 ) Here φ ( t ) : = ( 1−t ) + is the hinge loss and the expectation is taken with respect to the joint distribution ρ defined on X × Y . In the sparse setting , one often assumes that f∗ has an additive and sparse representation withinHM , namely , f∗ = ∑ m∈S f ∗ m with an unknown subset S ⊂ { 1 , 2 , ... , M } and f∗m ∈ Hm with ‖f∗m‖Km ≤ 1 for all m ∈ S. Of primary interest is the case with s : = |S| M . Note that f∗ here is an optimal estimator withinHM , as opposed to the classical Bayes decision rule defined over all the measurable functions . Actually , the minimizer of the expected risk corresponding to the hinge loss is not continuous against the conditional probability [ 5 ] . The main contribution of this paper is to develop a set of theoretical results on multi-kernel SVM and the rates of convergence for nonparametric classification within the multi-kernel setting , including high-dimensional linear or additive classification as special cases . This paper provides refined error bounds for the excess risk , defined as E ( f̂ ) − E ( f∗ ) , of the proposed estimator f̂ , as well as its estimation error ‖f̂ − f∗‖2 . The proposed estimator in ( 5 ) is based on a regularization of the hinge loss with two fold ` 1 penalty terms , where one is used to control nonparametric sparsity and the other is to control functional smoothness . Particularly , Corollary 1 shows that with high probability , the excess risk of the proposed estimator is upper bounded by s Γ2 ( S , ρX ) O ( n− 1 1+τ + logM n ) κ 2κ−1 under some regularity conditions , where τ corresponds to the spectral decay of each RKHS and is used to characterize functional complexity of kernels , κ is called the Bernstein parameter that reflects the low-density level of decision boundary , and Γ ( S , ρX ) is used to characterize correlation structure among multiple RKHS ’ s . The same rate also holds true for ‖f̂ − f∗‖2κ2 following the Bernstein condition and the established results on the excess risk . Furthermore , we establish the oracle rate of the relative misclassification error of f̂ in Theorem 2 , which consists of the same rate as the excess risk of the estimator plus some additional sparse approximation error . By using this specific form of RKHS ’ s , we can incorporate many existing results as special cases of MKL for SVMs , and more importantly , several existing results can be improved by our derived ones . The detailed comparison between our general results with existing results is conducted in Section 3.3 . Notations . Define the L2 norm of a function f by ‖f‖22 = ∫ X |f ( x ) |2dρX ( x ) with marginal distribution ρX . Given sample points { xi } ni=1 , ‖fm‖2n : = 1n ∑n i=1 fm ( xi ) 2 and ‖fm‖n is viewed as the empirical L2-norm . a ' bmeans that there are two positive constants c , C such that ca ≤ b ≤ Ca . Also , a = O ( b ) means that there is some positive constant C such that a ≤ Cb in probability , and a = o ( b ) means that a/b→ 0 in probability . To ease the notations , we write [ M ] : = { 1 , ... , M } . 2 Preliminaries and Algorithms . Given a compact set X , we denote a positive semidefinite kernel on X ×X , i.e. , a symmetric function K : X × X → R satisfying ∑n i , j=1 cicjK ( xi , xj ) ≥ 0 for any x1 , ... , xn in X and c1 , ... , cn ∈ R. As showed in [ 2 ] , a positive semidefinite kernel on X is associated with a unique Hilbert space HK consisting of functions on X , andHK is called a RKHS associated with the kernel K. RKHS is known for its reproducing property , i.e. , for any f ∈ HK and x ∈ X , f ( x ) = 〈f , Kx〉K with Kx = K ( x , · ) . Another key property of RKHS is the spectral theorem , which assures that K admits the following eigen-decomposition : K ( x , x′ ) = ∑ ` ≥1 µ ` ψ ` ( x ) ψ ` ( x ′ ) , for any x , x′ ∈ X , ( 2 ) where µ1 ≥ µ2 ≥ ... ≥ 0 are its eigenvalues and { ψ ` : ` ≥ 1 } are the corresponding eigenfunctions , leading to an orthogonal basis in L2 ( ρX ) . These two fundamental properties of RKHS will be the foundation of our theoretical analysis . To characterize functional complexities , we first introduce several basic facts of the empirical processes defined in RKHS . On basis of ( 2 ) , for any δ ∈ ( 0 , 1 ] , we define ωn ( δ ) : = ( 1 n n∑ ` =1 ( µ ` ∧ δ2 ) ) 1/2 , where a ∧ b means min { a , b } . For some constant A > 0 , we define a quantity associated with ωn ( δ ) as ( K ) : = inf { ≥ √ A logM n : ωn ( δ ) ≤ δ + 2 , ∀ δ ∈ ( 0 , 1 ] } . The quantity ( K ) plays a key role in bounding the excess risk in RKHS [ 40 , 20 ] . Given appropriate decay of µ ` , the upper bound of ( K ) can be derived explicitly in Section 3.1 . Given the training sample { Xi , Yi } ni=1 i.i.d . from X ×Y , the standard SVM with a single kernel [ 45 ] solves the following optimization task min f∈HK { 1 n n∑ i=1 φ ( Yif ( Xi ) ) + λ‖f‖2K } , ( 3 ) where λ is a regularization parameter that balances the tradeoff between the empirical risk and the function complexity . The theoretical properties of the standard SVM has been extensively studied in literature ; see the earlier work of [ 11 , 39 , 7 ] . In the context of MKL , the sparse multi-kernel SVM [ 34 ] equips the empirical hinge loss with the ` 1 penalty , min f= ∑M m=1 fm , fm∈HKm { 1 n n∑ i=1 φ ( Yif ( Xi ) ) + λ M∑ m=1 ‖fm‖Km } . ( 4 ) This type of sparse MKL method has been studied extensively in [ 34 , 3 , 20 , 16 ] , and among others . Essentially , it can be regarded as an infinite-dimensional version of the Lasso-type penalization [ 43 ] . In this paper , we present a different approach from ( 4 ) to formulate the sparse multi-kernel SVM . For simplicity , we assume that supx∈X |Km ( x , x ) | ≤ 1 for all m ∈ [ M ] . We use a new L1-type regularization term partially inspired by the additive mean models [ 32 ] . Denote the bounded ball ofHM by BM : = { f = ∑M j=1 fm : fm ∈ HKm , ‖fm‖Km ≤ 1 } , the regularization term we adopt for the multiple-kernel SVM combines the empirical L2-norms and RKHS-norms . Specifically , the proposed sparse multi-kernel SVM is formulated as f̂ = M∑ m=1 f̂m : = arg min f∈BM { 1 n n∑ i=1 φ ( Yif ( Xi ) ) + M∑ m=1 λm √ ‖fm‖2n + γm‖fm‖2Km } , ( 5 ) where ‖fm‖n is used to enforce the sparsity in f̂ , whereas ‖fm‖Km is used to enforce the smoothness of each f̂m . Here ( λm , γm ) Mm=1 are the regularization parameters , which will be specified in our theoretical results . By finite representation of reproducing kernel , each additive estimator fm ( · ) = ∑n i=1 α m i Km ( xi , · ) for all m = 1 , ... , M . A direct computation leads to√ ‖fm‖2n + γn‖fm‖2K = √ ( αm ) T K̃mαm where K̃m = K2m n + γnKm . Here Km is the kernel matrix induced by Km at points { xi } n i=1 . Note that K̃m is a semi-definite matrix , which can be written as K̃m = A2 with some matrix A . Hence , our original learning scheme in Eq . ( 5 ) can be transformed into a group Lasso optimization [ 32 ] , and there exists several efferent numerical algorithms for solving it , such as proximal methods and coordinate descent ones . This penalty term significantly differs from other sparsity penalties for nonparametric models in literature , such as ∑M m=1 ‖fm‖Km [ 22 ] and ∑M m=1 ( λ1‖fm‖n +λ2‖fm‖Km ) [ 20 , 37 ] . Although the former penalty often generates sparse solutions , it is difficult to establish its theoretical results , due to the fact that the ‖ · ‖K-norm can not fully reflect the marginal distribution information . The latter penalty has been proved to enjoy some theoretical properties in [ 37 ] , which is equivalent to the mixed L1-norm ( ∑M m=1 λm √ ‖fm‖2n + γm‖fm‖2Km ) in ( 5 ) . Theoretically , based on empirical processes theory , our proposed approach can also achieve improved learning rates , like the penalized method with the penalty ∑M m=1 ( λ1‖fm‖n + λ2‖fm‖Km ) , which has been considered in [ 20 , 31 , 37 ] and others . Remark . In general , ‖f̂‖n = 0 does not imply f = 0 . However , in the case of any kernel-based minimization problem , ‖f̂‖n = 0 always implies f = 0 . Based on the reproducing property of Mercer kernel , f ( x ) = 〈f , Kx〉K for any f ∈ HK , where HK is a reproducing kernel Hilbert space . In fact , if ‖f̂‖n = 0 holds , that is , f̂ ( xi ) = 0 for all i = 1 , . . . , n , by the reproducing property we have 〈f̂ , Kxi〉K = 0 for all i . Hence , f̂ is orthogonal to the subspace Sn : = span { Kx1 , ... , Kxn } . On the other hand , using the reproducing property again , any solution f̂ of kernel-based minimization problems has a finite representation within Sn . So we conclude f̂ = 0 . Remark . The use of the mixed norm regularization is mainly motivated by the following fact : i ) the proposed estimation with the mixed norm can lead to very sharp learning rates ; ii ) the empirical norm ‖ · ‖n used for sparsity is much milder than ‖ · ‖K . There exists several related work on multi-kernel SVM . Under the setting of one-dimensional additive kernels , Christmann and Hable [ 12 ] constructed kernels for additive SVM and provided consistent and statistically robust estimators under the fixed dimensional and non-sparse setting . Zhao and Liu [ 53 ] proposed a group Lasso penalty by means of finite-bases approximation to a RKHS , and particularly they developed an efficient accelerated proximal gradient descent algorithm and established oracle properties of the SVM under sparse ultra-high dimensional setting ( e.g . M = o ( en ) ) . However , the additive SVM model may suffer from the lack of algorithmic flexibility and underfitting especially when the true model involves interaction effects . Similar concerns have been raised towards linear SVM as in [ 55 , 21 , 52 ] . Note that in the past decade , a lot of work on general MKL with logarithmic dependence on M have emerged [ 18 , 41 , 20 ] , yet their analysis requires the loss function to be strongly convex , which rules out the commonly-used hinge loss for SVM . We also note that [ 13 ] provided an upper bound Op ( √ logM/n ) of the Rademacher complexity with the L1-norm constraint , which may lead to the same decay rate of the excess risk of SVM . However , this rate is not tight in general , since it is known that the fast rate of order 1/n can be attained for the linear case [ 11 , 44 ] . This paper primarily focuses on the non-asymptotic analysis of the proposed sparse multi-kernel SVM method in ( 5 ) with an exponential number of kernels . Under the best ideal settings , the relative classification error of the proposed method is of near-oracle rate O ( s log ( M ) /n ) , as if we knew the true sparsity in advance . Moreover , the method is adaptive to the sparsity of the learning problem and the margin parameter . In our proof , we have to face some technical challenges , such as dealing with non-smoothness of the hinge loss , functional complexities and NP-dimensionality .

Summary: This paper focus on the theoretical analysis of SVM with sparse multi-kernel approximation, where the total number of kernels may be larger than the sample size. More specifically, the refined bounds for the excess risk on multi-kernel SVM are provided, including high-dimensional linear or additive classification as special cases. The oracle rates of convergence of classifiers depend on the complexity of multi-kernels, the sparsity, the Bernstein condition and the sample size, which significantly improves the previous results even for the additive or linear cases.