component selection and smoothing in multivariate nonparametric regression we propose a new method for model selection and model fitting in multivariate nonparametric regression models in the framework of smoothing spline anova the cosso is a method of regularization with the penalty functional being the sum of component norms instead of the squared norm employed in the traditional smoothing spline method the cosso provides a unified framework for several recent proposals for model selection in linear models and smoothing spline anova models theoretical properties such as the existence and the rate of convergence of the cosso estimator are studied in the special case of a tensor product design with periodic functions a detailed analysis reveals that the cosso does model selection by applying a novel soft thresholding type operation to the function components we give an equivalent formulation of the cosso estimator which leads naturally to an iterative algorithm we compare the cosso with mars a popular method that builds functional anova models in simulations and real examples the cosso method can be extended to classification problems and we compare its performance with those of a number of machine learning algorithms on real datasets the cosso gives very competitive performance in these studies

consistency of the group lasso and multiple kernel learning we consider the least square regression problem with regularization by a block l1 norm that is a sum of euclidean norms over spaces of dimensions larger than one this problem referred to as the group lasso extends the usual regularization by the l1 norm where all spaces have dimension one where it is commonly referred to as the lasso in this paper we study the asymptotic group selection consistency of the group lasso we derive necessary and sufficient conditions for the consistency of group lasso under practical assumptions such as model mis specification when the linear predictors and euclidean norms are replaced by functions and reproducing kernel hilbert norms the problem is usually referred to as multiple kernel learning and is commonly used for learning from heterogeneous data sources and for non linear variable selection using tools from functional analysis and in particular covar iance operators we extend the consistency results to this infinite dimensional case and also propose an adaptive scheme to obtain a consistent model estimate even when the necessary condition required for the non adaptive scheme is not satisfied

structured multicategory support vector machines with analysis of variance decomposition the support vector machine has been a popular choice of classification method for many applications in machine learning while it often outperforms other methods in terms of classification accuracy the implicit nature of its solution renders the support vector machine less attractive in providing insights into the relationship between covariates and classes use of structured kernels can remedy the drawback borrowing the flexible model building idea of functional analysis of variance decomposition we consider multicategory support vector machines with analysis of variance kernels in this paper an additional penalty is imposed on the sum of weights of functional subspaces which encourages a sparse representation of the solution incorporation of the additional penalty enhances the interpretability of a resulting classifier with often improved accuracy the proposed method is demonstrated through simulation studies and an application to real data

ell_ p ell_ q penalty for sparse linear and sparse multiple kernel multitask learning recently there has been much interest around multitask learning mtl problem with the constraints that tasks should share a common sparsity profile such a problem can be addressed through a regularization framework where the regularizer induces a joint sparsity pattern between task decision functions we follow this principled framework and focus on lp lq with 0 p 1 and 1 q 2 mixed norms as sparsity inducing penalties our motivation for addressing such a larger class of penalty is to adapt the penalty to a problem at hand leading thus to better performances and better sparsity pattern for solving the problem in the general multiple kernel case we first derive a variational formulation of the l1 lq penalty which helps us in proposing an alternate optimization algorithm although very simple the latter algorithm provably converges to the global minimum of the l1 lq penalized problem for the linear case we extend existing works considering accelerated proximal gradient to this penalty our contribution in this context is to provide an efficient scheme for computing the l1 lq proximal operator then for the more general case when 0 p 1 we solve the resulting nonconvex problem through a majorization minimization approach the resulting algorithm is an iterative scheme which at each iteration solves a weighted l1 lq sparse mtl problem empirical evidences from toy dataset and real word datasets dealing with brain computer interface single trial electroencephalogram classification and protein subcellular localization show the benefit of the proposed approaches and algorithms

a generalization of n matrices abstract ky fan defines an n matrix to be a matrix of the form a ti b b 0 t b where b is the spectral radius of b and is the maximum of the spectral radii of all principal submatrices of b of order n 1 in this paper we define the closure n 0 matrices of n matrices by letting t it is shown that if a z and a 1 a n certain inequalities of n matrices are shown to hold for n 0 matrices and a method for constructing an n matrix from an m matrix is given

polynomial maps and a conjecture of samuelson abstract let f r n r n be a c 1 mapping it was conjectured by samuelson that if the upper left hand principal minors of the jacobian of f do not vanish on r n then f is injective however in 1965 gale and nikaido gave a simple counterexample to the case n 2 in this paper we show that the samuelson conjecture is true for polynomial mappings from c n to c n furthermore we give a precise description of such maps

deterministic error analysis of support vector regression and related regularized kernel methods we introduce a new technique for the analysis of kernel based regression problems the basic tools are sampling inequalities which apply to all machine learning problems involving penalty terms induced by kernels related to sobolev spaces they lead to explicit deterministic results concerning the worst case behaviour of e and svrs using these we show how to adjust regularization parameters to get best possible approximation orders for regression the results are illustrated by some numerical examples

classifier learning with a new locality regularization method it is well known that the generalization capability is one of the most important criterions to develop and evaluate a classifier for a given pattern classification problem the localized generalization error model r s m recently proposed by ng et al localized generalization error and its application to rbfnn training in proceedings of the international conference on machine learning and cybernetics china 2005 image classification with the use of radial basis function neural networks and the minimization of the localized generalization error pattern recognition 40 1 2007 4 18 provides a more intuitive look at the generalization error although r s m gives a brand new method to promote the generalization performance it is in nature equivalent to another type of regularization in this paper we first prove the essential relationship between r s m and regularization and demonstrate that the stochastic sensitivity measure in r s m exactly corresponds to a regularizing term then we develop a new generalization error bound from the regularization viewpoint which is inspired by the proved relationship between r s m and regularization moreover we derive a new regularization method called as locality regularization lr from the bound different from the existing regularization methods which artificially and externally append the regularizing term in order to smooth the solution lr is naturally and internally deduced from the defined expected risk functional and calculated by employing locality information through combining with spectral graph theory lr introduces the local structure information of the samples into the regularizing term and further improves the generalization capability in contrast with r s m which is relatively sensitive to the different sampling of the samples lr uses the discrete k neighborhood rather than the common continuous q neighborhood in r s m to differentiate the relative position of different training samples automatically and avoid the complex computation of q for various classifiers furthermore lr uses the regularization parameter to control the trade off between the training accuracy and the classifier stability experimental results on artificial and real world problems show that lr yields better generalization capability than both r s m and some traditional regularization methods

the entire regularization path for the support vector machine the support vector machine svm is a widely used tool for classification many efficient implementations exist for fitting a two class svm model the user has to supply values for the tuning parameters the regularization cost parameter and the kernel parameters it seems a common practice is to use a default value for the cost parameter often leading to the least restrictive model in this paper we argue that the choice of the cost parameter can be critical we then derive an algorithm that can fit the entire path of svm solutions for every value of the cost parameter with essentially the same computational cost as fitting one svm model we illustrate our algorithm on some examples and use our representation to give further insight into the range of svm solutions

on the convergence rate of l p norm multiple kernel learning we derive an upper bound on the local rademacher complexity of lp norm multiple kernel learning which yields a tighter excess risk bound than global approaches previous local approaches analyzed the case p 1 only while our analysis covers all cases 1 p assuming the different feature mappings corresponding to the different kernels to be uncorrelated we also show a lower bound that shows that the bound is tight and derive consequences regarding excess loss namely fast convergence rates of the order o n 1 a where is the minimum eigenvalue decay rate of the individual kernels

multiple kernel learning with gaussianity measures kernel methods are known to be effective for nonlinear multivariate analysis one of the main issues in the practical use of kernel methods is the selection of kernel there have been a lot of studies on kernel selection and kernel learning multiple kernel learning mkl is one of the promising kernel optimization approaches kernel methods are applied to various classifiers including fisher discriminant analysis fda fda gives the bayes optimal classification axis if the data distribution of each class in the feature space is a gaussian with a shared covariance structure based on this fact an mkl framework based on the notion of gaussianity is proposed as a concrete implementation an empirical characteristic function is adopted to measure gaussianity in the feature space associated with a convex combination of kernel functions and two mkl algorithms are derived from experimental results on some data sets we show that the proposed kernel learning followed by fda offers strong classification power

orthogonality from disjoint support in reproducing kernel hilbert spaces we investigate reproducing kernel hilbert spaces rkhs where two functions are orthogonal whenever they have disjoint support necessary and sufficient conditions in terms of feature maps for the reproducing kernel are established we also present concrete examples of finite dimensional rkhs and rkhs with a translation invariant reproducing kernel in particular it is shown that a sobolev space has the orthogonality from disjoint support property if and only if it is of integer index

kernels for linear time invariant system identification in this paper we study the problem of identifying the impulse response of a linear time invariant lti dynamical system from the knowledge of the input signal and a finite set of noisy output observations we adopt an approach based on regularization in a reproducing kernel hilbert space rkhs that takes into account both continuous and discrete time systems the focus of the paper is on designing spaces that are well suited for temporal impulse response modeling to this end we construct and characterize general families of kernels that incorporate system properties such as stability relative degree absence of oscillatory behavior smoothness or delay in addition we discuss the possibility of automatically searching over these classes by means of kernel learning techniques so as to capture different modes of the system to be identified

multilevel augmentation algorithms based on fast collocation methods for solving ill posed integral equations abstract this paper continues the theme of the recent work chen et al 2008 18 in which fast collocation methods are introduced for solving ill posed fredholm integral equations of the first kind we develop in this paper multilevel augmentation algorithms which lead to fast solutions of the discrete equations resulting from fast collocation methods regularization parameter choice strategies are given for proposed methods the theoretical analysis and numerical experiments illustrate the accuracy and efficiency of the algorithm

integer solution for linear complementarity problem we consider the problem of finding an integer solution to a linear complementarity problem we introduce the class i of matrices for which the corresponding linear complementarity problem has an integer complementary solution for every vector q for which it has a solution strong principal unimodularity forms a sufficient condition for inclusion in the class i it is also shown to be necessary for some well known classes of matrices including the class of positive semidefinite matrices this is used in deriving necessary and sufficient conditions for a convex quadratic program to have an integer optimal solution for all b and c for which it has an optimal solution characterizations are also derived for some other well known classes of matrices in linear complementarity to belong to the class i in the end a peeling algorithm for finding integer solution for linear complementarity problems is given and related results are derived

lipschitzian q matrices are p matrices in this note we show that lipschitzianq matrices arep matrices by obtaining a necessary condition on lipschitzianq0 matrices the sufficiency of this condition has also been established by the first two authors along with another coauthor murthy parthasarathy and sriparna 1995

an equivalence between sparse approximation and support vector machines in the first part of this paper we show a similarity between the principle of structural risk minimization principle srm vapnik 1982 and the idea of sparse approximation as defined in chen donoho and saunders 1995 and olshausen and field 1996 then we focus on two specific approximate implementations of srm and sparse approximation which have been used to solve the problem of function approximation for srm we consider the support vector machine technique proposed by v vapnik and his team at at t bell labs and for sparse approximation we consider a modification of the basis pursuit de noising algorithm proposed by chen donoho and saunders 1995 we show that under certain conditions these two techniques are equivalent they give the same solution and they require the solution of the same quadratic programming problem

almost n matrices and linear complementarity abstract an almost n matrix a is one with real entries whose determinant is positive and proper principal minors are negative in this paper we obtain some characterization results for almost n matrices that are similar to n matrices among other things we show that an almost n matrix of first category with n 4 belongs to the class of q matrices as an application we derive a new univalence result

statistical behavior and consistency of classification methods based on convex risk minimization we study how closely the optimal bayes error rate can be approximately reached using a classification algorithm that computes a classifier by minimizing a convex upper bound of the classification error function the measurement of closeness is characterized by the loss function used in the estimation we show that such a classification scheme can be generally regarded as a nonmaximum likelihood conditional in class probability estimate and we use this analysis to compare various convex loss functions that have appeared in the literature furthermore the theoretical insight allows us to design good loss functions with desirable properties another aspect of our analysis is to demonstrate the consistency of certain classification methods using convex risk minimization this study sheds light on the good performance of some recently proposed linear classification methods including boosting and support vector machines it also shows their limitations and suggests possible improvements

generalized hierarchical kernel learning this paper generalizes the framework of hierarchical kernel learning hkl and illustrates its utility in the domain of rule learning hkl involves multiple kernel learning over a set of given base kernels assumed to be embedded on a directed acyclic graph this paper proposes a two fold generalization of hkl the first is employing a generic l1 lp block norm regularizer 1 2 that alleviates a key limitation of the hkl formulation the second is a generalization to the case of multi class multi label and more generally multi task applications the main technical contribution of this work is the derivation of a highly specialized partial dual of the proposed generalized hkl formulation and an efficient mirror descent based active set algorithm for solving it importantly the generic regularizer enables the proposed formulation to be employed in the rule ensemble learning rel where the goal is to construct an ensemble of conjunctive propositional rules experiments on benchmark rel data sets illustrate the efficacy of the proposed generalizations

learning the kernel matrix by maximizing a kfd based class separability criterion the advantage of a kernel method often depends critically on a proper choice of the kernel function a promising approach is to learn the kernel from data automatically in this paper we propose a novel method for learning the kernel matrix based on maximizing a class separability criterion that is similar to those used by linear discriminant analysis lda and kernel fisher discriminant kfd it is interesting to note that optimizing this criterion function does not require inverting the possibly singular within class scatter matrix which is a computational problem encountered by many lda and kfd methods we have conducted experiments on both synthetic data and real world data from uci and feret showing that our method consistently outperforms some previous kernel learning methods

building sparse multiple kernel svm classifiers the support vector machines svms have been very successful in many machine learning problems however they can be slow during testing because of the possibly large number of support vectors obtained recently wu 2005 proposed a sparse formulation that restricts the svm to use a small number of expansion vectors in this paper we further extend this idea by integrating with techniques from multiple kernel learning mkl the kernel function in this sparse svm formulation no longer needs to be fixed but can be automatically learned as a linear combination of kernels two formulations of such sparse multiple kernel classifiers are proposed the first one is based on a convex combination of the given base kernels while the second one uses a convex combination of the so called ldquoequivalentrdquo kernels empirically the second formulation is particularly competitive experiments on a large number of toy and real world data sets show that the resultant classifier is compact and accurate and can also be easily trained by simply alternating linear program and standard svm solver

distributed parametric and nonparametric regression with on line performance bounds computation in this paper we focus on collaborative multi agent systems where agents are distributed over a region of interest and collaborate to achieve a common estimation goal in particular we introduce two consensus based distributed linear estimators the first one is designed for a bayesian scenario where an unknown common finite dimensional parameter vector has to be reconstructed while the second one regards the nonparametric reconstruction of an unknown function sampled at different locations by the sensors both of the algorithms are characterized in terms of the trade off between estimation performance communication computation and memory complexity in the finite dimensional setting we derive mild sufficient conditions which ensure that a distributed estimator performs better than the local optimal ones in terms of estimation error variance in the nonparametric setting we introduce an on line algorithm that allows the agents to simultaneously compute the function estimate with small computational communication and data storage efforts as well as to quantify its distance from the centralized estimate given by a regularization network one of the most powerful regularized kernel methods these results are obtained by deriving bounds on the estimation error that provide insights on how the uncertainty inherent in a sensor network such as imperfect knowledge on the number of agents and the measurement models used by the sensors can degrade the performance of the estimation process numerical experiments are included to support the theoretical findings

fast learning rate of multiple kernel learning trade off between sparsity and smoothness we investigate the learning rate of multiple kernel leaning mkl with l1 and elastic net regularizations the elastic net regularization is a composition of an l1 regularizer for inducing the sparsity and an l2 regularizer for controlling the smoothness we focus on a sparse setting where the total number of kernels is large but the number of non zero components of the ground truth is relatively small and show sharper convergence rates than the learning rates ever shown for both l1 and elastic net regularizations our analysis shows there appears a trade off between the sparsity and the smoothness when it comes to selecting which of l1 and elastic net regularizations to use if the ground truth is smooth the elastic net regularization is preferred otherwise the l1 regularization is preferred

a family of simple non parametric kernel learning algorithms previous studies of non parametric kernel learning npkl usually formulate the learning task as a semi definite programming sdp problem that is often solved by some general purpose sdp solvers however for n data examples the time complexity of npkl using a standard interior point sdp solver could be as high as o n6 5 which prohibits npkl methods applicable to real applications even for data sets of moderate size in this paper we present a family of efficient npkl algorithms termed simplenpkl which can learn non parametric kernels from a large set of pairwise constraints efficiently in particular we propose two efficient simplenpkl algorithms one is simplenpkl algorithm with linear loss which enjoys a closed form solution that can be efficiently computed by the lanczos sparse eigen decomposition technique another one is simplenpkl algorithm with other loss functions including square hinge loss hinge loss square loss that can be re formulated as a saddle point optimization problem which can be further resolved by a fast iterative algorithm in contrast to the previous npkl approaches our empirical results show that the proposed new technique maintaining the same accuracy is significantly more efficient and scalable finally we also demonstrate that the proposed new technique is also applicable to speed up many kernel learning tasks including colored maximum variance unfolding minimum volume embedding and structure preserving embedding

decomposing samuelson maps abstract a samuelson map is here defined as a differentiable self mapping of n space with the property that the leading principal minors of its jacobian matrix vanish nowhere structure theorems are proved for samuelson maps that satisfy various simple conditions on the pivots arising from gaussian elimination of the jacobian matrix the theorems yield representations of a samuelson map as a unique composition of invertible maps that alter only a single coordinate

learning with sample dependent hypothesis spaces many learning algorithms use hypothesis spaces which are trained from samples but little theoretical work has been devoted to the study of these algorithms in this paper we show that mathematical analysis for these algorithms is essentially different from that for algorithms with hypothesis spaces independent of the sample or depending only on the sample size the difficulty lies in the lack of a proper characterization of approximation error to overcome this difficulty we propose an idea of using a larger function class not necessarily linear space containing the union of all possible hypothesis spaces varying with the sample to measure the approximation ability of the algorithm we show how this idea provides error analysis for two particular classes of learning algorithms in kernel methods learning the kernel via regularization and coefficient based regularization we demonstrate the power of this approach by its wide applicability

matrices and the linear complementarity problem the paper is a collection of results on the linear complementarity problem q m the results are stated in terms of the matrix m

stability and generalization we define notions of stability for learning algorithms and show how to use these notions to derive generalization error bounds based on the empirical error and the leave one out error the methods we use can be applied in the regression framework as well as in the classification one when the classifier is obtained by thresholding a real valued function we study the stability properties of large classes of learning algorithms such as regularization based algorithms in particular we focus on hilbert space regularization and kullback leibler regularization we demonstrate how to apply the results to svm for regression and classification

the kernel least mean square algorithm the combination of the famed kernel trick and the least mean square lms algorithm provides an interesting sample by sample update for an adaptive filter in reproducing kernel hilbert spaces rkhs which is named in this paper the klms unlike the accepted view in kernel methods this paper shows that in the finite training data case the klms algorithm is well posed in rkhs without the addition of an extra regularization term to penalize solution norms as was suggested by kivinen kivinen smola and williamson ldquoonline learning with kernels rdquo ieee transactions on signal processing vol 52 no 8 pp 2165 2176 aug 2004 and smale smale and yao ldquoonline learning algorithms rdquo foundations in computational mathematics vol 6 no 2 pp 145 176 2006 this result is the main contribution of the paper and enhances the present understanding of the lms algorithm with a machine learning perspective the effect of the klms step size is also studied from the viewpoint of regularization two experiments are presented to support our conclusion that with finite data the klms algorithm can be readily used in high dimensional spaces and particularly in rkhs to derive nonlinear stable algorithms with comparable performance to batch regularized solutions

microwave billiards with broken time reversal invariance we consider a microwave resonator with three single channel waveguides attached one of these serves to couple waves into and out of the resonator the remaining two are connected to form a one way handle so as to break time reversal invariance the poles of the input output scattering coefficient of such a resonator are shown to be the eigenvalues of a non hermitian effective hamiltonian heff d h i0 the anti hermitian part 0 of which has rank 1 and is responsible for the breaking of time reversal invariance all of the spectral statistics recently observed for such a microwave billiard are reproduced quantitatively by taking h and 0 as random matrices in particular the distribution of nearest neighbour spacings of the resonances is close to that of the gue when h belongs to the goe corresponding to a sinai shape of the resonator linear level repulsion results when h belongs to the poissonian ensemble as it corresponds to a rectangular resonator

algorithms for learning kernels based on centered alignment this paper presents new and effective algorithms for learning kernels in particular as shown by our empirical results these algorithms consistently outperform the so called uniform combination solution that has proven to be difficult to improve upon in the past as well as other algorithms for learning kernels based on convex combinations of base kernels in both classification and regression our algorithms are based on the notion of centered alignment which is used as a similarity measure between kernels or kernel matrices we present a number of novel algorithmic theoretical and empirical results for learning kernels based on our notion of centered alignment in particular we describe efficient algorithms for learning a maximum alignment kernel by showing that the problem can be reduced to a simple qp and discuss a one stage algorithm for learning both a kernel and a hypothesis based on that kernel using an alignment based regularization our theoretical results include a novel concentration bound for centered alignment between kernel matrices the proof of the existence of effective predictors for kernels with high alignment both for classification and for regression and the proof of stability based generalization bounds for a broad family of algorithms for learning kernels based on centered alignment we also report the results of experiments with our centered alignment based algorithms in both classification and regression

parameter choice strategies for least squares approximation of noisy smooth functions on the sphere we consider a polynomial reconstruction of smooth functions from their noisy values at discrete nodes on the unit sphere by a variant of the regularized least squares method of an et al siam j numer anal 50 2012 pp 1513 1534 as nodes we use the points of a positive weight cubature formula that is exact for all spherical polynomials of degree up to 2m where m is the degree of the reconstructing polynomial we first obtain a reconstruction error bound in terms of the regularization parameter and the penalization parameters in the regularization operator then we discuss a priori and a posteriori strategies for choosing these parameters finally we give numerical examples illustrating the theoretical results

ky fan s n matrices and linear complementarity problems we consider the linear complementarity problem lcp w az q w 0 z 0 wtz 0 when all the off diagonal entries ofa are nonpositive the class of z matrices all the proper principal minors ofa are positive and the determinant ofa is negative the class of almost p matrices we shall call this the class of f matrices we show that ifa is a z matrix thena is an f matrix if and only if lcp q a has exactly two solutions for anyq 0 q 0 and has at most two solutions for any otherq

refinement of reproducing kernels we continue our recent study on constructing a refinement kernel for a given kernel so that the reproducing kernel hilbert space associated with the refinement kernel contains that with the original kernel as a subspace to motivate this study we first develop a refinement kernel method for learning which gives an efficient algorithm for updating a learning predictor several characterizations of refinement kernels are then presented it is shown that a nontrivial refinement kernel for a given kernel always exists if the input space has an infinite cardinal number refinement kernels for translation invariant kernels and hilbert schmidt kernels are investigated various concrete examples are provided

regularizing multiple kernel learning using response surface methodology in recent years several methods have been proposed to combine multiple kernels using a weighted linear sum of kernels these different kernels may be using information coming from multiple sources or may correspond to using different notions of similarity on the same source we note that such methods in addition to the usual ones of the canonical support vector machine formulation introduce new regularization parameters that affect the solution quality and in this work we propose to optimize them using response surface methodology on cross validation data on several bioinformatics and digit recognition benchmark data sets we compare multiple kernel learning and our proposed regularized variant in terms of accuracy support vector count and the number of kernels selected we see that our proposed variant achieves statistically similar or higher accuracy results by using fewer kernel functions and or support vectors through suitable regularization it also allows better knowledge extraction because unnecessary kernels are pruned and the favored kernels reflect the properties of the problem at hand

on characterizing n matrices using linear complementarity we show that a square matrix a with at least one positive entry and all principal minors negative can be characterized in terms of the number of solutions the linear complementarity problem q a with the matrix a has for different vectors q such a matrix can also be characterized in terms of a sign nonreversal property these results complement the known results for a square matrix a 0 with all its principal minors negative

multikernel adaptive filtering this paper exemplifies that the use of multiple kernels leads to efficient adaptive filtering for nonlinear systems two types of multikernel adaptive filtering algorithms are proposed one is a simple generalization of the kernel normalized least mean square knlms algorithm 2 adopting a coherence criterion for dictionary designing the other is derived by applying the adaptive proximal forward backward splitting method to a certain squared distance function plus a weighted block l1 norm penalty encouraging the sparsity of an adaptive filter at the block level for efficiency the proposed multikernel approach enjoys a higher degree of freedom than those approaches which design a kernel as a convex combination of multiple kernels numerical examples show that the proposed approach achieves significant gains particularly for nonstationary data as well as insensitivity to the choice of some design parameters

the two sample problem for poisson processes adaptive tests with a non asymptotic wild bootstrap approach considering two independent poisson processes we address the question of testing equality of their respective intensities we first propose single tests whose test statistics are u statistics based on general kernel functions the corresponding critical values are constructed from a non asymptotic wild bootstrap approach leading to level alpha tests various choices for the kernel functions are possible including projection approximation or reproducing kernels in this last case we obtain a parametric rate of testing for a weak metric defined in the rkhs associated with the considered reproducing kernel then we introduce in the other cases an aggregation procedure which allows us to import ideas coming from model selection thresholding and or approximation kernels adaptive estimation the resulting multiple tests are proved to be of level alpha and to satisfy non asymptotic oracle type conditions for the classical l2 norm from these conditions we deduce that they are adaptive in the minimax sense over a large variety of classes of alternatives based on classical and weak besov bodies in the univariate case but also sobolev and anisotropic nikol skii besov balls in the multivariate case

minimax optimal rates of estimation in high dimensional additive models we establish minimax optimal rates of convergence for estimation in a high dimensional additive model assuming that it is approximately sparse our results reveal a behavior universal to this class of high dimensional problems in the sparse regime when the components are sufficiently smooth or the dimensionality is sufficiently large the optimal rates are identical to those for high dimensional linear regression and therefore there is no additional cost to entertain a nonparametric model otherwise in the so called smooth regime the rates coincide with the optimal rates for estimating a univariate function and therefore they are immune to the curse of dimensionality

feature selection and kernel learning for local learning based clustering the performance of the most clustering algorithms highly relies on the representation of data in the input space or the hilbert space of kernel methods this paper is to obtain an appropriate data representation through feature selection or kernel learning within the framework of the local learning based clustering llc wu and scholkopf 2006 method which can outperform the global learning based ones when dealing with the high dimensional data lying on manifold specifically we associate a weight to each feature or kernel and incorporate it into the built in regularization of the llc algorithm to take into account the relevance of each feature or kernel for the clustering accordingly the weights are estimated iteratively in the clustering process we show that the resulting weighted regularization with an additional constraint on the weights is equivalent to a known sparse promoting penalty hence the weights of those irrelevant features or kernels can be shrunk toward zero extensive experiments show the efficacy of the proposed methods on the benchmark data sets

learning the kernel function via regularization we study the problem of finding an optimal kernel from a prescribed convex set of kernels k for learning a real valued function by regularization we establish for a wide variety of regularization functionals that this leads to a convex optimization problem and for square loss regularization we characterize the solution of this problem we show that although k may be an uncountable set the optimal kernel is always obtained as a convex combination of at most m 2 basic kernels where m is the number of data examples in particular our results apply to learning the optimal radial kernel or the optimal dot product kernel

when is there a representer theorem vector versus matrix regularizers we consider a general class of regularization methods which learn a vector of parameters on the basis of linear measurements it is well known that if the regularizer is a nondecreasing function of the l2 norm then the learned vector is a linear combination of the input data this result known as the representer theorem lies at the basis of kernel based methods in machine learning in this paper we prove the necessity of the above condition in the case of differentiable regularizers we further extend our analysis to regularization methods which learn a matrix a problem which is motivated by the application to multi task learning in this context we study a more general representer theorem which holds for a larger class of regularizers we provide a necessary and sufficient condition characterizing this class of matrix regularizers and we highlight some concrete examples of practical importance our analysis uses basic principles from matrix theory especially the useful notion of matrix nondecreasing functions

properties of support vector machines support vector machines svms perform pattern recognition between two point classes by finding a decision surface determined by certain points of the training set termed support vectors sv this surface which in some feature space of possibly infinite dimension can be regarded as a hyperplane is obtained from the solution of a problem of quadratic programming that depends on a regularization parameter in this paper we study some mathematical properties of support vectors and show that the decision surface can be written as the sum of two orthogonal terms the first depending only on the em margin vectors which are svs lying on the margin the second proportional to the regularization parameter for almost all values of the parameter this enables us to predict how the decision surface varies for small parameter changes in the special but important case of feature space of finite dimension m we also show that there are at most m 1 margin vectors and observe that m 1 svs are usually sufficient to fully determine the decision surface for relatively small m this latter result leads to a consistent reduction of the sv number

multi kernel learning with online batch optimization in recent years there has been a lot of interest in designing principled classification algorithms over multiple cues based on the intuitive notion that using more features should lead to better performance in the domain of kernel methods a principled way to use multiple features is the multi kernel learning mkl approach here we present a mkl optimization algorithm based on stochastic gradient descent that has a guaranteed convergence rate we directly solve the mkl problem in the primal formulation by having a p norm formulation of mkl we introduce a parameter that controls the level of sparsity of the solution while leading to an easier optimization problem we prove theoretically and experimentally that 1 our algorithm has a faster convergence rate as the number of kernels grows 2 the training complexity is linear in the number of training examples 3 very few iterations are sufficient to reach good solutions experiments on standard benchmark databases support our claims

kernel based reconstruction of graph signals a number of applications in engineering social sciences physics and biology involve inference over networks in this context graph signals are widely encountered as descriptors of vertex attributes or features in graph structured data estimating such signals in all vertices given noisy observations of their values on a subset of vertices has been extensively analyzed in the literature of signal processing on graphs spog this paper advocates kernel regression as a framework generalizing popular spog modeling and reconstruction and expanding their capabilities formulating signal reconstruction as a regression task on reproducing kernel hilbert spaces of graph signals permeates benefits from statistical learning offers fresh insights and allows for estimators that leverage richer forms of prior information than existing alternatives a number of spog notions such as bandlimitedness graph filters and the graph fourier transform are naturally accommodated in the kernel framework additionally this paper capitalizes on the so called representer theorem to devise simpler versions of existing tikhonov regularized estimators and offers a novel probabilistic interpretation of kernel methods on graphs based on graphical models motivated by the challenges of selecting the bandwidth parameter in spog estimators or the kernel map in kernel based methods this paper further proposes two multikernel approaches with complementary strengths whereas the first enables estimation of the unknown bandwidth of bandlimited signals the second allows for efficient graph filter selection numerical tests with synthetic as well as real data demonstrate the merits of the proposed methods relative to state of the art alternatives

on learning vector valued functions in this letter we provide a study of learning in a hilbert space of vectorvalued functions we motivate the need for extending learning theory of scalar valued functions by practical considerations and establish some basic results for learning vector valued functions that should prove useful in applications specifically we allow an output space y to be a hilbert space and we consider a reproducing kernel hilbert space of functions whose values lie in y in this setting we derive the form of the minimal norm interpolant to a finite set of data and apply it to study some regularization functionals that are important in learning theory we consider specific examples of such functionals corresponding to multiple output regularization networks and support vector machines for both regression and classification finally we provide classes of operator valued kernels of the dot product and translation invariant type

principal pivot transforms of some classes of matrices in this paper we study and characterize various classes of matrices that are defined based on principal pivot transforms we show that matrices in these classes have nonnegative principal minors

online primal dual learning for a data dependent multi kernel combination model with multiclass visual categorization applications the task of visual categorization becomes very challenging as the number of samples or classes increases this is mainly due to large memory requirements or a crowded semantic space which causes class separation difficulties in this paper a new approach called oprdum2 online primal dual multiple kernel multiple class is proposed this new algorithm can significantly reduce the complexity of visual categorization under this approach an elegant heterogeneous feature fusion machine is used to consolidate complementary class discriminative information and online learning is applied to deal with large sample sizes to be specific a data dependent multi kernel machine that fuses multiple heterogeneous features in a nonlinear manner is defined by extending the multi kernel learning model and multivariate hinge loss and conservative updating rules are used to increase the sample level sparsity in the model given the intrinsic hierarchical structure of samples features and classes an innovative triple mixed norm regularizer with strong convexity is utilized to facilitate optimization an iterative solution is derived using the language of duality leading to a low regret bound of o t where t is the number of online iterations this paper details a proof of the algorithm s convergence and describes extensive visual data classification experiments that demonstrate the effectiveness and robustness of this new approach

learning translation invariant kernels for classification appropriate selection of the kernel function which implicitly defines the feature space of an algorithm has a crucial role in the success of kernel methods in this paper we consider the problem of optimizing a kernel function over the class of translation invariant kernels for the task of binary classification the learning capacity of this class is invariant with respect to rotation and scaling of the features and it encompasses the set of radial kernels we show that how translation invariant kernel functions can be embedded in a nested set of sub classes and consider the kernel learning problem over one of these sub classes this allows the choice of an appropriate sub class based on the problem at hand we use the criterion proposed by lanckriet et al 2004 to obtain a functional formulation for the problem it will be proven that the optimal kernel is a finite mixture of cosine functions the kernel learning problem is then formulated as a semi infinite programming sip problem which is solved by a sequence of quadratically constrained quadratic programming qcqp sub problems using the fact that the cosine kernel is of rank two we propose a formulation of a qcqp sub problem which does not require the kernel matrices to be loaded into memory making the method applicable to large scale problems we also address the issue of including other classes of kernels such as individual kernels and isotropic gaussian kernels in the learning process another interesting feature of the proposed method is that the optimal classifier has an expansion in terms of the number of cosine kernels instead of support vectors leading to a remarkable speedup at run time as a by product we also generalize the kernel trick to complex valued kernel functions our experiments on artificial and real world benchmark data sets including the usps and the mnist digit recognition data sets show the usefulness of the proposed method

learning the kernel matrix with semidefinite programming kernel based learning algorithms work by embedding the data into a euclidean space and then searching for linear relations among the embedded data points the embedding is performed implicitly by specifying the inner products between each pair of points in the embedding space this information is contained in the so called kernel matrix a symmetric and positive semidefinite matrix that encodes the relative positions of all points specifying this matrix amounts to specifying the geometry of the embedding space and inducing a notion of similarity in the input space classical model selection problems in machine learning in this paper we show how the kernel matrix can be learned from data via semidefinite programming sdp techniques when applied to a kernel matrix associated with both training and test data this gives a powerful transductive algorithm using the labeled part of the data one can learn an embedding also for the unlabeled part the similarity between test points is inferred from training points and their labels importantly these learning problems are convex so we obtain a method for learning both the model class and the function without local minima furthermore this approach leads directly to a convex method for learning the 2 norm soft margin parameter in support vector machines solving an important open problem

multiple kernel learning for sparse representation based classification in this paper we propose a multiple kernel learning mkl algorithm that is based on the sparse representation based classification src method taking advantage of the nonlinear kernel src in efficiently representing the nonlinearities in the high dimensional feature space we propose an mkl method based on the kernel alignment criteria our method uses a two step training method to learn the kernel weights and sparse codes at each iteration the sparse codes are updated first while fixing the kernel mixing coefficients and then the kernel mixing coefficients are updated while fixing the sparse codes these two steps are repeated until a stopping criteria is met the effectiveness of the proposed method is demonstrated using several publicly available image classification databases and it is shown that this method can perform significantly better than many competitive image classification algorithms

some properties of regularized kernel methods in regularized kernel methods the solution of a learning problem is found by minimizing functionals consisting of the sum of a data and a complexity term in this paper we investigate some properties of a more general form of the above functionals in which the data term corresponds to the expected risk first we prove a quantitative version of the representer theorem holding for both regression and classification for both differentiable and non differentiable loss functions and for arbitrary offset terms second we show that the case in which the offset space is non trivial corresponds to solving a standard problem of regularization in a reproducing kernel hilbert space in which the penalty term is given by a seminorm finally we discuss the issues of existence and uniqueness of the solution from the specialization of our analysis to the discrete setting it is immediate to establish a connection between the solution properties of sparsity and coefficient boundedness and some properties of the loss function for the case of support vector machines for classification we also obtain a complete characterization of the whole method in terms of the khun tucker conditions with no need to introduce the dual formulation

algorithmic detection and construction of n matrices abstract n matrices are real n n matrices all of whose principal minors are negative we provide i an o 2 n test to detect whether or not a given matrix is an n matrix and ii a characterization of n matrices leading to the recursive construction of every n matrix

discriminatively regularized least squares classification over the past decades regularization theory is widely applied in various areas of machine learning to derive a large family of novel algorithms traditionally regularization focuses on smoothing only and does not fully utilize the underlying discriminative knowledge which is vital for classification in this paper we propose a novel regularization algorithm in the least squares sense called discriminatively regularized least squares classification drlsc method which is specifically designed for classification inspired by several new geometrically motivated methods drlsc directly embeds the discriminative information as well as the local geometry of the samples into the regularization term so that it can explore as much underlying knowledge inside the samples as possible and aim to maximize the margins between the samples of different classes in each local area furthermore by embedding equality type constraints in the formulation the solutions of drlsc can follow from solving a set of linear equations and the framework naturally contains multi class problems experiments on both toy and real world problems demonstrate that drlsc is often superior in classification performance to the classical regularization algorithms including regularization networks support vector machines and some of the recent studied manifold regularization techniques

efficient sparse generalized multiple kernel learning kernel methods have been successfully applied in various applications to succeed in these applications it is crucial to learn a good kernel representation whose objective is to reveal the data similarity precisely in this paper we address the problem of multiple kernel learning mkl searching for the optimal kernel combination weights through maximizing a generalized performance measure most mkl methods employ the norm simplex constraints on the kernel combination weights which therefore involve a sparse but non smooth solution for the kernel weights despite the success of their efficiency they tend to discard informative complementary or orthogonal base kernels and yield degenerated generalization performance alternatively imposing the norm constraint on the kernel weights will keep all the information in the base kernels this leads to non sparse solutions and brings the risk of being sensitive to noise and incorporating redundant information to tackle these problems we propose a generalized mkl gmkl model by introducing an elastic net type constraint on the kernel weights more specifically it is an mkl model with a constraint on a linear combination of the norm and the squared norm on the kernel weights to seek the optimal kernel combination weights therefore previous mkl problems based on the norm or the norm constraints can be regarded as special cases furthermore our gmkl enjoys the favorable sparsity property on the solution and also facilitates the grouping effect moreover the optimization of our gmkl is a convex optimization problem where a local solution is the global optimal solution we further derive a level method to efficiently solve the optimization problem a series of experiments on both synthetic and real world datasets have been conducted to show the effectiveness and efficiency of our gmkl