the em algorithm for kernel matrix completion with auxiliary data in biological data it is often the case that observed data are available only for a subset of samples when a kernel matrix is derived from such data we have to leave the entries for unavailable samples as missing in this paper the missing entries are completed by exploiting an auxiliary kernel matrix derived from another information source the parametric model of kernel matrices is created as a set of spectral variants of the auxiliary kernel matrix and the missing entries are estimated by fitting this model to the existing entries for model fitting we adopt the em algorithm distinguished from the em algorithm of dempster et al 1977 based on the information geometry of positive definite matrices we will report promising results on bacteria clustering experiments using two marker sequences 16s and gyrb

multiview hessian regularization for image annotation the rapid development of computer hardware and internet technology makes large scale data dependent models computationally tractable and opens a bright avenue for annotating images through innovative machine learning algorithms semisupervised learning ssl therefore received intensive attention in recent years and was successfully deployed in image annotation one representative work in ssl is laplacian regularization lr which smoothes the conditional distribution for classification along the manifold encoded in the graph laplacian however it is observed that lr biases the classification function toward a constant function that possibly results in poor generalization in addition lr is developed to handle uniformly distributed data or single view data although instances or objects such as images and videos are usually represented by multiview features such as color shape and texture in this paper we present multiview hessian regularization mhr to address the above two problems in lr based image annotation in particular mhr optimally combines multiple hr each of which is obtained from a particular view of instances and steers the classification function that varies linearly along the data manifold we apply mhr to kernel least squares and support vector machines as two examples for image annotation extensive experiments on the pascal voc 07 dataset validate the effectiveness of mhr by comparing it with baseline algorithms including lr and hr

a maximum likelihood approach to density estimation with semidefinite programming density estimation plays an important and fundamental role in pattern recognition machine learning and statistics in this article we develop a parametric approach to univariate or low dimensional density estimation based on semidefinite programming sdp our density model is expressed as the product of a nonnegative polynomial and a base density such as normal distribution exponential distribution and uniform distribution when the base density is specified the maximum likelihood estimation of the polynomial is formulated as a variant of sdp that is solved in polynomial timewith the interior point methods since the base density typically contains just one or two parameters computation of the maximum likelihood estimate reduces to a one or two dimensional easy optimization problem with this use of sdp thus the rigorous maximum likelihood estimate can be computed in our approach furthermore such conditions as symmetry and unimodality of the density function can be easily handled within this framework aic is used to choose the best model through applications to several instances we demonstrate flexibility of the model and performance of the proposed procedure combination with amixture approach is also presented the proposed approach has possible other applications beyond density estimation this point is clarified through an application to the maximum likelihood estimation of the intensity function of a nonstationary poisson process

contextual object localization with multiple kernel nearest neighbor recently many object localization models have shown that incorporating contextual cues can greatly improve accuracy over using appearance features alone therefore many of these models have explored different types of contextual sources but only considering one level of contextual interaction at the time thus what context could truly contribute to object localization through integrating cues from all levels simultaneously remains an open question moreover the relative importance of the different contextual levels and appearance features across different object classes remains to be explored here we introduce a novel framework for multiple class object localization that incorporates different levels of contextual interactions we study contextual interactions at the pixel region and object level based upon three different sources of context semantic boundary support and contextual neighborhoods our framework learns a single similarity metric from multiple kernels combining pixel and region interactions with appearance features and then applies a conditional random field to incorporate object level interactions to effectively integrate different types of feature descriptions we extend the large margin nearest neighbor to a novel algorithm that supports multiple kernels we perform experiments on three challenging image databases graz 02 msrc and pascal voc 2007 experimental results show that our model outperforms current state of the art contextual frameworks and reveals individual contributions for each contextual interaction level as well as appearance features indicating their relative importance for object localization

feature combination and the knn framework in object classification in object classification feature combination can usually be used to combine the strength of multiple complementary features and produce better classification results than any single one while multiple kernel learning mkl is a popular approach to feature combination in object classification it does not always perform well in practical applications on one hand the optimization process in mkl usually involves a huge consumption of computation and memory space on the other hand in some cases mkl is found to perform no better than the baseline combination methods this observation motivates us to investigate the underlying mechanism of feature combination with average combination and weighted average combination as a result we empirically find that in average combination it is better to use a sample of the most powerful features instead of all whereas in one type of weighted average combination the best classification accuracy comes from a nearly sparse combination we integrate these observations into the k nearest neighbors knns framework based on which we further discuss some issues related to sparse solution and mkl finally by making use of the knn framework we present a new weighted average combination method which is shown to perform better than mkl in both accuracy and efficiency in experiments we believe that the work in this paper is helpful in exploring the mechanism underlying feature combination

large margin classification in infinite neural networks we introduce a new family of positive definite kernels for large margin classification in support vector machines svms these kernels mimic the computation in large neural networks with one layer of hidden units we also show how to derive new kernels by recursive composition that may be viewed as mapping their inputs through a series of nonlinear feature spaces these recursively derived kernels mimic the computation in deep networks with multiple hidden layers we evaluate svms with these kernels on problems designed to illustrate the advantages of deep architectures compared to previous benchmarks we find that on some problems these svms yield state of the art results beating not only other svms but also deep belief nets

multiple spectral kernel learning and a gaussian complexity computation multiple kernel learning mkl partially solves the kernel selection problem in support vector machines and similar classifiers by minimizing the empirical risk over a subset of the linear combination of given kernel matrices for large sample sets the size of the kernel matrices becomes a numerical issue in many cases the kernel matrix is of low efficient rank however the low rank property is not efficiently utilized in mkl algorithms here we suggest multiple spectral kernel learning that efficiently uses the low rank property by finding a kernel matrix from a set of gram matrices of a few eigenvectors from all given kernel matrices called a spectral kernel set we provide a new bound for the gaussian complexity of the proposed kernel set which depends on both the geometry of the kernel set and the number of gram matrices this characterization of the complexity implies that in an mkl setting adding more kernels may not monotonically increase the complexity while previous bounds show otherwise

a new approach to collaborative filtering operator estimation with spectral regularization we present a general approach for collaborative filtering cf using spectral regularization to learn linear operators mapping a set of users to a set of possibly desired objects in particular several recent low rank type matrix completion methods for cf are shown to be special cases of our proposed framework unlike existing regularization based cf our approach can be used to incorporate additional information such as attributes of the users objects a feature currently lacking in existing regularization based cf approaches using popular and well known kernel methods we provide novel representer theorems that we use to develop new estimation methods we then provide learning algorithms based on low rank decompositions and test them on a standard cf data set the experiments indicate the advantages of generalizing the existing regularization based cf methods to incorporate related information about users and objects finally we show that certain multi task learning methods can be also seen as special cases of our proposed approach

scalable large margin mahalanobis distance metric learning for many machine learning algorithms such as k nearest neighbor k nn classifiers and k means clustering often their success heavily depends on the metric used to calculate distances between different data points an effective solution for defining such a metric is to learn it from a set of labeled training samples in this work we propose a fast and scalable algorithm to learn a mahalanobis distance metric the mahalanobis metric can be viewed as the euclidean distance metric on the input data that have been linearly transformed by employing the principle of margin maximization to achieve better generalization performances this algorithm formulates the metric learning as a convex optimization problem and a positive semidefinite p s d matrix is the unknown variable based on an important theorem that a p s d trace one matrix can always be represented as a convex combination of multiple rank one matrices our algorithm accommodates any differentiable loss function and solves the resulting optimization problem using a specialized gradient descent procedure during the course of optimization the proposed algorithm maintains the positive semidefiniteness of the matrix variable that is essential for a mahalanobis metric compared with conventional methods like standard interior point algorithms or the special solver used in large margin nearest neighbor our algorithm is much more efficient and has a better performance in scalability experiments on benchmark data sets suggest that compared with state of the art metric learning algorithms our algorithm can achieve a comparable classification accuracy with reduced computational complexity

on the convergence rate of l p norm multiple kernel learning we derive an upper bound on the local rademacher complexity of lp norm multiple kernel learning which yields a tighter excess risk bound than global approaches previous local approaches analyzed the case p 1 only while our analysis covers all cases 1 p assuming the different feature mappings corresponding to the different kernels to be uncorrelated we also show a lower bound that shows that the bound is tight and derive consequences regarding excess loss namely fast convergence rates of the order o n 1 a where is the minimum eigenvalue decay rate of the individual kernels

learning multi modal similarity in many applications involving multi media data the definition of similarity between items is integral to several key tasks including nearest neighbor retrieval classification and recommendation data in such regimes typically exhibits multiple modalities such as acoustic and visual content of video integrating such heterogeneous data to form a holistic similarity space is therefore a key challenge to be overcome in many real world applications we present a novel multiple kernel learning technique for integrating heterogeneous data into a single unified similarity space our algorithm learns an optimal ensemble of kernel transformations which conform to measurements of human perceptual similarity as expressed by relative comparisons to cope with the ubiquitous problems of subjectivity and inconsistency in multi media similarity we develop graph based techniques to filter similarity measurements resulting in a simplified and robust training procedure

multiple kernel learning with gaussianity measures kernel methods are known to be effective for nonlinear multivariate analysis one of the main issues in the practical use of kernel methods is the selection of kernel there have been a lot of studies on kernel selection and kernel learning multiple kernel learning mkl is one of the promising kernel optimization approaches kernel methods are applied to various classifiers including fisher discriminant analysis fda fda gives the bayes optimal classification axis if the data distribution of each class in the feature space is a gaussian with a shared covariance structure based on this fact an mkl framework based on the notion of gaussianity is proposed as a concrete implementation an empirical characteristic function is adopted to measure gaussianity in the feature space associated with a convex combination of kernel functions and two mkl algorithms are derived from experimental results on some data sets we show that the proposed kernel learning followed by fda offers strong classification power

wang landau sampling improving accuracy in this work we investigate the behavior of the microcanonical and canonical averages of the two dimensional ising model during the wang landau simulation the simulations were carried out using conventional wang landau sampling and the 1 t scheme our findings reveal that the microcanonical average should not be accumulated during the initial modification factors f and they outline a criterion to find this limit which we define as f micro we show that updating the density of states only after every l 2 spin flip trials leads to a much better precision we present a mechanism to determine for the given model up to what final modification factor f final the simulations should be carried out altogether these small adjustments lead to an improved procedure for simulations with much more reliable results we compare our results with 1 t simulations we also present an application of the procedure to a self avoiding homopolymer

a novel intelligent option price forecasting and trading system by multiple kernel adaptive filters abstract derivatives such as options are complex financial instruments the risk in option trading leads to the demand of trading support systems for investors to control and hedge their risk the nonlinearity and non stationarity of option dynamics are the main challenge of option price forecasting to address the problem this study develops a multi kernel adaptive filters mkaf for online option trading mkaf is an improved version of the adaptive filter which employs multiple kernels to enhance the richness of nonlinear feature representation the mkaf is a fully adaptive online algorithm the strength of mkaf is that the weights to the kernels are simultaneous optimally determined in filter coefficient updates we do not need to design the weights separately therefore mkaf is good at tracking nonstationary nonlinear option dynamics moreover to reduce the computation time in updating the filter and prevent overadaptation the number of kernels is restricted by using coherence based sparsification which constructs a set of dictionary and uses a coherence threshold to restrict the dictionary size this study compared the new method with traditional ones we found the performance improvement is significant and robust especially the cumulated trading profits are substantially increased

multik mhks a novel multiple kernel learning algorithm in this paper we develop a new effective multiple kernel learning algorithm first we map the input data into m different feature spaces by m empirical kernels where each generated feature space is taken as one view of the input space then through borrowing the motivating argument from canonical correlation analysis cca that can maximally correlate the m views in the transformed coordinates we introduce a special term called inter function similarity loss rifsi into the existing regularization framework so as to guarantee the agreement of multiview outputs in implementation we select the modification of ho kashyap algorithm with squared approximation of the misclassification errors mhks as the incorporated paradigm and the experimental results on benchmark data sets demonstrate the feasibility and effectiveness of the proposed algorithm named multik mhks

sublinear time algorithms for approximate semidefinite programming we consider semidefinite optimization in a saddle point formulation where the primal solution is in the spectrahedron and the dual solution is a distribution over affine functions we present an approximation algorithm for this problem that runs in sublinear time in the size of the data to the best of our knowledge this is the first algorithm to achieve this our algorithm is also guaranteed to produce low rank solutions we further prove lower bounds on the running time of any algorithm for this problem showing that certain terms in the running time of our algorithm cannot be further improved finally we consider a non affine version of the saddle point problem and give an algorithm that under certain assumptions runs in sublinear time

pac bayesian generalisation error bounds for gaussian process classification approximate bayesian gaussian process gp classification techniques are powerful non parametric learning methods similar in appearance and performance to support vector machines based on simple probabilistic models they render interpretable results and can be embedded in bayesian frameworks for model selection feature selection etc in this paper by applying the pac bayesian theorem of mcallester 1999a we prove distribution free generalisation error bounds for a wide range of approximate bayesian gp classification techniques we also provide a new and much simplified proof for this powerful theorem making use of the concept of convex duality which is a backbone of many machine learning techniques we instantiate and test our bounds for two particular gpc techniques including a recent sparse method which circumvents the unfavourable scaling of standard gp algorithms as is shown in experiments on a real world task the bounds can be very tight for moderate training sample sizes to the best of our knowledge these results provide the tightest known distribution free error bounds for approximate bayesian gpc methods giving a strong learning theoretical justification for the use of these techniques

discriminative semi supervised feature selection via manifold regularization feature selection has attracted a huge amount of interest in both research and application communities of data mining we consider the problem of semi supervised feature selection where we are given a small amount of labeled examples and a large amount of unlabeled examples since a small number of labeled samples are usually insufficient for identifying the relevant features the critical problem arising from semi supervised feature selection is how to take advantage of the information underneath the unlabeled data to address this problem we propose a novel discriminative semi supervised feature selection method based on the idea of manifold regularization the proposed approach selects features through maximizing the classification margin between different classes and simultaneously exploiting the geometry of the probability distribution that generates both labeled and unlabeled data in comparison with previous semi supervised feature selection algorithms our proposed semi supervised feature selection method is an embedded feature selection method and is able to find more discriminative features we formulate the proposed feature selection method into a convex concave optimization problem where the saddle point corresponds to the optimal solution to find the optimal solution the level method a fairly recent optimization method is employed we also present a theoretic proof of the convergence rate for the application of the level method to our problem empirical evaluation on several benchmark data sets demonstrates the effectiveness of the proposed semi supervised feature selection method

regularization techniques for learning with matrices there is growing body of learning problems for which it is natural to organize the parameters into a matrix as a result it becomes easy to impose sophisticated prior knowledge by appropriately regularizing the parameters under some matrix norm this work describes and analyzes a systematic method for constructing such matrix based regularization techniques in particular we focus on how the underlying statistical properties of a given problem can help us decide which regularization function is appropriate our methodology is based on a known duality phenomenon a function is strongly convex with respect to some norm if and only if its conjugate function is strongly smooth with respect to the dual norm this result has already been found to be a key component in deriving and analyzing several learning algorithms we demonstrate the potential of this framework by deriving novel generalization and regret bounds for multi task learning multi class learning and multiple kernel learning

gaussian kernel optimization for pattern classification this paper presents a novel algorithm to optimize the gaussian kernel for pattern classification tasks where it is desirable to have well separated samples in the kernel feature space we propose to optimize the gaussian kernel parameters by maximizing a classical class separability criterion and the problem is solved through a quasi newton algorithm by making use of a recently proposed decomposition of the objective criterion the proposed method is evaluated on five data sets with two kernel based learning algorithms the experimental results indicate that it achieves the best overall classification performance compared with three competing solutions in particular the proposed method provides a valuable kernel optimization solution in the severe small sample size scenario

multiset canonical correlations using globality preserving projections with applications to feature extraction and recognition multiset features extracted from the same patterns always represent different characteristics of data thus it is very valuable to perform the extraction on multiple feature sets this paper addresses the issue of multiset correlation feature extraction mcfe in multiple feature representations a novel method is proposed to carry out the mcfe for classification called multiset canonical correlations using globality preserving projections mcc gps which can perform joint dimensionality reduction for high dimensional data mcc gp integrates correlational characteristics of feature pairs and global geometric information of data in the transformed low dimensional space this makes mcc gps have better discriminant ability than a previous method proposed by the authors called multiset integrated canonical correlation analysis micca which only considers correlations for recognition tasks furthermore mcc gp can subsume two popular feature extraction methods into its framework under some constraints this also provides a new insight for these two methods the proposed method is applied to pattern recognition and examined using the coil 100 and eth 80 object databases and ar cmu pie and yale face databases extensive experimental results show that mcc gp outperforms micca and multiset canonical correlation analysis in terms of classification accuracy and efficiency

learning the kernel matrix by maximizing a kfd based class separability criterion the advantage of a kernel method often depends critically on a proper choice of the kernel function a promising approach is to learn the kernel from data automatically in this paper we propose a novel method for learning the kernel matrix based on maximizing a class separability criterion that is similar to those used by linear discriminant analysis lda and kernel fisher discriminant kfd it is interesting to note that optimizing this criterion function does not require inverting the possibly singular within class scatter matrix which is a computational problem encountered by many lda and kfd methods we have conducted experiments on both synthetic data and real world data from uci and feret showing that our method consistently outperforms some previous kernel learning methods

mixing linear svms for nonlinear classification in this paper we address the problem of combining linear support vector machines svms for classification of large scale nonlinear datasets the motivation is to exploit both the efficiency of linear svms lsvms in learning and prediction and the power of nonlinear svms in classification to this end we develop a lsvm mixture model that exploits a divide and conquer strategy by partitioning the feature space into subregions of linearly separable datapoints and learning a lsvm for each of these regions we do this implicitly by deriving a generative model over the joint data and label distributions consequently we can impose priors on the mixing coefficients and do implicit model selection in a top down manner during the parameter estimation process this guarantees the sparsity of the learned model experimental results show that the proposed method can achieve the efficiency of lsvms in the prediction phase while still providing a classification performance comparable to nonlinear svms

generalized hierarchical kernel learning this paper generalizes the framework of hierarchical kernel learning hkl and illustrates its utility in the domain of rule learning hkl involves multiple kernel learning over a set of given base kernels assumed to be embedded on a directed acyclic graph this paper proposes a two fold generalization of hkl the first is employing a generic l1 lp block norm regularizer 1 2 that alleviates a key limitation of the hkl formulation the second is a generalization to the case of multi class multi label and more generally multi task applications the main technical contribution of this work is the derivation of a highly specialized partial dual of the proposed generalized hkl formulation and an efficient mirror descent based active set algorithm for solving it importantly the generic regularizer enables the proposed formulation to be employed in the rule ensemble learning rel where the goal is to construct an ensemble of conjunctive propositional rules experiments on benchmark rel data sets illustrate the efficacy of the proposed generalizations

building sparse multiple kernel svm classifiers the support vector machines svms have been very successful in many machine learning problems however they can be slow during testing because of the possibly large number of support vectors obtained recently wu 2005 proposed a sparse formulation that restricts the svm to use a small number of expansion vectors in this paper we further extend this idea by integrating with techniques from multiple kernel learning mkl the kernel function in this sparse svm formulation no longer needs to be fixed but can be automatically learned as a linear combination of kernels two formulations of such sparse multiple kernel classifiers are proposed the first one is based on a convex combination of the given base kernels while the second one uses a convex combination of the so called ldquoequivalentrdquo kernels empirically the second formulation is particularly competitive experiments on a large number of toy and real world data sets show that the resultant classifier is compact and accurate and can also be easily trained by simply alternating linear program and standard svm solver

ensemble estimators for multivariate entropy estimation the problem of estimation of density functionals like entropy and mutual information has received much attention in the statistics and information theory communities a large class of estimators of functionals of the probability density suffer from the curse of dimensionality wherein the mean squared error decays increasingly slowly as a function of the sample size t as the dimension d of the samples increases in particular the rate is often glacially slow of order o t d where 0 is a rate parameter examples of such estimators include kernel density estimators k nearest neighbor k nn density estimators k nn entropy estimators intrinsic dimension estimators and other examples in this paper we propose a weighted affine combination of an ensemble of such estimators where optimal weights can be chosen such that the weighted estimator converges at a much faster dimension invariant rate of o t1 furthermore we show that these optimal weights can be determined by solving a convex optimization problem which can be performed offline and does not require training data we illustrate the superior performance of our weighted estimator for two important applications 1 estimating the panter dite distortion rate factor and 2 estimating the shannon entropy for testing the probability distribution of a random sample

empirical margin distributions and bounding the generalization error of combined classifiers we prove new probabilistic upper bounds on generalization error of complex classifiers that are combinations of simple classifiers such combinations could be implemented by neural networks or by voting methods of combining the classifiers such as boosting and bagging the bounds are in terms of the empirical distribution of the margin of the combined classifier they are based on the methods of the theory of gaussian and empirical processes comparison inequalities symmetrization method concentration inequalities and they improve previous results of bartlett 1998 on bounding the generalization error of neural networks in terms of ell 1 norms of the weights of neurons and of schapire freund bartlett and lee 1998 on bounding the generalization error of boosting we also obtain rates of convergence in levy distance of empirical margin distribution to the true margin distribution uniformly over the classes of classifiers and prove the optimality of these rates

fast learning rate of multiple kernel learning trade off between sparsity and smoothness we investigate the learning rate of multiple kernel leaning mkl with l1 and elastic net regularizations the elastic net regularization is a composition of an l1 regularizer for inducing the sparsity and an l2 regularizer for controlling the smoothness we focus on a sparse setting where the total number of kernels is large but the number of non zero components of the ground truth is relatively small and show sharper convergence rates than the learning rates ever shown for both l1 and elastic net regularizations our analysis shows there appears a trade off between the sparsity and the smoothness when it comes to selecting which of l1 and elastic net regularizations to use if the ground truth is smooth the elastic net regularization is preferred otherwise the l1 regularization is preferred

localized multiple kernel learning with dynamical clustering and matrix regularization localized multiple kernel learning lmkl is an attractive strategy for combining multiple heterogeneous features with regard to their discriminative power for each individual sample however the learning of numerous local solutions may not scale well even for a moderately sized training set and the independently learned local models may suffer from overfitting hence in existing local methods the distributed samples are typically assumed to share the same weights and various unsupervised clustering methods are applied as preprocessing in this paper to enable the learner to discover and benefit from the underlying local coherence and diversity of the samples we incorporate the clustering procedure into the canonical support vector machine based lmkl framework then to explore the relatedness among different samples which has been ignored in a vector ell p norm analysis we organize the cluster specific kernel weights into a matrix and introduce a matrix based extension of the ell p norm for constraint enforcement by casting the joint optimization problem as a problem of alternating optimization we show how the cluster structure is gradually revealed and how the matrix regularized kernel weights are obtained a theoretical analysis of such a regularizer is performed using a rademacher complexity bound and complementary empirical experiments on real world data sets demonstrate the effectiveness of our technique

maximum margin clustering made practical motivated by the success of large margin methods in supervised learning maximum margin clustering mmc is a recent approach that aims at extending large margin methods to unsupervised learning however its optimization problem is nonconvex and existing mmc methods all rely on reformulating and relaxing the nonconvex optimization problem as semidefinite programs sdp though sdp is convex and standard solvers are available they are computationally very expensive and only small data sets can be handled to make mmc more practical we avoid sdp relaxations and propose in this paper an efficient approach that performs alternating optimization directly on the original nonconvex problem a key step to avoid premature convergence in the resultant iterative procedure is to change the loss function from the hinge loss to the laplacian square loss so that overconfident predictions are penalized experiments on a number of synthetic and real world data sets demonstrate that the proposed approach is more accurate much faster hundreds to tens of thousands of times faster and can handle data sets that are hundreds of times larger than the largest data set reported in the mmc literature

a family of simple non parametric kernel learning algorithms previous studies of non parametric kernel learning npkl usually formulate the learning task as a semi definite programming sdp problem that is often solved by some general purpose sdp solvers however for n data examples the time complexity of npkl using a standard interior point sdp solver could be as high as o n6 5 which prohibits npkl methods applicable to real applications even for data sets of moderate size in this paper we present a family of efficient npkl algorithms termed simplenpkl which can learn non parametric kernels from a large set of pairwise constraints efficiently in particular we propose two efficient simplenpkl algorithms one is simplenpkl algorithm with linear loss which enjoys a closed form solution that can be efficiently computed by the lanczos sparse eigen decomposition technique another one is simplenpkl algorithm with other loss functions including square hinge loss hinge loss square loss that can be re formulated as a saddle point optimization problem which can be further resolved by a fast iterative algorithm in contrast to the previous npkl approaches our empirical results show that the proposed new technique maintaining the same accuracy is significantly more efficient and scalable finally we also demonstrate that the proposed new technique is also applicable to speed up many kernel learning tasks including colored maximum variance unfolding minimum volume embedding and structure preserving embedding

symmetric rbf classifier for nonlinear detection in multiple antenna aided systems in this paper we propose a powerful symmetric radial basis function rbf classifier for nonlinear detection in the so called overloaded multiple antenna aided communication systems by exploiting the inherent symmetry property of the optimal bayesian detector the proposed symmetric rbf classifier is capable of approaching the optimal classification performance using noisy training data the classifier construction process is robust to the choice of the rbf width and is computationally efficient the proposed solution is capable of providing a signal to noise ratio snr gain in excess of 8 db against the powerful linear minimum bit error rate ber benchmark when supporting four users with the aid of two receive antennas or seven users with four receive antenna elements

lovasz ϑ function svms and finding dense subgraphs in this paper we establish that the lovasz function on a graph can be restated as a kernel learning problem we introduce the notion of svm graphs on which lovasz function can be approximated well by a support vector machine svm we show that erdos renyi random g n p graphs are svm graphs for log4 n n p 1 even if we embed a large clique of size np 1 p in a g n p graph the resultant graph still remains a svm graph this immediately suggests an svm based algorithm for recovering a large planted clique in random graphs associated with the function is the notion of orthogonal labellings we introduce common orthogonal labellings which extends the idea of orthogonal labellings to multiple graphs this allows us to propose a multiple kernel learning mkl based solution which is capable of identifying a large common dense subgraph in multiple graphs both in the planted clique case and common subgraph detection problem the proposed solutions beat the state of the art by an order of magnitude

low rank kernel learning with bregman matrix divergences in this paper we study low rank matrix nearness problems with a focus on learning low rank positive semidefinite kernel matrices for machine learning applications we propose efficient algorithms that scale linearly in the number of data points and quadratically in the rank of the input matrix existing algorithms for learning kernel matrices often scale poorly with running times that are cubic in the number of data points we employ bregman matrix divergences as the measures of nearness these divergences are natural for learning low rank kernels since they preserve rank as well as positive semidefiniteness special cases of our framework yield faster algorithms for various existing learning problems and experimental results demonstrate that our algorithms can effectively learn both low rank and full rank kernel matrices

low rank decomposition meets kernel learning a generalized nystrom method abstract low rank matrix decomposition and kernel learning are two useful techniques in building advanced learning systems low rank decomposition can greatly reduce the computational cost of manipulating large kernel matrices however existing approaches are mostly unsupervised and do not incorporate side information such as class labels making the decomposition less effective for a specific learning task on the other hand kernel learning techniques aim at constructing kernel matrices whose structure is well aligned with the learning target which improves the generalization performance of kernel methods however most kernel learning approaches are computationally very expensive to obtain the advantages of both techniques and address their limitations in this paper we propose a novel kernel low rank decomposition formulation called the generalized nystrom method our approach inherits the linear time and space complexity via matrix decomposition while at the same time fully exploits partial label information in computing task dependent decomposition in addition the resultant low rank factors can generalize to arbitrary new samples rendering great flexibility in inductive learning scenarios we further extend the algorithm to a multiple kernel learning setup the experimental results on semi supervised classification demonstrate the usefulness of the proposed method

automatic model selection for the optimization of svm kernels this approach aims to optimize the kernel parameters and to efficiently reduce the number of support vectors so that the generalization error can be reduced drastically the proposed methodology suggests the use of a new model selection criterion based on the estimation of the probability of error of the svm classifier for comparison we considered two more model selection criteria gacv generalized approximate cross validation and vc vapnik chernovenkis dimension these criteria are algebraic estimates of upper bounds of the expected error for the former we also propose a new minimization scheme the experiments conducted on a bi class problem show that we can adequately choose the svm hyper parameters using the empirical error criterion moreover it turns out that the criterion produces a less complex model with fewer support vectors for multi class data the optimization strategy is adapted to the one against one data partitioning the approach is then evaluated on images of handwritten digits from the usps database

a kernel optimization method based on the localized kernel fisher criterion it is widely recognized that whether the selected kernel matches the data determines the performance of kernel based methods ideally it is expected that the data is linearly separable in the kernel induced feature space therefore fisher linear discriminant criterion can be used as a cost function to optimize the kernel function however the data may not be linearly separable even after kernel transformation in many applications e g the data may exist as multimodally distributed structure in this case a nonlinear classifier is preferred and obviously fisher criterion is not a suitable choice as kernel optimization rule motivated by this issue we propose a localized kernel fisher criterion instead of traditional fisher criterion as the kernel optimization rule to increase the local margins between embedded classes in kernel induced feature space experimental results based on some benchmark data and measured radar high resolution range profile hrrp data show that the classification performance can be improved by using the proposed method

feature selection and kernel learning for local learning based clustering the performance of the most clustering algorithms highly relies on the representation of data in the input space or the hilbert space of kernel methods this paper is to obtain an appropriate data representation through feature selection or kernel learning within the framework of the local learning based clustering llc wu and scholkopf 2006 method which can outperform the global learning based ones when dealing with the high dimensional data lying on manifold specifically we associate a weight to each feature or kernel and incorporate it into the built in regularization of the llc algorithm to take into account the relevance of each feature or kernel for the clustering accordingly the weights are estimated iteratively in the clustering process we show that the resulting weighted regularization with an additional constraint on the weights is equivalent to a known sparse promoting penalty hence the weights of those irrelevant features or kernels can be shrunk toward zero extensive experiments show the efficacy of the proposed methods on the benchmark data sets

selective multiple kernel learning for classification with ensemble strategy abstract multiple kernel learning mkl aims to seek a better result than single kernel learning by combining a compact set of sub kernels however mkl with l1 norm easily discards the sub kernels with complementary information and mkl with lp norm p 2 often gets the redundant solution to address these problems a selective multiple kernel learning smkl method inspired by ensemble learning el is proposed comparing mkl with lp norm p 2 smkl obtains a sparse solution by a pre selection procedure comparing mkl with l1 norm smkl preserves the sub kernels with complementary information by guaranteeing the high discrimination and large diversity of pre selected sub kernels for quantifying the discrimination and diversity of sub kernels a new kernel evaluation is designed smkl reduces the scale of mkl optimization and saves the memory storing of the sub kernels which extends the scale of problem that mkl could solve specially a fast smkl method using l norm constraint is focused which needs no mkl optimization process it means that the memory is hardly a limitation for mkl with the large scale problem experiments state that our method is effective for classification

large margin multi viewinformation bottleneck in this paper we extend the theory of the information bottleneck ib to learning from examples represented by multi view features we formulate the problem as one of encoding a communication system with multiple senders each of which represents one view of the data based on the precise components filtered out from multiple information sources through a bottleneck a margin maximization approach is then used to strengthen the discrimination of the encoder by improving the code distance within the frame of coding theory the resulting algorithm therefore inherits all the merits of the ib principle and coding theory it has two distinct advantages over existing algorithms namely that our method finds a tradeoff between the accuracy and complexity of the multi view model and that the encoded multi view data retains sufficient discrimination for classification we also derive the robustness and generalization error bound of the proposed algorithm and reveal the specific properties of multi view learning first the complementarity of multi view features guarantees the robustness of the algorithm second the consensus of multi view features reduces the empirical rademacher complexity of the objective function enhances the accuracy of the solution and improves the generalization error bound of the algorithm the resulting objective function is solved efficiently using the alternating direction method experimental results on annotation classification and recognition tasks demonstrate that the proposed algorithm is promising for practical applications

learning the kernel function via regularization we study the problem of finding an optimal kernel from a prescribed convex set of kernels k for learning a real valued function by regularization we establish for a wide variety of regularization functionals that this leads to a convex optimization problem and for square loss regularization we characterize the solution of this problem we show that although k may be an uncountable set the optimal kernel is always obtained as a convex combination of at most m 2 basic kernels where m is the number of data examples in particular our results apply to learning the optimal radial kernel or the optimal dot product kernel

pareto path multitask multiple kernel learning a traditional and intuitively appealing multitask multiple kernel learning mt mkl method is to optimize the sum thus the average of objective functions with partially shared kernel function which allows information sharing among the tasks we point out that the obtained solution corresponds to a single point on the pareto front pf of a multiobjective optimization problem which considers the concurrent optimization of all task objectives involved in the multitask learning mtl problem motivated by this last observation and arguing that the former approach is heuristic we propose a novel support vector machine mt mkl framework that considers an implicitly defined set of conic combinations of task objectives we show that solving our framework produces solutions along a path on the aforementioned pf and that it subsumes the optimization of the average of objective functions as a special case using the algorithms we derived we demonstrate through a series of experimental results that the framework is capable of achieving a better classification performance when compared with other similar mtl approaches

least square regularized regression in sum space this paper proposes a least square regularized regression algorithm in sum space of reproducing kernel hilbert spaces rkhss for nonflat function approximation and obtains the solution of the algorithm by solving a system of linear equations this algorithm can approximate the low and high frequency component of the target function with large and small scale kernels respectively the convergence and learning rate are analyzed we measure the complexity of the sum space by its covering number and demonstrate that the covering number can be bounded by the product of the covering numbers of basic rkhss for sum space of rkhss with gaussian kernels by choosing appropriate parameters we tradeoff the sample error and regularization error and obtain a polynomial learning rate which is better than that in any single rkhs the utility of this method is illustrated with two simulated data sets and five real life databases

multi kernel learning with online batch optimization in recent years there has been a lot of interest in designing principled classification algorithms over multiple cues based on the intuitive notion that using more features should lead to better performance in the domain of kernel methods a principled way to use multiple features is the multi kernel learning mkl approach here we present a mkl optimization algorithm based on stochastic gradient descent that has a guaranteed convergence rate we directly solve the mkl problem in the primal formulation by having a p norm formulation of mkl we introduce a parameter that controls the level of sparsity of the solution while leading to an easier optimization problem we prove theoretically and experimentally that 1 our algorithm has a faster convergence rate as the number of kernels grows 2 the training complexity is linear in the number of training examples 3 very few iterations are sufficient to reach good solutions experiments on standard benchmark databases support our claims

kernel based reconstruction of graph signals a number of applications in engineering social sciences physics and biology involve inference over networks in this context graph signals are widely encountered as descriptors of vertex attributes or features in graph structured data estimating such signals in all vertices given noisy observations of their values on a subset of vertices has been extensively analyzed in the literature of signal processing on graphs spog this paper advocates kernel regression as a framework generalizing popular spog modeling and reconstruction and expanding their capabilities formulating signal reconstruction as a regression task on reproducing kernel hilbert spaces of graph signals permeates benefits from statistical learning offers fresh insights and allows for estimators that leverage richer forms of prior information than existing alternatives a number of spog notions such as bandlimitedness graph filters and the graph fourier transform are naturally accommodated in the kernel framework additionally this paper capitalizes on the so called representer theorem to devise simpler versions of existing tikhonov regularized estimators and offers a novel probabilistic interpretation of kernel methods on graphs based on graphical models motivated by the challenges of selecting the bandwidth parameter in spog estimators or the kernel map in kernel based methods this paper further proposes two multikernel approaches with complementary strengths whereas the first enables estimation of the unknown bandwidth of bandlimited signals the second allows for efficient graph filter selection numerical tests with synthetic as well as real data demonstrate the merits of the proposed methods relative to state of the art alternatives

learning translation invariant kernels for classification appropriate selection of the kernel function which implicitly defines the feature space of an algorithm has a crucial role in the success of kernel methods in this paper we consider the problem of optimizing a kernel function over the class of translation invariant kernels for the task of binary classification the learning capacity of this class is invariant with respect to rotation and scaling of the features and it encompasses the set of radial kernels we show that how translation invariant kernel functions can be embedded in a nested set of sub classes and consider the kernel learning problem over one of these sub classes this allows the choice of an appropriate sub class based on the problem at hand we use the criterion proposed by lanckriet et al 2004 to obtain a functional formulation for the problem it will be proven that the optimal kernel is a finite mixture of cosine functions the kernel learning problem is then formulated as a semi infinite programming sip problem which is solved by a sequence of quadratically constrained quadratic programming qcqp sub problems using the fact that the cosine kernel is of rank two we propose a formulation of a qcqp sub problem which does not require the kernel matrices to be loaded into memory making the method applicable to large scale problems we also address the issue of including other classes of kernels such as individual kernels and isotropic gaussian kernels in the learning process another interesting feature of the proposed method is that the optimal classifier has an expansion in terms of the number of cosine kernels instead of support vectors leading to a remarkable speedup at run time as a by product we also generalize the kernel trick to complex valued kernel functions our experiments on artificial and real world benchmark data sets including the usps and the mnist digit recognition data sets show the usefulness of the proposed method

online primal dual learning for a data dependent multi kernel combination model with multiclass visual categorization applications the task of visual categorization becomes very challenging as the number of samples or classes increases this is mainly due to large memory requirements or a crowded semantic space which causes class separation difficulties in this paper a new approach called oprdum2 online primal dual multiple kernel multiple class is proposed this new algorithm can significantly reduce the complexity of visual categorization under this approach an elegant heterogeneous feature fusion machine is used to consolidate complementary class discriminative information and online learning is applied to deal with large sample sizes to be specific a data dependent multi kernel machine that fuses multiple heterogeneous features in a nonlinear manner is defined by extending the multi kernel learning model and multivariate hinge loss and conservative updating rules are used to increase the sample level sparsity in the model given the intrinsic hierarchical structure of samples features and classes an innovative triple mixed norm regularizer with strong convexity is utilized to facilitate optimization an iterative solution is derived using the language of duality leading to a low regret bound of o t where t is the number of online iterations this paper details a proof of the algorithm s convergence and describes extensive visual data classification experiments that demonstrate the effectiveness and robustness of this new approach

learning the kernel matrix with semidefinite programming kernel based learning algorithms work by embedding the data into a euclidean space and then searching for linear relations among the embedded data points the embedding is performed implicitly by specifying the inner products between each pair of points in the embedding space this information is contained in the so called kernel matrix a symmetric and positive semidefinite matrix that encodes the relative positions of all points specifying this matrix amounts to specifying the geometry of the embedding space and inducing a notion of similarity in the input space classical model selection problems in machine learning in this paper we show how the kernel matrix can be learned from data via semidefinite programming sdp techniques when applied to a kernel matrix associated with both training and test data this gives a powerful transductive algorithm using the labeled part of the data one can learn an embedding also for the unlabeled part the similarity between test points is inferred from training points and their labels importantly these learning problems are convex so we obtain a method for learning both the model class and the function without local minima furthermore this approach leads directly to a convex method for learning the 2 norm soft margin parameter in support vector machines solving an important open problem

feature fusion using locally linear embedding for classification in most complex classification problems many types of features have been captured or extracted feature fusion is used to combine features for better classification and to reduce data dimensionality kernel based feature fusion methods are very effective for classification but they do not reduce data dimensionality in this brief we propose an effective feature fusion method using locally linear embedding lle the proposed method overcomes the limitations of lle which could not handle different types of features and is inefficient for classification we propose an efficient algorithm to solve the optimization problem in obtaining weights of different features and design an efficient method for lle based classification in comparison to other kernel based feature fusion methods the proposed method fuses features to a significantly lower dimensional feature space with the same discriminant power we have conducted experiments to demonstrate the effectiveness of the proposed feature fusion method

soft margin multiple kernel learning multiple kernel learning mkl has been proposed for kernel methods by learning the optimal kernel from a set of predefined base kernels however the traditional l 1 rm mkl method often achieves worse results than the simplest method using the average of base kernels i e average kernel in some practical applications in order to improve the effectiveness of mkl this paper presents a novel soft margin perspective for mkl specifically we introduce an additional slack variable called kernel slack variable to each quadratic constraint of mkl which corresponds to one support vector machine model using a single base kernel we first show that l 1 rm mkl can be deemed as hard margin mkl and then we propose a novel soft margin framework for mkl three commonly used loss functions including the hinge loss the square hinge loss and the square loss can be readily incorporated into this framework leading to the new soft margin mkl objective functions many existing mkl methods can be shown as special cases under our soft margin framework for example the hinge loss soft margin mkl leads to a new box constraint for kernel combination coefficients using different hyper parameter values for this formulation we can inherently bridge the method using average kernel l 1 rm mkl and the hinge loss soft margin mkl the square hinge loss soft margin mkl unifies the family of elastic net constraint regularizer based approaches and the square loss soft margin mkl incorporates l 2 rm mkl naturally moreover we also develop efficient algorithms for solving both the hinge loss and square hinge loss soft margin mkl comprehensive experimental studies for various mkl algorithms on several benchmark data sets and two real world applications including video action recognition and event recognition demonstrate that our proposed algorithms can efficiently achieve an effective yet sparse solution for mkl