boosting the margin a new explanation for the effectiveness of voting methods one of the surprising recurring phenomena observed in experiments with boosting is that the test error of the generated classifier usually does not increase as its size becomes very large and often is observed to decrease even after the training error reaches zero in this paper we show that this phenomenon is related to the distribution of margins of the training examples with respect to the generated voting classification rule where the margin of an example is simply the difference between the number of correct votes and the maximum number of votes received by any incorrect label we show that techniques used in the analysis of vapnik s support vector classifiers and of neural networks with small weights can be applied to voting methods to relate the margin distribution to the test error we also show theoretically and experimentally that boosting is especially effective at increasing the margins of the training examples finally we compare our explanation to those based on the bias variance decomposition

multiregion image segmentation by parametric kernel graph cuts the purpose of this study is to investigate multiregion graph cut image partitioning via kernel mapping of the image data the image data is transformed implicitly by a kernel function so that the piecewise constant model of the graph cut formulation becomes applicable the objective function contains an original data term to evaluate the deviation of the transformed data within each segmentation region from the piecewise constant model and a smoothness boundary preserving regularization term the method affords an effective alternative to complex modeling of the original image data while taking advantage of the computational benefits of graph cuts using a common kernel function energy minimization typically consists of iterating image partitioning by graph cut iterations and evaluations of region parameters via fixed point computation a quantitative and comparative performance assessment is carried out over a large number of experiments using synthetic grey level data as well as natural images from the berkeley database the effectiveness of the method is also demonstrated through a set of experiments with real images of a variety of types such as medical synthetic aperture radar and motion maps

posterior probability support vector machines for unbalanced data this paper proposes a complete framework of posterior probability support vector machines ppsvms for weighted training samples using modified concepts of risks linear separability margin and optimal hyperplane within this framework a new optimization problem for unbalanced classification problems is formulated and a new concept of support vectors established furthermore a soft ppsvm with an interpretable parameter spl nu is obtained which is similar to the spl nu svm developed by scho spl uml lkopf et al and an empirical method for determining the posterior probability is proposed as a new approach to determine spl nu the main advantage of an ppsvm classifier lies in that fact that it is closer to the bayes optimal without knowing the distributions to validate the proposed method two synthetic classification examples are used to illustrate the logical correctness of ppsvms and their relationship to regular svms and bayesian methods several other classification experiments are conducted to demonstrate that the performance of ppsvms is better than regular svms in some cases compared with fuzzy support vector machines fsvms the proposed ppsvm is a natural and an analytical extension of regular svms based on the statistical learning theory

on optimal neural network learning abstract we introduce optimal learning with a neural network which we define as building a network with a minimal expectation generalisation error this procedure may be analysed exactly in idealized problems by exploiting the relationship between sampling a space of hypotheses and the replica method of statistical physics we find that the optimally trained spherical perceptron may learn a linearly separable rule as well as any possible computer and present simulation results supporting our conclusions optimal learning of a well known unlearnable problem the mismatched weight problem gives better asymptotic learning than conventional techniques and may be simulated more easily unlike many other perceptron learning schemes optimal learning extends to more general networks learning more complex rules

on the convergence rate of l p norm multiple kernel learning we derive an upper bound on the local rademacher complexity of lp norm multiple kernel learning which yields a tighter excess risk bound than global approaches previous local approaches analyzed the case p 1 only while our analysis covers all cases 1 p assuming the different feature mappings corresponding to the different kernels to be uncorrelated we also show a lower bound that shows that the bound is tight and derive consequences regarding excess loss namely fast convergence rates of the order o n 1 a where is the minimum eigenvalue decay rate of the individual kernels

a general wrapper approach to selection of class dependent features in this paper we argue that for a c class classification problem c 2 class classifiers each of which discriminating one class from the other classes and having a characteristic input feature subset should in general outperform or at least match the performance of a c class classifier with one single input feature subset for each class we select a desirable feature subset which leads to the lowest classification error rate for this class using a classifier for a given feature subset search algorithm to fairly compare all models we propose a weight method for the class dependent classifier i e assigning a weight to each model s output before the comparison is carried out the method s performance is evaluated on two artificial data sets and several real world benchmark data sets with the support vector machine svm as the classifier and with the relief class separability and minimal redundancy maximal relevancy mrmr as attribute importance measures our results indicate that the class dependent feature subsets found by our approach can effectively remove irrelevant or redundant features while maintaining or improving sometimes substantially the classification accuracy in comparison with other feature selection methods

multik mhks a novel multiple kernel learning algorithm in this paper we develop a new effective multiple kernel learning algorithm first we map the input data into m different feature spaces by m empirical kernels where each generated feature space is taken as one view of the input space then through borrowing the motivating argument from canonical correlation analysis cca that can maximally correlate the m views in the transformed coordinates we introduce a special term called inter function similarity loss rifsi into the existing regularization framework so as to guarantee the agreement of multiview outputs in implementation we select the modification of ho kashyap algorithm with squared approximation of the misclassification errors mhks as the incorporated paradigm and the experimental results on benchmark data sets demonstrate the feasibility and effectiveness of the proposed algorithm named multik mhks

entropic one class classifiers the one class classification problem is a well known research endeavor in pattern recognition the problem is also known under different names such as outlier and novelty anomaly detection the core of the problem consists in modeling and recognizing patterns belonging only to a so called target class all other patterns are termed nontarget and therefore they should be recognized as such in this paper we propose a novel one class classification system that is based on an interplay of different techniques primarily we follow a dissimilarity representation based approach we embed the input data into the dissimilarity space ds by means of an appropriate parametric dissimilarity measure this step allows us to process virtually any type of data the dissimilarity vectors are then represented by weighted euclidean graphs which we use to determine the entropy of the data distribution in the ds and at the same time to derive effective decision regions that are modeled as clusters of vertices since the dissimilarity measure for the input data is parametric we optimize its parameters by means of a global optimization scheme which considers both mesoscopic and structural characteristics of the data represented through the graphs the proposed one class classifier is designed to provide both hard boolean and soft decisions about the recognition of test patterns allowing an accurate description of the classification process we evaluate the performance of the system on different benchmarking data sets containing either feature based or structured patterns experimental results demonstrate the effectiveness of the proposed technique

regularized discriminant analysis abstract linear and quadratic discriminant analysis are considered in the small sample high dimensional setting alternatives to the usual maximum likelihood plug in estimates for the covariance matrices are proposed these alternatives are characterized by two parameters the values of which are customized to individual situations by jointly minimizing a sample based estimate of future misclassification risk computationally fast implementations are presented and the efficacy of the approach is examined through simulation studies and application to data these studies indicate that in many circumstances dramatic gains in classification accuracy can be achieved

how many variables should be entered in a regression equation abstract the optimal number of regressors is determined to minimize mean squared prediction error and is shown to be a small fraction of the number of data points as the number of regressors grows large the sp criterion provides an asymptotically optimal rule for the number of variables to enter

improvements to platt s smo algorithm for svm classifier design this article points out an important source of inefficiency in platt s sequential minimal optimization smo algorithm that is caused by the use of a single threshold value using clues from the kkt conditions for the dual problem two threshold parameters are employed to derive modifications of smo these modified algorithms perform significantly faster than the original smo on all benchmark data sets tried

codimension 2 bifurcation of twisted double homoclinic loops a local active coordinates approach is employed to obtain bifurcation equations of twisted double homoclinic loops under the condition of one twisted orbit we obtain the existence and uniqueness and of the 1 1 double homoclinic loop 2 1 double homoclinic loop 2 1 right homoclinic loop 1 1 large homoclinic loop 2 1 large homoclinic loop and 2 1 large period orbit for the case of double twisted orbits we obtain the existence or non existence of 1 1 double homoclinic loop 1 2 double homoclinic loop 2 1 double homoclinic loop 2 2 double homoclinic loop 2 1 large homoclinic loop 1 2 large homoclinic loop 2 2 large homoclinic loop 2 2 right homoclinic loop 2 2 large homoclinic loop 2 2 left homoclinic loop and 2 2 large period orbit moreover the bifurcation surfaces and their existence regions are given besides bifurcation sets are presented on the 2 dimensional subspace spanned by the first two melnikov vectors

optimizing the kernel in the empirical feature space in this paper we present a method of kernel optimization by maximizing a measure of class separability in the empirical feature space an euclidean space in which the training data are embedded in such a way that the geometrical structure of the data in the feature space is preserved employing a data dependent kernel we derive an effective kernel optimization algorithm that maximizes the class separability of the data in the empirical feature space it is shown that there exists a close relationship between the class separability measure introduced here and the alignment measure defined recently by cristianini extensive simulations are carried out which show that the optimized kernel is more adaptive to the input data and leads to a substantial sometimes significant improvement in the performance of various data classification algorithms

fisher regularized support vector machine this paper proposes fisher regularized support vector machine fishersvm fishersvm is a graph based supervised learning method fishersvm has two regularization terms maximum margin and fisher regularizations fishersvm aims to maximize the margin and minimize the within class scatter support vector machine svm and fisher discriminant analysis fda are two commonly used methods in machine learning and pattern recognition a combined method of the linear svm and fda called svm lda linear discriminant analysis has been proposed only for the linear case this paper generalizes this combined method to the nonlinear case from the view of regularization a fisher regularization is defined and incorporated into svm to obtain a fisher regularized support vector machine fishersvm in fishersvm there are two regularization terms the maximum margin regularization and fisher regularization which allow fishersvm to maximize the classification margin and minimize the within class scatter roughly speaking fishersvm can approximatively fulfill the fisher criterion and obtain good statistical separability this paper also discusses the connections of fishersvm to graph based regularization methods and the mathematical programming method to kernel fisher discriminant analysis kfda it shows that fishersvm can be thought of as a graph based supervised learning method or a robust kfda experimental results on artificial and real world data show that fishersvm has a promising generalization performance

an equivalence between sparse approximation and support vector machines in the first part of this paper we show a similarity between the principle of structural risk minimization principle srm vapnik 1982 and the idea of sparse approximation as defined in chen donoho and saunders 1995 and olshausen and field 1996 then we focus on two specific approximate implementations of srm and sparse approximation which have been used to solve the problem of function approximation for srm we consider the support vector machine technique proposed by v vapnik and his team at at t bell labs and for sparse approximation we consider a modification of the basis pursuit de noising algorithm proposed by chen donoho and saunders 1995 we show that under certain conditions these two techniques are equivalent they give the same solution and they require the solution of the same quadratic programming problem

multiplicative update rules for incremental training of multiclass support vector machines we present a new method for the incremental training of multiclass support vector machines that can simultaneously modify each class separating hyperplane and provide computational efficiency for training tasks where the training data collection is sequentially enriched and dynamic adaptation of the classifier is required over time an auxiliary function has been designed that incorporates some desired characteristics in order to provide an upper bound for the objective function which summarizes the multiclass classification task a novel set of multiplicative update rules is proposed which is independent from any kind of learning rate parameter provides computational efficiency compared to the conventional batch training approach and is easy to implement convergence to the global minimum is guaranteed since the optimization problem is convex and the global minimizer for the enriched dataset is found using a warm start algorithm experimental evidence on various data collections verified that our method is faster than retraining the classifier from scratch while the achieved classification accuracy rate is maintained at the same level

a survey of kernel and spectral methods for clustering clustering algorithms are a useful tool to explore data structures and have been employed in many disciplines the focus of this paper is the partitioning clustering problem with a special interest in two recent approaches kernel and spectral methods the aim of this paper is to present a survey of kernel and spectral clustering methods two approaches able to produce nonlinear separating hypersurfaces between clusters the presented kernel clustering methods are the kernel version of many classical clustering algorithms e g k means som and neural gas spectral clustering arise from concepts in spectral graph theory and the clustering problem is configured as a graph cut problem where an appropriate objective function has to be optimized an explicit proof of the fact that these two paradigms have the same objective is reported since it has been proven that these two seemingly different approaches have the same mathematical foundation besides fuzzy kernel clustering methods are presented as extensions of kernel k means clustering algorithm

soft margin multiple kernel learning multiple kernel learning mkl has been proposed for kernel methods by learning the optimal kernel from a set of predefined base kernels however the traditional l 1 rm mkl method often achieves worse results than the simplest method using the average of base kernels i e average kernel in some practical applications in order to improve the effectiveness of mkl this paper presents a novel soft margin perspective for mkl specifically we introduce an additional slack variable called kernel slack variable to each quadratic constraint of mkl which corresponds to one support vector machine model using a single base kernel we first show that l 1 rm mkl can be deemed as hard margin mkl and then we propose a novel soft margin framework for mkl three commonly used loss functions including the hinge loss the square hinge loss and the square loss can be readily incorporated into this framework leading to the new soft margin mkl objective functions many existing mkl methods can be shown as special cases under our soft margin framework for example the hinge loss soft margin mkl leads to a new box constraint for kernel combination coefficients using different hyper parameter values for this formulation we can inherently bridge the method using average kernel l 1 rm mkl and the hinge loss soft margin mkl the square hinge loss soft margin mkl unifies the family of elastic net constraint regularizer based approaches and the square loss soft margin mkl incorporates l 2 rm mkl naturally moreover we also develop efficient algorithms for solving both the hinge loss and square hinge loss soft margin mkl comprehensive experimental studies for various mkl algorithms on several benchmark data sets and two real world applications including video action recognition and event recognition demonstrate that our proposed algorithms can efficiently achieve an effective yet sparse solution for mkl

improvements on cross validation the 632 bootstrap method abstract a training set of data has been used to construct a rule for predicting future responses what is the error rate of this rule this is an important question both for comparing models and for assessing a final selected model the traditional answer to this question is given by cross validation the cross validation estimate of prediction error is nearly unbiased but can be highly variable here we discuss bootstrap estimates of prediction error which can be thought of as smoothed versions of cross validation we show that a particular bootstrap method the 632 rule substantially outperforms cross validation in a catalog of 24 simulation experiments besides providing point estimates we also consider estimating the variability of an error rate estimate all of the results here are nonparametric and apply to any possible prediction rule however we study only classification problems with 0 1 loss in detail our simulations include smooth prediction rules like fisher s linear discriminant fun

generalization performance of regularization networks and support vector machines via entropy numbers of compact operators we derive new bounds for the generalization error of kernel machines such as support vector machines and related regularization networks by obtaining new bounds on their covering numbers the proofs make use of a viewpoint that is apparently novel in the field of statistical learning theory the hypothesis class is described in terms of a linear operator mapping from a possibly infinite dimensional unit ball in feature space into a finite dimensional space the covering numbers of the class are then determined via the entropy numbers of the operator these numbers which characterize the degree of compactness of the operator can be bounded in terms of the eigenvalues of an integral operator induced by the kernel function used by the machine as a consequence we are able to theoretically explain the effect of the choice of kernel function on the generalization performance of support vector machines

on the perturbations of a hamiltonian system the perturbations of a hamiltonian system having compounded cycle are studied in this paper the existence theory and stability theory of singular closed orbits are applied to study the given perturbed systems by using the small parametric perturbation techniques of differential equations we study hopf bifurcation singular closed orbits bifurcation and give the number and distributions of limit cycles in the above perturbed near hamiltonian system

nonlinear component analysis as a kernel eigenvalue problem a new method for performing a nonlinear form of principal component analysis is proposed by the use of integral operator kernel functions one can efficiently compute principal components in high dimensional feature spaces related to input space by some nonlinear map for instance the space of all possible five pixel products in 16 16 images we give the derivation of the method and present experimental results on polynomial feature extraction for pattern recognition

cooperative and penalized competitive learning with application to kernel based clustering abstract competitive learning approaches with individual penalization or cooperation mechanisms have the attractive ability of automatic cluster number selection in unsupervised data clustering in this paper we further study these two mechanisms and propose a novel learning algorithm called cooperative and penalized competitive learning cpcl which implements the cooperation and penalization mechanisms simultaneously in a single competitive learning process the integration of these two different kinds of competition mechanisms enables the cpcl to locate the cluster centers more quickly and be insensitive to the number of seed points and their initial positions additionally to handle nonlinearly separable clusters we further introduce the proposed competition mechanism into kernel clustering framework correspondingly a new kernel based competitive learning algorithm which can conduct nonlinear partition without knowing the true cluster number is presented the promising experimental results on real data sets demonstrate the superiority of the proposed methods

solving nonstationary classification problems with coupled support vector machines many learning problems may vary slowly over time in particular some critical real world applications when facing this problem it is desirable that the learning method could find the correct input output function and also detect the change in the concept and adapt to it we introduce the time adaptive support vector machine ta svm which is a new method for generating adaptive classifiers capable of learning concepts that change with time the basic idea of ta svm is to use a sequence of classifiers each one appropriate for a small time window but in contrast to other proposals learning all the hyperplanes in a global way we show that the addition of a new term in the cost function of the set of svms that penalizes the diversity between consecutive classifiers produces a coupling of the sequence that allows ta svm to learn as a single adaptive classifier we evaluate different aspects of the method using appropriate drifting problems in particular we analyze the regularizing effect of changing the number of classifiers in the sequence or adapting the strength of the coupling a comparison with other methods in several problems including the well known stagger dataset and the real world electricity pricing domain shows the good performance of ta svm in all tested situations

lagrangian support vector machines an implicit lagrangian for the dual of a simple reformulation of the standard quadratic program of a linear support vector machine is proposed this leads to the minimization of an unconstrained differentiable convex function in a space of dimensionality equal to the number of classified points this problem is solvable by an extremely simple linearly convergent lagrangian support vector machine lsvm algorithm lsvm requires the inversion at the outset of a single matrix of the order of the much smaller dimensionality of the original input space plus one the full algorithm is given in this paper in 11 lines of matlab code without any special optimization tools such as linear or quadratic programming solvers this lsvm code can be used as is to solve classification problems with millions of points for example 2 million points in 10 dimensional input space were classified by a linear surface in 82 minutes on a pentium iii 500 mhz notebook with 384 megabytes of memory and additional swap space and in 7 minutes on a 250 mhz ultrasparc ii processor with 2 gigabytes of memory other standard classification test problems were also solved nonlinear kernel classification can also be solved by lsvm although it does not scale up to very large problems it can handle any positive semidefinite kernel and is guaranteed to converge a short matlab code is also given for nonlinear kernels and tested on a number of problems

bounds on error expectation for support vector machines we introduce the concept of span of support vectors sv and show that the generalization ability of support vector machines svm depends on this new geometrical concept we prove that the value of the span is always smaller and can be much smaller than the diameter of the smallest sphere containing the support vectors used in previous bounds vapnik 1998 we also demonstate experimentally that the prediction of the test error given by the span is very accurate and has direct application in model selection choice of the optimal parameters of the svm

novelty detection using level set methods this paper presents a level set boundary description lsbd approach for novelty detection that treats the nonlinear boundary directly in the input space the proposed approach consists of level set function lsf construction boundary evolution and termination of the training process it employs kernel density estimation to construct the lsf of the initial boundary for the training data set then a sign of the lsf based algorithm is proposed to evolve the boundary and make it fit more tightly in the data distribution the training process terminates when an expected fraction of rejected normal data is reached the evolution process utilizes the signs of the lsf values at all training data points to decide whether to expand or shrink the boundary extensive experiments are conducted on benchmark data sets to evaluate the proposed lsbd method and compare it against four representative novelty detection methods the experimental results demonstrate that the novelty detector modeled with the proposed lsbd can effectively detect anomalies

complex support vector machines for regression and quaternary classification the paper presents a new framework for complex support vector regression svr as well as support vector machines svm for quaternary classification the method exploits the notion of widely linear estimation to model the input out relation for complex valued data and considers two cases 1 the complex data are split into their real and imaginary parts and a typical real kernel is employed to map the complex data to a complexified feature space and 2 a pure complex kernel is used to directly map the data to the induced complex feature space the recently developed wirtinger s calculus on complex reproducing kernel hilbert spaces is employed to compute the lagrangian and derive the dual optimization problem as one of our major results we prove that any complex svm svr task is equivalent with solving two real svm svr tasks exploiting a specific real kernel which is generated by the chosen complex kernel in particular the case of pure complex kernels leads to the generation of new kernels which have not been considered before in the classification case the proposed framework inherently splits the complex space into four parts this leads naturally to solving the four class task quaternary classification instead of the typical two classes of the real svm in turn this rationale can be used in a multiclass problem as a split class scenario based on four classes as opposed to the one versus all method this can lead to significant computational savings experiments demonstrate the effectiveness of the proposed framework for regression and classification tasks that involve complex data

density induced support vector data description the purpose of data description is to give a compact description of the target data that represents most of its characteristics in a support vector data description svdd the compact description of target data is given in a hyperspherical model which is determined by a small portion of data called support vectors despite the usefulness of the conventional svdd however it may not identify the optimal solution of target description especially when the support vectors do not have the overall characteristics of the target data to address the issue in svdd methodology we propose a new svdd by introducing new distance measurements based on the notion of a relative density degree for each data point in order to reflect the distribution of a given data set moreover for a real application we extend the proposed method for the protein localization prediction problem which is a multiclass and multilabel problem experiments with various real data sets show promising results

subclass graph embedding and a marginal fisher analysis paradigm subspace learning techniques have been extensively used for dimensionality reduction dr in many pattern classification problem domains recently methods like subclass discriminant analysis sda and clustering based discriminant analysis cda which use subclass information for the discrimination between the data classes have attracted much attention in parallel important work has been accomplished on graph embedding ge which is a general framework unifying several subspace learning techniques in this paper ge has been extended in order to integrate subclass discriminant information resulting to the novel subclass graph embedding sge framework which is the main contribution of our work it is proven that sge encapsulates a diversity of both supervised and unsupervised unimodal methods like locality preserving projections lpp principal component analysis pca and linear discriminant analysis lda the theoretical link of sda and cda methods with sge is also established along these lines it is shown that sge comprises a generalization of the typical ge framework including subclass dr methods moreover it allows for an easy utilization of kernels for confronting non linearly separable data employing sge in this paper a novel dr algorithm which uses subclass discriminant information called subclass marginal fisher analysis smfa has been proposed through a series of experiments on various real world datasets it is shown that smfa outperforms in most of the cases the state of the art demonstrating the efficacy and power of sge as a platform to develop new methods highlightsgraph embedding is extended in order to integrate subclass informationthe novel subclass graph embedding framework is proposed the kernelized version of the new framework is presentedsubclass graph embedding encapsulates various subspace learning methods a novel subclass marginal fisher analysis method is proposed

support vector learning for fuzzy rule based classification systems to design a fuzzy rule based classification system fuzzy classifier with good generalization ability in a high dimensional feature space has been an active research topic for a long time as a powerful machine learning approach for pattern recognition problems the support vector machine svm is known to have good generalization ability more importantly an svm can work very well on a high or even infinite dimensional feature space this paper investigates the connection between fuzzy classifiers and kernel machines establishes a link between fuzzy rules and kernels and proposes a learning algorithm for fuzzy classifiers we first show that a fuzzy classifier implicitly defines a translation invariant kernel under the assumption that all membership functions associated with the same input variable are generated from location transformation of a reference function fuzzy inference on the if part of a fuzzy rule can be viewed as evaluating the kernel function the kernel function is then proven to be a mercer kernel if the reference functions meet a certain spectral requirement the corresponding fuzzy classifier is named positive definite fuzzy classifier pdfc a pdfc can be built from the given training samples based on a support vector learning approach with the if part fuzzy rules given by the support vectors since the learning process minimizes an upper bound on the expected risk expected prediction error instead of the empirical risk training error the resulting pdfc usually has good generalization moreover because of the sparsity properties of the svms the number of fuzzy rules is irrelevant to the dimension of input space in this sense we avoid the curse of dimensionality finally pdfcs with different reference functions are constructed using the support vector learning approach the performance of the pdfcs is illustrated by extensive experimental results comparisons with other methods are also provided

a tree structured framework for purifying complex clusters with structural roles of individual data how can we find a natural clustering of a complex dataset which may contain an unknown number of overlapping clusters of arbitrary shape and be contaminated by noise a tree structured framework is proposed in this paper to purify such clusters by exploring the structural role of each data in practice each individual object within the internal organization of the data has its own specific role centroid hub or outlier due to distinctive associations with their respective neighbors adjacent centroids always interact on each other and serve as mediate nodes of one tree being members of some cluster hubs closed to some centroid become leaf nodes responsible for the termination of the growth of trees outliers that weakly touch with any centroid are often discarded from any trees as global noise all the data can thus be labeled by a specified criterion of centroids connected structural consistency ccsc free of domain specific information our framework with ccsc could widely adapt to many clustering related applications theoretical and experimental contributions both confirm that our framework is easy to interpret and implement efficient and effective in complex clustering

admissible variable selection procedures when fitting regression models by least squares for prediction given data with a single dependent variable arising from a normal linear regression model a variable selection procedure completely specifies a least squares fit by choosing the subset of the independent variables to include in the model for loss equal to squared error of prediction we prove that all variable selection procedures are admissible for choosing among least squares fits of the regression model

new support vector algorithms we propose a new class of support vector algorithms for regression and classification in these algorithms a parameter lets one effectively control the number of support vectors while this can be useful in its own right the parameterization has the additional benefit of enabling us to eliminate one of the other free parameters of the algorithm the accuracy parameter epsilon in the regression case and the regularization constant c in the classification case we describe the algorithms give some theoretical results concerning the meaning and the choice of and report experimental results

some comments on cp i study the typical configuration of a cp plot when the number of variables in the regression problem is large and there are many weak effects i show that a particular configuration that is very commonly seen can arise in a simple way i give a formula by means of which the risk incurred by the minimum cp rule can be estimated

decision theoretic measures of influence in regression the identification of influential observations is a crucial phase of fitting a model because such observations have a strong impact on inferences or decisions based on the fit an aberrant influential observation should be excluded from the data used to fit the model on the other hand if an influential observation arises from the same model underlying the rest of the data it ought to be included we address the characterization of influence measures justified by the principles of decision theory texts by belsley kuh and welsch 1980 and by cook and weisberg 1982 develop many measures of influence and survey others which have appeared in the literature most measures are motivated in a frequentist perspective but several bayesian measures have also been developed see for example pettit and smith 1985 and johnson and geisser 1982 armed with the ability to compute these measures one can generate reams of computer output when fitting a regression model to even a small data set apart from the discussion in section 4 4 of cook and weisberg 1982 comparing four types of influence diagnostics little has been said about the relative merits of different measures consequently data analysts lack practical guidance in parsimoniously choosing appropriate influence measures when fitting a regression model to data we use a bayesian decision theoretic framework to develop measures of influence in section 2 the regression fitting problem is set up as a formal decision problem complete with action space loss function and a prior distribution for the unknown parameters we then define influence measures in section 3 based on the change in bayes risk resulting from an observation s inclusion in or exclusion from the analysis to express regression fitting as a decision problem it must be simplified perhaps to an unrealistic extent still our study provides insight for defining influence measures in ideal settings an observation s influence will depend on the purpose and context of the regression fitting as characterized by the loss function and the prior distribution two examples are presented for some cases of the loss and prior these measures are equivalent to ones already proposed in the literature and in other cases new measures result

asymptotic analysis of penalized likelihood and related estimators a general approach to the first order asymptotic analysis ofpenalized likelihood and related estimators is described the method gives expansions for the systematic and random error asymptotic convergence rates in a family of spectral norms are obtained the theory applies to a broad range of function estimation prob erns including non paxametric dellsity hazard and generalized regression curve estimation some examples are provided ams 1980 subject classifications primary 62 g05 secondary 62j05 41 a35 41 a25 47 a53 45 llo 45 m05

bifurcations of limit cycles in equivariant quintic planar vector fields abstract in this paper we obtain 23 limit cycles for a z 3 equivariant near hamiltonian system of degree 5 which is the perturbation of a z 6 equivariant quintic hamiltonian system the configuration of these limit cycles is new and different from the configuration obtained by h s y chan k w chung and j li where the unperturbed system is a z 3 equivariant quintic hamiltonian system our unperturbed system is different from the unperturbed systems studied by y wu and m han the limit cycles are obtained by poincare pontryagin theorem and poincare bendixson theorem

clustering with proximity knowledge and relational knowledge in this article a proximity fuzzy framework for clustering relational data is presented where the relationships between the entities of the data are given in terms of proximity values we offer a comprehensive and in depth comparison of our clustering framework with proximity relational knowledge to clustering with distance relational knowledge such as the well known relational fuzzy c means fcm we conclude that proximity can provide a richer description of the relationships among the data and this offers a significant advantage when realizing clustering we further motivate clustering relational proximity data and provide both synthetic and real world experiments to demonstrate both the usefulness and advantage offered by clustering proximity data finally a case study of relational clustering is introduced where we apply proximity fuzzy clustering to the problem of clustering a set of trees derived from software requirements engineering the relationships between trees are based on the degree of closeness in both the location of the nodes in the trees and the semantics associated with the type of connections between the nodes

kernel matching pursuits with arbitrary loss functions the purpose of this research is to develop a classifier capable of state of the art performance in both computational efficiency and generalization ability while allowing the algorithm designer to choose arbitrary loss functions as appropriate for a give problem domain this is critical in applications involving heavily imbalanced noisy or non gaussian distributed data to achieve this goal a kernel matching pursuit kmp framework is formulated where the objective is margin maximization rather than the standard error minimization this approach enables excellent performance and computational savings in the presence of large imbalanced training data sets and facilitates the development of two general algorithms these algorithms support the use of arbitrary loss functions allowing the algorithm designer to control the degree to which outliers are penalized and the manner in which non gaussian distributed data is handled example loss functions are provided and algorithm performance is illustrated in two groups of experimental results the first group demonstrates that the proposed algorithms perform equivalent to several state of the art machine learning algorithms on well published balanced data the second group of results illustrates superior performance by the proposed algorithms on imbalanced non gaussian data achieved by employing loss functions appropriate for the data characteristics and problem domain

on utilizing search methods to select subspace dimensions for kernel based nonlinear subspace classifiers in kernel based nonlinear subspace kns methods the subspace dimensions have a strong influence on the performance of the subspace classifier in order to get a high classification accuracy a large dimension is generally required however if the chosen subspace dimension is too large it leads to a low performance due to the overlapping of the resultant subspaces and if it is too small it increases the classification error due to the poor resulting approximation the most common approach is of an ad hoc nature which selects the dimensions based on the so called cumulative proportion computed from the kernel matrix for each class we propose a new method of systematically and efficiently selecting optimal or near optimal subspace dimensions for kns classifiers using a search strategy and a heuristic function termed the overlapping criterion the rationale for this function has been motivated in the body of the paper the task of selecting optimal subspace dimensions is reduced to find the best ones from a given problem domain solution space using this criterion as a heuristic function thus the search space can be pruned to very efficiently find the best solution our experimental results demonstrate that the proposed mechanism selects the dimensions efficiently without sacrificing the classification accuracy

a new method to discriminate between enzyme kinetic models abstract a novel statistical procedure s zwanzig math opsforsch statist ser statist 11 23 47 1980 is applied to discriminate between kinetic models of enzyme reactions two examples are elucidated on the basis of numerically simulated data 1 a discrimination between a michaelis menten model and a hill equation 2 b discrimination between competitive and noncompetitive inhibition of a two substrate reaction next the proposed method is applied to assess a kinetic model of the allosteric enzyme phosphofructokinase from the malarial parasite plasmodium berghei it is demonstrated that the novel procedure allows a more sensitive discrimination as it is obtained with alternative statistical tests

two class support vector data description support vector data description svdd is a data description method that can give the target data set a spherically shaped description and be used to outlier detection or classification in real life the target data set often contains more than one class of objects and each class of objects need to be described and distinguished simultaneously in this case traditional svdd can only give a description for the target data set regardless of the differences between different target classes in the target data set or give a description for each class of objects in the target data set in this paper an improved support vector data description method named two class support vector data description tc svdd is presented the proposed method can give each class of objects in the target data set a hypersphere shaped description simultaneously if the target data set contains two classes of objects the characteristics of the improved support vector data descriptions are discussed the results of the proposed approach on artificial and actual data show that the proposed method works quite well on the 3 class classification problem with one object class being undersampled severely

effects of kernel function on nu support vector machines in extreme cases how we should choose a kernel function in support vector machines svms is an important but difficult problem in this paper we discuss the properties of the solution of the spl nu svm s a variation of svm s for normalized feature vectors in two extreme cases all feature vectors are almost orthogonal and all feature vectors are almost the same in the former case the solution of the spl nu svm is nearly the center of gravity of the examples given while the solution is approximated to that of the spl nu svm with the linear kernel in the latter case although extreme kernels are not employed in practice analyzes are helpful to understand the effects of a kernel function on the generalization performance

new configurations of 24 limit cycles in a quintic system this paper concerns with the number and distributions of limit cycles in a z 3 equivariant quintic planar polynomial system 24 limit cycles are found in this system and two different configurations of them are shown by combining the methods of double homoclinic loops bifurcation poincare bifurcation and qualitative analysis the two configurations of 24 limit cycles obtained in this paper are new the results obtained are useful to the study of weakened 16th hilbert problem

admissible selection of an accurate and parsimonious normal linear regression model let mo be a normal linear regression model and let m1 mk be distinct proper linear submodels of mo let k e 0 k be a model selection rule based on observed data from the true model given k let the unknown parameters of the selected model ms be fitted by the maximum likelihood method a loss function is introduced which depends additively on two parts i a measure of the difference between the fitted model ms and the true model and ii a measure cs of the complexity of the selected model a natural model selection rule k which minimizes an empirical version of this loss is shown to be admissible and very nearly bayes

statistical treatment rules for heterogeneous populations an important objective of empirical research on treatment response is to provide decision makers with information useful in choosing treatments this paper studies minimax regret treatment choice using the sample data generated by a classical randomized experiment consider a utilitarian social planner who must choose among the feasible statistical treatment rules these being functions that map the sample data and observed covariates of population members into a treatment allocation if the planner knew the population distribution of treatment response the optimal treatment rule would maximize mean welfare conditional on all observed covariates the appropriate use of covariate information is a more subtle matter when only sample data on treatment response are available i consider the class of conditional empirical success rules that is rules assigning persons to treatments that yield the best experimental outcomes conditional on alternative subsets of the observed covariates i derive a closed form bound on the maximum regret of any such rule comparison of the bounds for rules that condition on smaller and larger subsets of the covariates yields sufficient sample sizes for productive use of covariate information when the available sample size exceeds the sufficiency boundary a planner can be certain that conditioning treatment choice on more covariates is preferable in terms of minimax regret to conditioning on fewer covariates

fast support vector data descriptions for novelty detection support vector data description svdd has become a very attractive kernel method due to its good results in many novelty detection problems however the decision function of svdd is expressed in terms of the kernel expansion which results in a run time complexity linear in the number of support vectors for applications where fast real time response is needed how to speed up the decision function is crucial this paper aims at dealing with the issue of reducing the testing time complexity of svdd a method called fast svdd f svdd is proposed unlike the traditional methods which all try to compress a kernel expansion into one with fewer terms the proposed f svdd directly finds the preimage of a feature vector and then uses a simple relationship between this feature vector and the svdd sphere center to re express the center with a single vector the decision function of f svdd contains only one kernel term and thus the decision boundary of f svdd is only spherical in the original space hence the run time complexity of the f svdd decision function is no longer linear in the support vectors but is a constant no matter how large the training set size is in this paper we also propose a novel direct preimage finding method which is noniterative and involves no free parameters the unique preimage can be obtained in real time by the proposed direct method without taking trial and error for demonstration several real world data sets and a large scale data set the extended mit face data set are used in experiments in addition a practical industry example regarding liquid crystal display micro defect inspection is also used to compare the applicability of svdd and our proposed f svdd when faced with mass data input the results are very encouraging

cyclicity 1 and 2 conditions for a 2 polycycle of integrable systems on the plane abstract we obtain generic conditions for a 2 polycycle of an integrable system to have cyclicity one and two respectively under multiple parameter perturbations

generalized discriminant analysis using a kernel approach we present a new method that we call generalized discriminant analysis gda to deal with nonlinear discriminant analysis using kernel function operator the underlying theory is close to the support vector machines svm insofar as the gda method provides a mapping of the input vectors into high dimensional feature space in the transformed space linear properties make it easy to extend and generalize the classical linear discriminant analysis lda to nonlinear discriminant analysis the formulation is expressed as an eigenvalue problem resolution using a different kernel one can cover a wide class of nonlinearities for both simulated data and alternate kernels we give classification results as well as the shape of the decision function the results are confirmed using real data to perform seed classification

a general non local denoising model using multi kernel induced measures noises are inevitably introduced in digital image acquisition processes and thus image denoising is still a hot research problem different from local methods operating on local regions of images the non local methods utilize non local information even the whole image to accomplish image denoising due to their superior performance the non local methods have recently drawn more and more attention in the image denoising community however these methods generally do not work well in handling complicated noises with different levels and types inspired by the fact in machine learning field that multi kernel methods are more robust and effective in tackling complex problems than single kernel ones we establish a general non local denoising model based on multi kernel induced measures gnlmkim for short which provides us a platform to analyze some existing and design new filters with the help of gnlmkim we reinterpret two well known non local filters in the united view and extend them to their novel multi kernel counterparts the comprehensive experiments indicate that these novel filters achieve encouraging denoising results in both visual effect and psnr index

an introduction to kernel based learning algorithms this paper provides an introduction to support vector machines kernel fisher discriminant analysis and kernel principal component analysis as examples for successful kernel based learning methods we first give a short background about vapnik chervonenkis theory and kernel feature spaces and then proceed to kernel based learning in supervised and unsupervised scenarios including practical and algorithmic considerations we illustrate the usefulness of kernel algorithms by discussing applications such as optical character recognition and dna analysis

estimating the support of a high dimensional distribution suppose you are given some data set drawn from an underlying probability distribution p and you want to estimate a simple subset s of input space such that the probability that a test point drawn from p lies outside of s equals some a priori specified value between 0 and 1 we propose a method to approach this problem by trying to estimate a function f that is positive on s and negative on the complement the functional form of f is given by a kernel expansion in terms of a potentially small subset of the training data it is regularized by controlling the length of the weight vector in an associated feature space the expansion coefficients are found by solving a quadratic programming problem which we do by carrying out sequential optimization over pairs of input patterns we also provide a theoretical analysis of the statistical performance of our algorithm the algorithm is a natural extension of the support vector algorithm to the case of unlabeled data

local normalized linear summation kernel for fast and robust recognition kernel based methods are effective for object detection and recognition however the computational cost when using kernel functions is high except when using linear kernels to realize fast and robust recognition we apply normalized linear kernels to local regions of a recognition target and the kernel outputs are integrated by summation this kernel is referred to as a local normalized linear summation kernel here we show that kernel based methods that employ local normalized linear summation kernels can be computed by a linear kernel of local normalized features thus the computational cost of the kernel is nearly the same as that of a linear kernel and much lower than that of radial basis function rbf and polynomial kernels the effectiveness of the proposed method is evaluated in face detection and recognition problems and we confirm that our kernel provides higher accuracy with lower computational cost than rbf and polynomial kernels in addition our kernel is also robust to partial occlusion and shadows on faces since it is based on the summation of local kernels

kernel quadratic discriminant analysis for small sample size problem it is generally believed that quadratic discriminant analysis qda can better fit the data in practical pattern recognition applications compared to linear discriminant analysis lda method this is due to the fact that qda relaxes the assumption made by lda based methods that the covariance matrix for each class is identical however it still assumes that the class conditional distribution is gaussian which is usually not the case in many real world applications in this paper a novel kernel based qda method is proposed to further relax the gaussian assumption by using the kernel machine technique the proposed method solves the complex pattern recognition problem by combining the qda solution and the kernel machine technique and at the same time tackles the so called small sample size problem through a regularized estimation of the covariance matrix extensive experimental results indicate that the proposed method is a more sophisticated solution outperforming many traditional kernel based learning algorithms

kernel spectral matched filter for hyperspectral imagery in this paper a kernel based nonlinear spectral matched filter is introduced for target detection in hyperspectral imagery which is implemented by using the ideas in kernel based learning theory a spectral matched filter is defined in a feature space of high dimensionality which is implicitly generated by a nonlinear mapping associated with a kernel function a kernel version of the matched filter is derived by expressing the spectral matched filter in terms of the vector dot products form and replacing each dot product with a kernel function using the so called kernel trick property of the mercer kernels the proposed kernel spectral matched filter is equivalent to a nonlinear matched filter in the original input space which is capable of generating nonlinear decision boundaries the kernel version of the linear spectral matched filter is implemented and simulation results on hyperspectral imagery show that the kernel spectral matched filter outperforms the conventional linear matched filter

probabilistic principal component analysis principal component analysis pca is a ubiquitous technique for data analysis and processing but one which is not based upon a probability model in this paper we demonstrate how the principal axes of a set of observed data vectors may be determined through maximum likelihood estimation of parameters in a latent variable model closely related to factor analysis we consider the properties of the associated likelihood function giving an em algorithm for estimating the principal subspace iteratively and discuss the advantages conveyed by the definition of a probability density function for pca

penalized discriminant analysis fisher s linear discriminant analysis lda is a popular data analytic tool for studying the relationship between a set of predictors and a categorical response in this paper we describe a penalized version of lda it is designed for situations in which there are many highly correlated predictors such as those obtained by discretizing a function or the grey scale values of the pixels in a series of images in cases such as these it is natural efficient and sometimes essential to impose a spatial smoothness constraint on the coefficients both for improved prediction performance and interpretability we cast the classification problem into a regression framework via optimal scoring using this our proposal facilitates the use of any penalized regression technique in the classification setting the technique is illustrated with examples in speech recognition and handwritten character recognition

linear time computation of similarity measures for sequential data efficient and expressive comparison of sequences is an essential procedure for learning with sequential data in this article we propose a generic framework for computation of similarity measures for sequences covering various kernel distance and non metric similarity functions the basis for comparison is embedding of sequences using a formal language such as a set of natural words k grams or all contiguous subsequences as realizations of the framework we provide linear time algorithms of different complexity and capabilities using sorted arrays tries and suffix trees as underlying data structures experiments on data sets from bioinformatics text processing and computer security illustrate the efficiency of the proposed algorithms enabling peak performances of up to 106 pairwise comparisons per second the utility of distances and non metric similarity measures for sequences as alternatives to string kernels is demonstrated in applications of text categorization network intrusion detection and transcription site recognition in dna

generalization performance of bayes optimal classification algorithm for learning a perceptron the generalization error of the bayes optimal classification algorithm when learning a perceptron from noise free random training examples is calculated exactly using methods of statistical mechanics it is shown that if an assumption of replica symmetry is made then in the thermodynamic limit the error of the bayes optimal algorithm is less than the error of a canonical stochastic learning algorithm by a factor approaching ensuremath surd 2 as the ratio of the number of training examples to perceptron weights grows in addition it is shown that approximations to the generalization error of the bayes optimal algorithm can be achieved by learning algorithms that use a two layer neutral net to learn a perceptron

detecting generalized synchronization between chaotic signals a kernel based approach a unified framework for analysing generalized synchronization in coupled chaotic systems from data is proposed the key of the proposed approach is the use of the kernel methods recently developed in the field of machine learning several successful applications are presented which show the capability of the kernel based approach for detecting generalized synchronization and dynamical change of the coupling strength between two chaotic systems can be captured by the proposed approach it is also discussed how the kernel parameter is suitably chosen from data

an optimal selection of regression variables summary an asymptotically optimal selection of regression variables is proposed the key assumption is that the number of control variables is infinite or increases with the sample size it is also shown that mallows s qp akaike s fpe and aic methods are all asymptotically equivalent to this method

a kernel based parametric method for conditional density estimation a conditional density function which describes the relationship between response and explanatory variables plays an important role in many analysis problems in this paper we propose a new kernel based parametric method to estimate conditional density an exponential function is employed to approximate the unknown density and its parameters are computed from the given explanatory variable via a nonlinear mapping using kernel principal component analysis kpca we develop a new kernel function which is a variant to polynomial kernels to be used in kpca the proposed method is compared with the nadaraya watson estimator through numerical simulation and practical data experimental results show that the proposed method outperforms the nadaraya watson estimator in terms of revised mean integrated squared error rmise therefore the proposed method is an effective method for estimating the conditional densities

prediction games and arcing algorithms the theory behind the success of adaptive reweighting and combining algorithms arcing such as adaboost freund schapire 1996a 1997 and others in reducing generalization error has not been well understood by formulating prediction as a game where one player makes a selection from instances in the training set and the other a convex linear combination of predictors from a finite set existing arcing algorithms are shown to be algorithms for finding good game strategies the minimax theorem is an essential ingredient of the convergence proofs an arcing algorithm is described that converges to the optimal strategy a bound on the generalization error for the combined predictors in terms of their maximum error is proven that is sharper than bounds to date schapire freund bartlett and lee 1997 offered an explanation of why adaboost works in terms of its ability to produce generally high margins the empirical comparison of adaboost to the optimal arcing algorithm shows that their explanation is n

playing billiards in version space a ray tracing method inspired by ergodic billiards is used to estimate the theoretically best decision rule for a given set of linear separable examples for randomly distributed examples the billiard estimate of the single perceptron with best average generalization probability agrees with known analytic results while for real life classification problems the generalization probability is consistently enhanced when compared to the maximal stability perceptron

input space versus feature space in kernel based methods this paper collects some ideas targeted at advancing our understanding of the feature spaces associated with support vector sv kernel functions we first discuss the geometry of feature space in particular we review what is known about the shape of the image of input space under the feature space map and how this influences the capacity of sv methods following this we describe how the metric governing the intrinsic geometry of the mapped surface can be computed in terms of the kernel using the example of the class of inhomogeneous polynomial kernels which are often used in sv pattern recognition we then discuss the connection between feature space and input space by dealing with the question of how one can given some vector in feature space find a preimage exact or approximate in input space we describe algorithms to tackle this issue and show their utility in two applications of kernel methods first we use it to reduce the computational complexity of sv decision functions second we combine it with the kernel pca algorithm thereby constructing a nonlinear statistical denoising technique which is shown to perform well on real world data

quadratic projection based feature extraction with its application to biometric recognition this paper presents a novel quadratic projection based feature extraction framework where a set of quadratic matrices is learned to distinguish each class from all other classes we formulate quadratic matrix learning qml as a standard semidefinite programming sdp problem however the conventional interior point sdp solvers do not scale well to the problem of qml for high dimensional data to solve the scalability of qml we develop an efficient algorithm termed dualqml based on the lagrange duality theory to extract nonlinear features to evaluate the feasibility and effectiveness of the proposed framework we conduct extensive experiments on biometric recognition experimental results on three representative biometric recognition tasks including face palmprint and ear recognition demonstrate the superiority of the dualqml based feature extraction algorithm compared to the current state of the art algorithms highlightsa novel quadratic projection based feature extraction framework is developed a set of quadratic matrices is learnt to distinguish each class from other classes quadratic matrix learning qml is formulated as an sdp problem an efficient algorithm is developed to solve qml based on lagrange duality theory experiments on biometric recognition show the effectiveness of our algorithm

domain adaptation via transfer component analysis domain adaptation allows knowledge from a source domain to be transferred to a different but related target domain intuitively discovering a good feature representation across domains is crucial in this paper we first propose to find such a representation through a new learning method transfer component analysis tca for domain adaptation tca tries to learn some transfer components across domains in a reproducing kernel hilbert space using maximum mean miscrepancy in the subspace spanned by these transfer components data properties are preserved and data distributions in different domains are close to each other as a result with the new representations in this subspace we can apply standard machine learning methods to train classifiers or regression models in the source domain for use in the target domain furthermore in order to uncover the knowledge hidden in the relations between the data labels from the source and target domains we extend tca in a semisupervised learning setting which encodes label information into transfer components learning we call this extension semisupervised tca the main contribution of our work is that we propose a novel dimensionality reduction framework for reducing the distance between domains in a latent space for domain adaptation we propose both unsupervised and semisupervised feature extraction approaches which can dramatically reduce the distance between domain distributions by projecting data onto the learned transfer components finally our approach can handle large datasets and naturally lead to out of sample generalization the effectiveness and efficiency of our approach are verified by experiments on five toy datasets and two real world applications cross domain indoor wifi localization and cross domain text classification