estimation of gradients and coordinate covariation in classification we introduce an algorithm that simultaneously estimates a classification function as well as its gradient in the supervised learning framework the motivation for the algorithm is to find salient variables and estimate how they covary an efficient implementation with respect to both memory and time is given the utility of the algorithm is illustrated on simulated data as well as a gene expression data set an error analysis is given for the convergence of the estimate of the classification function and its gradient to the true classification function and true gradient

human detection in images via piecewise linear support vector machines human detection in images is challenged by the view and posture variation problem in this paper we propose a piecewise linear support vector machine pl svm method to tackle this problem the motivation is to exploit the piecewise discriminative function to construct a nonlinear classification boundary that can discriminate multiview and multiposture human bodies from the backgrounds in a high dimensional feature space a pl svm training is designed as an iterative procedure of feature space division and linear svm training aiming at the margin maximization of local linear svms each piecewise svm model is responsible for a subspace corresponding to a human cluster of a special view or posture in the pl svm a cascaded detector is proposed with block orientation features and a histogram of oriented gradient features extensive experiments show that compared with several recent svm methods our method reaches the state of the art in both detection accuracy and computational efficiency and it performs best when dealing with low resolution human regions in clutter backgrounds

stabilized nearest neighbor classifier and its statistical properties abstractthe stability of statistical analysis is an important indicator for reproducibility which is one main principle of the scientific method it entails that similar statistical conclusions can be reached based on independent samples from the same underlying population in this article we introduce a general measure of classification instability cis to quantify the sampling variability of the prediction made by a classification method interestingly the asymptotic cis of any weighted nearest neighbor classifier turns out to be proportional to the euclidean norm of its weight vector based on this concise form we propose a stabilized nearest neighbor snn classifier which distinguishes itself from other nearest neighbor classifiers by taking the stability into consideration in theory we prove that snn attains the minimax optimal convergence rate in risk and a sharp convergence rate in cis the latter rate result is established for general plug in classifiers under a low noise condition ext

stable particle like solutions to the nonlinear wave equations of three dimensional excitable media so called excitable media support nonlinear waves that propagate as pulses e g the action potential in nerve fibers and heart muscle and the oxidation pulse in certain chemical reactions in two dimensional media the right stimulus will also create rotating vortices which radiate such pulses periodically in every direction their investigation in diverse excitable media has dramatically expanded in the usa and ussr since their discovery in both countries in 1970 in three dimensional media the pivot of the rotating structure is not a point but a curved line generically a closed ring rings linked together and knotted constitute topologically distinctive organizing centers for periodic activity throughout the medium it is of interest to understand how stimuli create organizing centers and how organizing centers persist or decay the numerically discovered stable organizing centers exhibit enough symmetries to encourage mathematical attention this paper illustrates organizing centers by computer g

structured multicategory support vector machines with analysis of variance decomposition the support vector machine has been a popular choice of classification method for many applications in machine learning while it often outperforms other methods in terms of classification accuracy the implicit nature of its solution renders the support vector machine less attractive in providing insights into the relationship between covariates and classes use of structured kernels can remedy the drawback borrowing the flexible model building idea of functional analysis of variance decomposition we consider multicategory support vector machines with analysis of variance kernels in this paper an additional penalty is imposed on the sum of weights of functional subspaces which encourages a sparse representation of the solution incorporation of the additional penalty enhances the interpretability of a resulting classifier with often improved accuracy the proposed method is demonstrated through simulation studies and an application to real data

statistical analysis of some multi category large margin classification methods the purpose of this paper is to investigate statistical properties of risk minimization based multi category classification methods these methods can be considered as natural extensions of binary large margin classification we establish conditions that guarantee the consistency of classifiers obtained in the risk minimization framework with respect to the classification error examples are provided for four specific forms of the general formulation which extend a number of known methods using these examples we show that some risk minimization formulations can also be used to obtain conditional probability estimates for the underlying problem such conditional probability information can be useful for statistical inferencing tasks beyond classification

adaptive weighted learning for unbalanced multicategory classification summary in multicategory classification standard techniques typically treat all classes equally this treatment can be problematic when the dataset is unbalanced in the sense that certain classes have very small class proportions compared to others the minority classes may be ignored or discounted during the classification process due to their small proportions this can be a serious problem if those minority classes are important in this article we study the problem of unbalanced classification and propose new criteria to measure classification accuracy moreover we propose three different weighted learning procedures two one step weighted procedures as well as one adaptive weighted procedure we demonstrate the advantages of the new procedures using multicategory support vector machines through simulated and real datasets our results indicate that the proposed methodology can handle unbalanced classification problems effectively

on the characterization of a class of fisher consistent loss functions and its application to boosting accurate classification of categorical outcomes is essential in a wide range of applications due to computational issues with minimizing the empirical 0 1 loss fisher consistent losses have been proposed as viable proxies however even with smooth losses direct minimization remains a daunting task to approximate such a minimizer various boosting algorithms have been suggested for example with exponential loss the adaboost algorithm freund and schapire 1995 is widely used for two class problems and has been extended to the multi class setting zhu et al 2009 alternative loss functions such as the logistic and the hinge losses and their corresponding boosting algorithms have also been proposed zou et al 2008 wang 2012 in this paper we demonstrate that a broad class of losses including non convex functions achieve fisher consistency and in addition can be used for explicit estimation of the conditional class probabilities furthermore we provide a generic boosting algorithm that is not loss specific extensive simulation results suggest that the proposed boosting algorithms could outperform existing methods with properly chosen losses and bags of weak learners

multi class boosting with asymmetric binary weak learners we introduce a multi class generalization of adaboost with binary weak learners we use a vectorial codification to represent class labels and a multi class exponential loss function to evaluate classifier responses this representation produces a set of margin values that provide a range of punishments for failures and rewards for successes moreover the stage wise optimization of this model introduces an asymmetric boosting procedure whose costs depend on the number of classes separated by each weak learner in this way the boosting algorithm takes into account class imbalances when building the ensemble the experiments performed compare this new approach favorably to adaboost mh gentleboost and the samme algorithms

measuring and testing dependence by correlation of distances distance correlation is a new measure of dependence between random vectors distance covariance and distance correlation are analogous to product moment covariance and correlation but unlike the classical definition of correlation distance correlation is zero only if the random vectors are independent the empirical distance dependence measures are based on certain euclidean distances between sample elements rather than sample moments yet have a compact representation analogous to the classical covariance and correlation asymptotic properties and applications in testing independence are discussed implementation of the test and monte carlo results are also presented

the discrete frenkel kontorova model and its extensions i exact results for the ground states abstract we present a rigorous study of the classical ground states under boundary conditions of a class of one dimensional models generalizing the discrete frenkel kontorova model the extremalization equations of the energy of these models turn out to define area preserving twist maps which exhibits periodic quasi periodic and chaotic orbits for all boundary conditions we select among all the extremum solutions of the energy of the model those which correspond to the ground states of the infinite system we prove that these ground states are either periodic commensurate or quasi periodic incommensurate but are never chaotic we also prove the existence of elementary discommensurations which are minimum energy configuration of the model for certain special boundary conditions the topological structure of the whole set of ground states is described in details in addition to physical applications consequences for twist map homeomorphisms are mentioned part ii s aubry p y ledaeron and g andre will be mostly devoted to exact results on the transition by breaking of analyticity which occurs on the incommensurate ground states when the model parameters vary and on its connection with the stochasticity threshold in the corresponding twist map

a unified view on multi class support vector classification a unified view on multi class support vector machines svms is presented covering most prominent variants including the one vs all approach and the algorithms proposed by weston watkins crammer singer lee lin wahba and liu yuan the unification leads to a template for the quadratic training problems and new multi class svm formulations within our framework we provide a comparative analysis of the various notions of multi class margin and margin based loss in particular we demonstrate limitations of the loss function considered for instance in the crammer singer machine we analyze fisher consistency of multi class loss functions and universal consistency of the various machines on the one hand we give examples of svms that are in a particular hyperparameter regime universally consistent without being based on a fisher consistent loss these include the canonical extension of svms to multiple classes as proposed by weston watkins and vapnik as well as the one vs all approach on the other hand it is demonstrated that machines based on fisher consistent loss functions can fail to identify proper decision boundaries in low dimensional feature spaces we compared the performance of nine different multi class svms in a thorough empirical study our results suggest to use the weston watkins svm which can be trained comparatively fast and gives good accuracies on benchmark functions if training time is a major concern the one vs all approach is the method of choice

a geometrical theory for spiral waves in excitable media in this paper we develop a geometrical theory for waves in excitable reacting media using singular perturbation arguments and dispersion of traveling plane wave trains we derive an approximate theory of wave front propagation which has strong resemblance to the geometrical diffraction theory of high frequency waves in hyperbolic systems governed by the eikonal equation using this theory we study the effect of curvature on waves in excitable media specifically rotating spiral patterns in planar regions from this theory we are able to determine the frequency and wavelength for spiral patterns in excitable nonoscillatory media

piecewise affine regression via recursive multiple least squares and multicategory discrimination in nonlinear regression choosing an adequate model structure is often a challenging problem while simple models such as linear functions may not be able to capture the underlying relationship among the variables over parametrized models described by a large set of nonlinear basis functions tend to overfit the training data leading to poor generalization on unseen data piecewise affine pwa models can describe nonlinear and possible discontinuous relationships while maintaining simple local affine regressor to output mappings with extreme flexibility when the polyhedral partitioning of the regressor space is learned from data rather than fixed a priori in this paper we propose a novel and numerically very efficient two stage approach for pwa regression based on a combined use of i recursive multi model least squares techniques for clustering and fitting linear functions to data and ii linear multi category discrimination either offline batch via a newton like algorithm for computing a solution of unconstrained optimization problems with objective functions having a piecewise smooth gradient or online recursive via averaged stochastic gradient descent

optimal treatment assignment to maximize expected outcome with multiple treatments when there is substantial heterogeneity of treatment effectiveness it is crucial to identify individualized treatment assignment rules for comparative treatment selection traditional approaches directly model clinical outcome and define optimal treatment rule according to the interactions between treatment and covariates this approach relies on the success of separating the main effects from the covariate treatment interaction effects which may not be easy to overcome this shortcoming a recent approach called outcome weighted learning focuses on building an optimal treatment rule by maximizing the expected clinical outcome related with differential treatments however there seems to be a lack of approaches to explicitly deal with three or more treatments in this article we propose an outcome weighted learning method that extends estimating individualized treatment rules to multi treatment case by using a vector hinge loss as a target function consistency of the resulting estimator is shown in the article we demonstrate the performance of our approach in simulation studies and in a real data analysis

multicategory support vector machines two category support vector machines svm have been very popular in the machine learning community for classification problems solving multicategory problems by a series of binary classifiers is quite common in the svm paradigm however this approach may fail under various circumstances we propose the multicategory support vector machine msvm which extends the binary svm to the multicategory case and has good theoretical properties the proposed method provides a unifying framework when there are either equal or unequal misclassification costs as a tuning criterion for the msvm an approximate leave one out cross validation function called generalized approximate cross validation is derived analogous to the binary case the effectiveness of the msvm is demonstrated through the applications to cancer classification using microarray data and cloud classification with satellite radiance profiles

soft and hard classification by reproducing kernel hilbert space methods reproducing kernel hilbert space rkhs methods provide a unified context for solving a wide variety of statistical modelling and function estimation problems we consider two such problems we are given a training set yi ti i 1 n where yi is the response for the ith subject and ti is a vector of attributes for this subject the value of yi is a label that indicates which category it came from for the first problem we wish to build a model from the training set that assigns to each t in an attribute domain of interest an estimate of the probability pj t that a future subject with attribute vector t is in category j the second problem is in some sense less ambitious it is to build a model that assigns to each t a label which classifies a future subject with that t into one of the categories or possibly none of the above the approach to the first of these two problems discussed here is a special case of what is known as penalized likelihood estimation the approach to the second problem is known as the support vector machine we also note some alternate but closely related approaches to the second problem these approaches are all obtained as solutions to optimization problems in rkhs many other problems in particular the solution of ill posed inverse problems can be obtained as solutions to optimization problems in rkhs and are mentioned in passing we caution the reader that although a large literature exists in all of these topics in this inaugural article we are selectively highlighting work of the author former students and other collaborators

a unifying framework in vector valued reproducing kernel hilbert spaces for manifold regularization and co regularized multi view learning this paper presents a general vector valued reproducing kernel hilbert spaces rkhs framework for the problem of learning an unknown functional dependency between a structured input space and a structured output space our formulation encompasses both vector valued manifold regularization and co regularized multi view learning providing in particular a unifying framework linking these two important learning approaches in the case of the least square loss function we provide a closed form solution which is obtained by solving a system of linear equations in the case of support vector machine svm classification our formulation generalizes in particular both the binary laplacian svm to the multi class multi view settings and the multi class simplex cone svm to the semi supervised multi view settings the solution is obtained by solving a single quadratic optimization problem as in standard svm via the sequential minimal optimization smo approach empirical results obtained on the task of object recognition using several challenging data sets demonstrate the competitiveness of our algorithms compared with other state of the art methods

convex calibration dimension for multiclass loss matrices we study consistency properties of surrogate loss functions for general multiclass learning problems defined by a general multiclass loss matrix we extend the notion of classification calibration which has been studied for binary and multiclass 0 1 classification problems and for certain other specific learning problems to the general multiclass setting and derive necessary and sufficient conditions for a surrogate loss to be calibrated with respect to a loss matrix in this setting we then introduce the notion of convex calibration dimension of a multiclass loss matrix which measures the smallest size of a prediction space in which it is possible to design a convex surrogate that is calibrated with respect to the loss matrix we derive both upper and lower bounds on this quantity and use these results to analyze various loss matrices in particular we apply our framework to study various subset ranking losses and use the convex calibration dimension as a tool to show both the existence and non existence of various types of convex calibrated surrogates for these losses our results strengthen recent results of duchi et al 2010 and calauzenes et al 2012 on the non existence of certain types of convex calibrated surrogates in subset ranking we anticipate the convex calibration dimension may prove to be a useful tool in the study and design of surrogate losses for general multiclass learning problems

multicategory ψ learning in binary classification margin based techniques usually deliver high performance as a result a multicategory problem is often treated as a sequence of binary classifications in the absence of a dominating class this treatment may be suboptimal and may yield poor performance such as for support vector machines svms we propose a novel multicategory generalization of learning that treats all classes simultaneously the new generalization eliminates this potential problem while at the same time retaining the desirable properties of its binary counterpart we develop a statistical learning theory for the proposed methodology and obtain fast convergence rates for both linear and nonlinear learning examples we demonstrate the operational characteristics of this method through a simulation our results indicate that the proposed methodology can deliver accurate class prediction and is more robust against extreme observations than its svm counterpart

on the aubry mather theory in statistical mechanics we generalize aubry mather theory for configurations on the line to general sets with a group action cocycles on the group play the role of rotation numbers the notion of birkhoff configuration can be generalized to this setting under mild conditions on the group we show how to find birkhoff ground states for many body interactions which are ferromagnetic invariant under the group action and having periodic phase space

on reject and refine options in multicategory classification abstractin many real applications of statistical learning a decision made from misclassification can be too costly to afford in this case a reject option which defers the decision until further investigation is conducted is often preferred in recent years there has been much development for binary classification with a reject option yet little progress has been made for the multicategory case in this article we propose margin based multicategory classification methods with a reject option in addition and more importantly we introduce a new and unique refine option for the multicategory problem where the class of an observation is predicted to be from a set of class labels whose cardinality is not necessarily one the main advantage of both options lies in their capacity of identifying error prone observations moreover the refine option can provide more constructive information for classification by effectively ruling out implausible classes efficient implementations have been developed

bias corrected diagonal discriminant rules for high dimensional classification diagonal discriminant rules have been successfully used for high dimensional classification problems but suffer from the serious drawback of biased discriminant scores in this article we propose improved diagonal discriminant rules with bias corrected discriminant scores for high dimensional classification we show that the proposed discriminant scores dominate the standard ones under the quadratic loss function analytical results on why the bias corrected rules can potentially improve the predication accuracy are also provided finally we demonstrate the improvement of the proposed rules over the original ones through extensive simulation studies and real case studies

clifford support vector machines for classification regression and recurrence this paper introduces the clifford support vector machines csvm as a generalization of the real and complex valued support vector machines using the clifford geometric algebra in this framework we handle the design of kernels involving the clifford or geometric product in this approach one redefines the optimization variables as multivectors this allows us to have a multivector as output therefore we can represent multiple classes according to the dimension of the geometric algebra in which we work we show that one can apply csvm for classification and regression and also to build a recurrent csvm the csvm is an attractive approach for the multiple input multiple output processing of high dimensional geometric entities we carried out comparisons between csvm and the current approaches to solve multiclass classification and regression we also study the performance of the recurrent csvm with experiments involving time series the authors believe that this paper can be of great use for researchers and practitioners interested in multiclass hypercomplex computing particularly for applications in complex and quaternion signal and image processing satellite control neurocomputation pattern recognition computer vision augmented virtual reality robotics and humanoids

gensvm a generalized multiclass support vector machine traditional extensions of the binary support vector machine svm to multiclass problems are either heuristics or require solving a large dual optimization problem here a generalized multiclass svm is proposed called gensvm in this method classification boundaries for a k class problem are constructed in a k 1 dimensional space using a simplex encoding additionally several different weightings of the misclassification errors are incorporated in the loss function such that it generalizes three existing multiclass svms through a single optimization problem an iterative majorization algorithm is derived that solves the optimization problem without the need of a dual formulation this algorithm has the advantage that it can use warm starts during cross validation and during a grid search which significantly speeds up the training phase rigorous numerical experiments compare linear gensvm with seven existing multiclass svms on both small and large data sets these comparisons show that the proposed method is competitive with existing methods in both predictive accuracy and training time and that it significantly outperforms several existing methods on these criteria

spiral waves in the belousov zhabotinskii reaction abstract the beautiful spiral waves of oxidation in the belousov zhabotinskii reaction are the source of many interesting and important questions about periods structures in excitable media it has long been known that these spirals are similar to involutes of circles at least some distance from the center but until now no way has been known to determine the correct wavelength and frequency in this paper we show that the parameters of a spiral wave can be viwed s eigenvalues of a problem with unique solution the critical ingredients of the theory are the effects of curvature on the propagation of wavefronts in two dimensional media and the dispersion of plane waves our analytical results are shown to be in good agreement with experimental data for the belousov zhabotinskii reagent

classification by pairwise coupling we discuss a strategy for polychotomous classification that involves estimating class probabilities for each pair of classes and then coupling the estimates together the coupling model is similar to the bradley terry method for paired comparisons we study the nature of the class probability estimates that arise and examine the performance of the procedure in real and simulated data sets classifiers used include linear discriminants nearest neighbors adaptive nonlinear methods and the support vector machine

the eikonal approximation to excitable reaction diffusion systems travelling non planar wave fronts on the plane abstract exact non planar travelling solutions of the eikonal equation on an infinite plane are presented for the first time these solutions are matched to produce corrugated wave fronts and patterns such as spot solutions as well as extended parabolic type wave fronts the stability of these solutions is also analysed the variational equation which belongs to a generalised wangerin class of differential equations is solved first with the aid of the liouville green approximation for the estimated eigenvalues characterising stability and then by a more elaborate shooting matching method all of the three types of travelling solutions are found to be geometrically stable it is suggested that some of these predictions are experimentally testable

statistical behavior and consistency of classification methods based on convex risk minimization we study how closely the optimal bayes error rate can be approximately reached using a classification algorithm that computes a classifier by minimizing a convex upper bound of the classification error function the measurement of closeness is characterized by the loss function used in the estimation we show that such a classification scheme can be generally regarded as a nonmaximum likelihood conditional in class probability estimate and we use this analysis to compare various convex loss functions that have appeared in the literature furthermore the theoretical insight allows us to design good loss functions with desirable properties another aspect of our analysis is to demonstrate the consistency of certain classification methods using convex risk minimization this study sheds light on the good performance of some recently proposed linear classification methods including boosting and support vector machines it also shows their limitations and suggests possible improvements

successive overrelaxation for support vector machines successive overrelaxation sor for symmetric linear complementarity problems and quadratic programs is used to train a support vector machine svm for discriminating between the elements of two massive datasets each with millions of points because sor handles one point at a time similar to platt s sequential minimal optimization smo algorithm 1999 which handles two constraints at a time and joachims svm sup light 1998 which handles a small number of points at a time sor can process very large datasets that need not reside in memory the algorithm converges linearly to a solution encouraging numerical results are presented on datasets with up to 10 000 000 points such massive discrimination problems cannot be processed by conventional linear or quadratic programming methods and to our knowledge have not been solved by other methods on smaller problems sor was faster than svm sup light and comparable or faster than smo

diversity of three dimensional chemical waves abstract the propagation of spiral waves in the belousov zhabotinsky bz reaction has been extensively investigated the aim of this paper is to show the diversity of phenomenon in the three dimensional bz reagent with the aid of photographs of the evolving structures not all of which have been studied mathematically the most frequent observed wave forms are simple scrolls and toroidal scrolls a brief discussion of the existence of toroidal scroll solution to the eikonal equation in three dimensions is presented

a multi class svm approach based on the l1 norm minimization of the distances between the reduced convex hulls multi class classification is an important pattern recognition task that can be addressed accurately and efficiently by support vector machine svm in this work we present a novel svm based multi class classification approach based on the center of the configuration a point which is equidistant to all classes the center of the configuration is obtained from the dual formulation by minimizing the distances between the reduced convex hulls using the l1 norm while the decision functions are subsequently constructed from this point this work also extends the ideas of zhou et al 2002 37 to multi class classification the use of l1 norm provides a single linear programming formulation which reduces the complexity and confers scalability compared with other multi class svm methods based on quadratic programming formulations experiments on benchmark datasets demonstrate the virtues of our approach in terms of classification performance and running times compared with various other multi class svm methods highlightsnovel linear programming approach for multi class svm a geometrically grounded method based on the concept of reduced convex hulls good classification performance is achieved with short running times

convexity classification and risk bounds many of the classification algorithms developed in the machine learning literature including the support vector machine and boosting can be viewed as minimum contrast methods that minimize a convex surrogate of the 0 1 loss function the convexity makes these algorithms computationally efficient the use of a surrogate however has statistical consequences that must be balanced against the computational virtues of convexity to study these issues we provide a general quantitative relationship between the risk as assessed using the 0 1 loss and the risk as assessed using any nonnegative surrogate loss function we show that this relationship gives nontrivial upper bounds on excess risk under the weakest possible condition on the loss function that it satisfies a pointwise form of fisher consistency for classification the relationship is based on a simple variational transformation of the loss function that is easy to compute in many applications we also present a refined version of this result in the

data dependent generalization bounds for multi class classification in this paper we study data dependent generalization error bounds that exhibit a mild dependency on the number of classes making them suitable for multi class learning with a large number of label classes the bounds generally hold for empirical multi class risk minimization algorithms using an arbitrary norm as the regularizer key to our analysis is new structural results for multi class gaussian complexities and empirical ell infty norm covering numbers which exploit the lipschitz continuity of the loss function with respect to the ell 2 and ell infty norm respectively we establish data dependent error bounds in terms of the complexities of a linear function class defined on a finite set induced by training examples for which we show tight lower and upper bounds we apply the results to several prominent multi class learning machines and show a tighter dependency on the number of classes than the state of the art for instance for the multi class support vector machine of crammer and singer 2002 we obtain a data dependent bound with a logarithmic dependency which is a significant improvement of the previous square root dependency the experimental results are reported to verify the effectiveness of our theoretical findings

on detecting nonlinear patterns in discriminant problems we propose a two stage model for detecting nonlinear patterns in discriminant problems and for solving the problem the model deploys a linear programming based discriminator lpbd in stage one for treating linear patterns and a probabilistic neural network pnn in stage two for handling nonlinear patterns the lpbd in stage one divides the decision space into a clear zone where observations are almost linearly separable and a gray zone where nonlinear patterns are more likely to occur the pnn in stage two analyzes the gray zone and determines whether a significant nonlinear patterns exist in the observations our goal is to avoid using a nonlinear model unless the pnn strongly suggests so to maintain good interpretability and avoid overfitting our computational study demonstrates the effectiveness of the two stage model in both classification accuracy and computational efficiency

residual weighted learning for estimating individualized treatment rules abstractpersonalized medicine has received increasing attention among statisticians computer scientists and clinical practitioners a major component of personalized medicine is the estimation of individualized treatment rules itrs recently zhao et al proposed outcome weighted learning owl to construct itrs that directly optimize the clinical outcome although owl opens the door to introducing machine learning techniques to optimal treatment regimes it still has some problems in performance 1 the estimated itr of owl is affected by a simple shift of the outcome 2 the rule from owl tries to keep treatment assignments that subjects actually received 3 there is no variable selection mechanism with owl all of them weaken the finite sample performance of owl in this article we propose a general framework called residual weighted learning rwl to alleviate these problems and hence to improve finite sample performance unlike owl which weights misclassification errors by clinical outcom

optimization approaches to supervised classification abstract the supervised classification problem one of the oldest and most recurrent problems in applied data analysis has always been analyzed from many different perspectives when the emphasis is placed on its overall goal of developing classification rules with minimal classification cost supervised classification can be understood as an optimization problem on the other hand when the focus is in modeling the uncertainty involved in the classification of future unknown entities it can be formulated as a statistical problem other perspectives that pay particular attention to pattern recognition and machine learning aspects of supervised classification have also a long history that has lead to influential insights and different methodologies in this review two approaches to supervised classification strongly related to optimization theory will be discussed and compared in particular we will review methodologies based on mathematical programming models that optimize observable criteria linked to the true objective of misclassification error or cost minimization and approaches derived from the minimization of known bounds on the true misclassification error the former approach is known as the mathematical programming approach to supervised classification while the latter is in the origin of the well known classification support vector machines throughout the review two group as well as general multi group problems will be considered and the review will conclude with a discussion of the most promising research directions in this area

reducing multiclass to binary a unifying approach for margin classifiers we present a unifying framework for studying the solution of multiclass categorization problems by reducing them to multiple binary problems that are then solved using a margin based binary learning algorithm the proposed framework unifies some of the most popular approaches in which each class is compared against all others or in which all pairs of classes are compared to each other or in which output codes with error correcting properties are used we propose a general method for combining the classifiers generated on the binary problems and we prove a general empirical multiclass loss bound given the empirical loss of the individual binary learning algorithms the scheme and the corresponding bounds apply to many popular classification learning algorithms including support vector machines adaboost regression logistic regression and decision tree algorithms we also give a multiclass generalization error analysis for general output codes with adaboost as the binary learner experimental results with svm and adaboost show that our scheme provides a viable alternative to the most commonly used multiclass algorithms

in defense of one vs all classification we consider the problem of multiclass classification our main thesis is that a simple one vs all scheme is as accurate as any other approach assuming that the underlying binary classifiers are well tuned regularized classifiers such as support vector machines this thesis is interesting in that it disagrees with a large body of recent published work on multiclass classification we support our position by means of a critical review of the existing literature a substantial collection of carefully controlled experimental work and theoretical arguments

multicategory large margin classification methods hinge losses vs coherence functions generalization of large margin classification methods from the binary classification setting to the more general multicategory setting is often found to be non trivial in this paper we study large margin classification methods that can be seamlessly applied to both settings with the binary setting simply as a special case in particular we explore the fisher consistency properties of multicategory majorization losses and present a construction framework of majorization losses of the 0 1 loss under this framework we conduct an in depth analysis about three widely used multicategory hinge losses corresponding to the three hinge losses we propose three multicategory majorization losses based on a coherence function the limits of the three coherence losses as the temperature approaches zero are the corresponding hinge losses and the limits of the minimizers of their expected errors are the minimizers of the expected errors of the corresponding hinge losses finally we develop multicategory large margin classification methods by using a so called multiclass c loss

a moment bound for multi hinge classifiers the success of support vector machines in binary classification relies on the fact that hinge loss employed in the risk minimization targets the bayes rule recent research explores some extensions of this large margin based method to the multicategory case we show a moment bound for the socalled multi hinge loss minimizers based on two kinds of complexity constraints entropy with bracketing and empirical entropy obtaining such a result based on the latter is harder than finding one based on the former we obtain fast rates of convergence that adapt to the unknown margin

classification with a reject option using a hinge loss we consider the problem of binary classification where the classifier can for a particular cost choose not to classify an observation just as in the conventional classification problem minimization of the sample average of the cost is a difficult optimization problem as an alternative we propose the optimization of a certain convex loss function analogous to the hinge loss used in support vector machines svms its convexity ensures that the sample average of this surrogate loss can be efficiently minimized we study its statistical properties we show that minimizing the expected surrogate loss the risk also minimizes the risk we also study the rate at which the risk approaches its minimum value we show that fast rates are possible when the conditional probability p y 1 x is unlikely to be close to certain critical values

robust model free multiclass probability estimation classical statistical approaches for multiclass probability estimation are typically based on regression techniques such as multiple logistic regression or density estimation approaches such as linear discriminant analysis lda and quadratic discriminant analysis qda these methods often make certain assumptions on the form of probability functions or on the underlying distributions of subclasses in this article we develop a model free procedure to estimate multiclass probabilities based on large margin classifiers in particular the new estimation scheme is employed by solving a series of weighted large margin classifiers and then systematically extracting the probability information from these multiple classification rules a main advantage of the proposed probability estimation technique is that it does not impose any strong parametric assumption on the underlying distribution and can be applied for a wide range of large margin classification methods a general computational algorithm is develop

asymptotic analysis of penalized likelihood and related estimators a general approach to the first order asymptotic analysis ofpenalized likelihood and related estimators is described the method gives expansions for the systematic and random error asymptotic convergence rates in a family of spectral norms are obtained the theory applies to a broad range of function estimation prob erns including non paxametric dellsity hazard and generalized regression curve estimation some examples are provided ams 1980 subject classifications primary 62 g05 secondary 62j05 41 a35 41 a25 47 a53 45 llo 45 m05

on the consistency of ordinal regression methods many of the ordinal regression models that have been proposed in the literature can be seen as methods that minimize a convex surrogate of the zero one absolute or squared loss functions a key property that allows to study the statistical implications of such approximations is that of fisher consistency fisher consistency is a desirable property for surrogate loss functions and implies that in the population setting i e if the probability distribution that generates the data were available then optimization of the surrogate would yield the best possible model in this paper we will characterize the fisher consistency of a rich family of surrogate loss functions used in the context of ordinal regression including support vector ordinal regression orboosting and least absolute deviation we will see that for a family of surrogate loss functions that subsumes support vector ordinal regression and orboosting consistency can be fully characterized by the derivative of a real valued function at zero as happens for convex margin based surrogates in binary classification we also derive excess risk bounds for a surrogate of the absolute error that generalize existing risk bounds for binary classification finally our analysis suggests a novel surrogate of the squared error loss we compare this novel surrogate with competing approaches on 9 different datasets our method shows to be highly competitive in practice outperforming the least squares loss on 7 out of 9 datasets

automatic tagging with existing and novel tags summaryautomatic tagging by key words and phrases is important in multi label classification of a document in this paper we first introduce a tagging loss to measure the discrepancy between predicted and actual tag sets which is expressed in terms of a sum of weighted pairwise margins between two tags by their degree of similarity we then construct a regularized empirical loss to incorporate linguistic knowledge and identify a tagger maximizing the separations between the pairwise margins one salient feature of the proposed method is its capability to identify novel tags absent from a training sample by using their similarity to existing tags computationally the proposed method is implemented by an alternating direction method of multipliers integrated with a difference convex algorithm this permits scalable computation we show that the method achieves accurate tagging and that it compares favourably with existing methods finally we apply the proposed method to tagging a reuters news dataset

support vector hazards machine a counting process framework for learning risk scores for censored outcomes learning risk scores to predict dichotomous or continuous outcomes using machine learning approaches has been studied extensively however how to learn risk scores for time to event outcomes subject to right censoring has received little attention until recently existing approaches rely on inverse probability weighting or rank based regression which may be inefficient in this paper we develop a new support vector hazards machine svhm approach to predict censored outcomes our method is based on predicting the counting process associated with the time to event outcomes among subjects at risk via a series of support vector machines introducing counting processes to represent time to event data leads to a connection between support vector machines in supervised learning and hazards regression in standard survival analysis to account for di erent at risk populations at observed event times a time varying offset is used in estimating risk scores the resulting optimization is a convex quadratic programming problem that can easily incorporate nonlinearity using kernel trick we demonstrate an interesting link from the profiled empirical risk function of svhm to the cox partial likelihood we then formally show that svhm is optimal in discriminating covariate specific hazard function from population average hazard function and establish the consistency and learning rate of the predicted risk using the estimated risk scores simulation studies show improved prediction accuracy of the event times using svhm compared to existing machine learning methods and standard conventional approaches finally we analyze two real world biomedical study data where we use clinical markers and neuroimaging biomarkers to predict age at onset of a disease and demonstrate superiority of svhm in distinguishing high risk versus low risk subjects

using distance covariance for improved variable selection with application to learning genetic risk models variable selection is of increasing importance to address the difficulties of high dimensionality in many scientific areas in this paper we demonstrate a property for distance covariance which is incorporated in a novel feature screening procedure together with the use of distance correlation the approach makes no distributional assumptions for the variables and does not require the specification of a regression model and hence is especially attractive in variable selection given an enormous number of candidate attributes without much information about the true model with the response the method is applied to two genetic risk problems where issues including uncertainty of variable selection via cross validation subgroup of hard to classify cases and the application of a reject option are discussed

feature screening via distance correlation learning this article is concerned with screening features in ultrahigh dimensional data analysis which has become increasingly important in diverse scientific fields we develop a sure independence screening procedure based on the distance correlation dc sis the dc sis can be implemented as easily as the sure independence screening sis procedure based on the pearson correlation proposed by fan and lv however the dc sis can significantly improve the sis fan and lv established the sure screening property for the sis based on linear models but the sure screening property is valid for the dc sis under more general settings including linear models furthermore the implementation of the dc sis does not require model specification e g linear model or generalized linear model for responses or predictors this is a very appealing property in ultrahigh dimensional data analysis moreover the dc sis can be used directly to screen grouped predictor variables and multivariate response variables we establish

robust multicategory support matrix machines we consider the classification problem when the input features are represented as matrices rather than vectors to preserve the intrinsic structures for classification a successful method is the support matrix machine smm in 19 which optimizes an objective function with a hinge loss plus a so called spectral elastic net penalty however the issues of extending smm to multicategory classification still remain moreover in practice it is common to see the training data contaminated by outlying observations which can affect the robustness of existing matrix classification methods in this paper we address these issues by introducing a robust angle based classifier which boils down binary and multicategory problems to a unified framework benefitting from the use of truncated hinge loss functions the proposed classifier achieves certain robustness to outliers the underlying optimization model becomes nonconvex but admits a natural dc difference of two convex functions representation we develop a new and efficient algorithm by incorporating the dc algorithm and primal dual first order methods together the proposed dc algorithm adaptively chooses the accuracy of the subproblem at each iteration while guaranteeing the overall convergence of the algorithm the use of primal dual methods removes a natural complexity of the linear operator in the subproblems and enables us to use the proximal operator of the objective functions and matrix vector operations this advantage allows us to solve large scale problems efficiently theoretical and numerical results indicate that for problems with potential outliers our method can be highly competitive among existing methods

flexible high dimensional classification machines and their asymptotic properties classification is an important topic in statistics and machine learning with great potential in many real applications in this paper we investigate two popular large margin classification methods support vector machine svm and distance weighted discrimination dwd under two contexts the high dimensional low sample size data and the imbalanced data a unified family of classification machines the flexible assortment machine flame is proposed within which dwd and svm are special cases the flame family helps to identify the similarities and differences between svm and dwd it is well known that many classifiers overfit the data in the high dimensional setting and others are sensitive to the imbalanced data that is the class with a larger sample size overly influences the classifier and pushes the decision boundary towards the minority class svm is resistant to the imbalanced data issue but it overfits high dimensional data sets by showing the undesired data piling phenomenon the dwd method was proposed to improve svm in the high dimensional setting but its decision boundary is sensitive to the imbalanced ratio of sample sizes our flame family helps to understand an intrinsic connection between svm and dwd and provides a trade off between sensitivity to the imbalanced data and overfitting the high dimensional data several asymptotic properties of the flame classifiers are studied simulations and real data applications are investigated to illustrate theoretical findings

converse kam theory and practice we unify extend reinterpret and apply criteria of birkhoff 1 herman 9 mather 2 3 aubry et al 4 5 and newman and percival 6 for the nonexistence of invariant circles for area preserving twist maps the criteria enable one to establish regions of phase space through which no rotational invariant circles pass for families of maps the same can be done for regions of the combined space of phase points and parameters the criteria can be implemented rigorously on a computer and give a practical method of proving quite strong results as an example we present a computer program which proved that the standard map has no rotational invariant circles for any parameter value k 63 64

bayesian generalized kernel mixed models we propose a fully bayesian methodology for generalized kernel mixed models gkmms which are extensions of generalized linear mixed models in the feature space induced by a reproducing kernel we place a mixture of a point mass distribution and silverman s g prior on the regression vector of a generalized kernel model gkm this mixture prior allows a fraction of the components of the regression vector to be zero thus it serves for sparse modeling and is useful for bayesian computation in particular we exploit data augmentation methodology to develop a markov chain monte carlo mcmc algorithm in which the reversible jump method is used for model selection and a bayesian model averaging method is used for posterior prediction when the feature basis expansion in the reproducing kernel hilbert space is treated as a stochastic process this approach can be related to the karhunen loeve expansion of a gaussian process gp thus our sparse modeling framework leads to a flexible approximation method for gps

multicategory angle based large margin classification large margin classifiers are popular methods for classification among existing simultaneous multicategory large margin classifiers a common approach is to learn k different functions for a k class problem with a sum to zero constraint such a formulation can be inefficient we propose a new multicategory angle based large margin classification framework the proposed angle based classifiers consider a simplex based prediction rule without the sum to zero constraint and enjoy more efficient computation many binary large margin classifiers can be naturally generalized for multicategory problems through the angle based framework theoretical and numerical studies demonstrate the usefulness of the angle based methods

distance correlation detecting lyapunov instabilities noise induced escape times and mixing abstract the properties of the statistical method of distance correlation between multivariate data are analysed in the context of nonlinear dynamical systems the distance correlation between the noisy and the noiseless quadratic maps are studied in periodic and chaotic regimes results are compared to the classical method of pearson s correlation while distance and pearson s correlations are affected by the lyapunov exponent from the noiseless orbits only the distance correlation is able to recognize the correct qualitative behaviour of noise induced escape times decays and the mixing of chaotic trajectories in addition the distance correlation is capable of detecting distinct size of attractors the main goal of this work is to establish the validity of the distance correlation as an method of correlation between multivariate data in dynamical systems

on l1 norm multiclass support vector machines binary support vector machines svms have been proven to deliver high performance in multiclass classification however issues remain with respect to variable selection one challenging issue is classification and variable selection in the presence of variables in the magnitude of thousands greatly exceeding the size of training sample this often occurs in genomics classification to meet the challenge this article proposes a novel multiclass support vector machine which performs classification and variable selection simultaneously through an l1 norm penalized sparse representation the proposed methodology together with the developed regularization solution path permits variable selection in such a situation for the proposed methodology a statistical learning theory is developed to quantify the generalization error in an attempt to gain insight into the basic structure of sparse learning permitting the number of variables to greatly exceed the sample size the operating characteristics of the

enhancing directed binary trees for multi class classification one approach to multi class classification consists in decomposing the original problem into a collection of binary classification tasks the outputs of these binary classifiers are combined to produce a single prediction winner takes all max wins and tree voting schemes are the most popular methods for this purpose however tree schemes can deliver faster predictions because they need to evaluate less binary models despite previous conclusions reported in the literature this paper shows that their performance depends on the organization of the tree scheme i e the positions where each pairwise classifier is placed on the graph different metrics are studied for this purpose proposing a new one that considers the precision and the complexity of each pairwise model what makes the method to be classifier dependent the study is performed using support vector machines svms as base classifiers but it could be extended to other kind of binary classifiers the proposed method tested on benchmark data sets and on one real world application is able to improve the accuracy of other decomposition multi class classifiers producing even faster predictions

a novel and quick svm based multi class classifier use different real positive numbers p i to represent all kinds of pattern categories after mapping the inputted patterns into a special feature space by a non linear mapping a linear relation between the mapped patterns and numbers p i is assumed whose bias and coefficients are undetermined and the hyper plane corresponding to zero output of the linear relation is looked as the base hyper plane to determine the pending parameters an objective function is founded aiming to minimize the difference between the outputs of the patterns belonging to a same type and the corresponding p i and to maximize the distance between any two different hyper planes corresponding to different pattern types the objective function is same to that of support vector regression in form so the coefficients and bias of the linear relation are calculated by some known methods such as svm l i g h t approach simultaneously three methods are also given to determine p i the best one is to determine them in training process which has relatively high accuracy experiment results of the iris data set show that the accuracy of this method is better than those of many svm based multi class classifiers and close to that of dagsvm decision directed acyclic graph svm emphatically the recognition speed is the highest

the eikonal equation stability of reaction diffusion waves on a sphere abstract the eikonal approach to excitable reaction diffusion systems provides an analytical hold on the phenomenon of wave propagation on curved surfaces and the evolution of toroidal scroll structures in the neighbourhood of organising filaments the extent to which the eikonal equation contains information on the stability of these wave solutions is important from a theoretical and experimental point of view we initiated this study in 1987 by demonstrating the stability of an expanding spherical wave recent experiments on wave propagation on a sphere encouraged us further to investigate stability of waves on surfaces we analyse in this paper the geometrical stability of symmetric counter rotating spiral waves propagating on the unit sphere we demonstrate that these solutions are stable under small perturbations normal to the wave fron lying on the unit sphere

vc theory of large margin multi category classifiers in the context of discriminant analysis vapnik s statistical learning theory has mainly been developed in three directions the computation of dichotomies with binary valued functions the computation of dichotomies with real valued functions and the computation of polytomies with functions taking their values in finite sets typically the set of categories itself the case of classes of vector valued functions used to compute polytomies has seldom been considered independently which is unsatisfactory for three main reasons first this case encompasses the other ones second it cannot be treated appropriately through a naive extension of the results devoted to the computation of dichotomies third most of the classification problems met in practice involve multiple categories in this paper a vc theory of large margin multi category classifiers is introduced central in this theory are generalized vc dimensions called the dimensions first a uniform convergence bound on the risk of the classifiers of interest is derived the capacity measure involved in this bound is a covering number this covering number can be upper bounded in terms of the dimensions thanks to generalizations of sauer s lemma as is illustrated in the specific case of the scale sensitive natarajan dimension a bound on this latter dimension is then computed for the class of functions on which multi class svms are based this makes it possible to apply the structural risk minimization inductive principle to those machines

some theory for generalized boosting algorithms we give a review of various aspects of boosting clarifying the issues through a few simple results and relate our work and that of others to the minimax paradigm of statistics we consider the population version of the boosting algorithm and prove its convergence to the bayes classifier as a corollary of a general result about gauss southwell optimization in hilbert space we then investigate the algorithmic convergence of the sample version and give bounds to the time until perfect separation of the sample we conclude by some results on the statistical optimality of the l2 boosting

singular perturbation theory of traveling waves in excitable media a review abstract waves of chemical or electrical activity traveling through space have been observed in several contexts chemical reaction mixtures in non convecting liquid phase cell suspensions nerve axons and neuromuscular tissues typically wave supporting preparations are excitable that is they respond and sensitivity to perturbations are rapidly damped out but suprathreshold disturbances trigger an abrupt and substantial response the abruptness of the response can be exploited by the methods of singular perturbation theory to obtain a mathematical description of wave propagation in spatially distributed excitable media singular perturbation analysis of propagating waves in one spatial dimension is straightforward and uncontentious but the analysis of propagating waves in one spatial dimension is straightforward and several fundamentally different ways we compare and contrast the approaches taken by greenberg zykov fife krinskii and his collaborators and ourselves with particular emphasis on the case of rotating spiral waves our intention is to bring some order to the important but difficult theory of propagating waves in two dimensional excitable media in conclusion we discuss briefly some possible extensions of the singular perturbation approach to propagating wave surfaces in three dimensional space

a regularization framework for multiclass classification a deterministic annealing approach in this paper we propose a general regularization framework for multiclass classification based on discriminant functions since the objective function in the primal optimization problem of this framework is always not differentiable the optimal solution cannot be obtained directly with the aid of the deterministic annealing approach a differentiable objective function is derived subject to a constraint on the randomness of the solution the problem can be approximated by solving a sequence of differentiable optimization problems and such approximation converges to the original problem asymptotically based on this approach class conditional posterior probabilities can be calculated directly without assuming the underlying probabilistic model we also notice that there is a connection between our approach and some existing statistical models such as fisher discriminant analysis and logistic regression

total destruction of invariant tori for the generalized frenkel kontorova model we consider generalized frenkel kontorova models on higher dimensional lattices we show that the invariant tori which are parameterized by continuous hull functions can be destroyed by small perturbations in the cr topology with r 1

on the influence of the kernel on the consistency of support vector machines in this article we study the generalization abilities of several classifiers of support vector machine svm type using a certain class of kernels that we call universal it is shown that the soft margin algorithms with universal kernels are consistent for a large class of classification problems including some kind of noisy tasks provided that the regularization parameter is chosen well in particular we derive a simple sufficient condition for this parameter in the case of gaussian rbf kernels on the one hand our considerations are based on an investigation of an approximation property the so called universality of the used kernels that ensures that all continuous functions can be approximated by certain kernel expressions this approximation property also gives a new insight into the role of kernels in these and other algorithms on the other hand the results are achieved by a precise study of the underlying optimization problems of the classifiers furthermore we show consistency for the maximal margin classifier as well as for the soft margin svm s in the presence of large margins in this case it turns out that also constant regularization parameters ensure consistency for the soft margin svm s finally we prove that even for simple noise free classification problems svm s with polynomial kernels can behave arbitrarily badly