boosting the margin a new explanation for the effectiveness of voting methods one of the surprising recurring phenomena observed in experiments with boosting is that the test error of the generated classifier usually does not increase as its size becomes very large and often is observed to decrease even after the training error reaches zero in this paper we show that this phenomenon is related to the distribution of margins of the training examples with respect to the generated voting classification rule where the margin of an example is simply the difference between the number of correct votes and the maximum number of votes received by any incorrect label we show that techniques used in the analysis of vapnik s support vector classifiers and of neural networks with small weights can be applied to voting methods to relate the margin distribution to the test error we also show theoretically and experimentally that boosting is especially effective at increasing the margins of the training examples finally we compare our explanation to those based on the bias variance decomposition

controllability of heterogeneous multi agent systems under directed and weighted topology abstractthis paper considers the controllability problem for both continuous and discrete time linear heterogeneous multi agent systems with directed and weighted communication topology first two kinds of neighbour based control protocols based on the distributed protocol of first order and second order multi agent systems are proposed under which it is shown that a heterogeneous multi agent system is controllable if the underlying communication topology is controllable then under special leader selection the result shows that the controllability of a heterogeneous multi agent system is solely decided by its communication topology graph furthermore some necessary and or sufficient conditions are derived for controllability of communication topology from algebraic and graphical perspectives finally simulation examples are presented to demonstrate the effectiveness of the theoretical results

kernel methods in machine learning we review machine learning methods employing positive definite kernels these methods formulate learning and estimation problems in a reproducing kernel hilbert space rkhs of functions defined on the data domain expanded in terms of a kernel working in linear spaces of function has the benefit of facilitating the construction and analysis of learning algorithms while at the same time allowing large classes of functions the latter include nonlinear functions as well as functions defined on nonvectorial data we cover a wide range of methods ranging from binary classifiers to sophisticated methods for estimation with structured data

input addition and leader selection for the controllability of graph based systems in this paper we consider dynamical graph based models which are well fitted for the structural analysis of complex systems a significant amount of work has been devoted to the controllability of such graph based models e g recently for multi agent systems or complex networks we study here the controllability through input addition in this framework we present several variants of this problem depending on the freedom which is left to the designer on the additional inputs we use a unified framework which allows us to encompass the different applications and representations large scale systems complex communications networks multi agent systems and provide convenient graph tools for their analysis our contribution is to characterize the structural modifications of the system resulting from an input addition or a leader selection and of the mechanisms which lead to controllability we provide information on the possible location of additional inputs and on the minimal number of inputs to be added for controllability

training ν support vector classifiers theory and algorithms the support vector machine svm for classification proposed by scholkopf smola williamson and bartlett 2000 has the advantage of using a parameter on controlling the number of support vectors in this article we investigate the relation between svm and c svm in detail we show that in general they are two different problems with the same optimal solution set hence we may expect that many numerical aspects of solving them are similar however compared to regular c svm the formulation of svm is more complicated so up to now there have been no effective methods for solving large scale svm we propose a decomposition method for svm that is competitive with existing methods for c svm we also discuss the behavior of svm by some numerical experiments

a unified formulation and fast accelerated proximal gradient method for classification binary classification is the problem of predicting the class a given sample belongs to to achieve a good prediction performance it is important to find a suitable model for a given dataset however it is often time consuming and impractical for practitioners to try various classification models because each model employs a different formulation and algorithm the difficulty can be mitigated if we have a unified formulation and an efficient universal algorithmic framework for various classification models to expedite the comparison of performance of different models for a given dataset in this paper we present a unified formulation of various classification models including c svm l2 svm svm mm fda mm mpm logistic regression distance weighted discrimination and develop a general optimization algorithm based on an accelerated proximal gradient apg method for the formulation we design various techniques such as backtracking line search and adaptive restarting strategy in order to speed up the practical convergence of our method we also give a theoretical convergence guarantee for the proposed fast apg algorithm numerical experiments show that our algorithm is stable and highly competitive to specialized algorithms designed for specific models e g sequential minimal optimization smo for svm

convex calibration dimension for multiclass loss matrices we study consistency properties of surrogate loss functions for general multiclass learning problems defined by a general multiclass loss matrix we extend the notion of classification calibration which has been studied for binary and multiclass 0 1 classification problems and for certain other specific learning problems to the general multiclass setting and derive necessary and sufficient conditions for a surrogate loss to be calibrated with respect to a loss matrix in this setting we then introduce the notion of convex calibration dimension of a multiclass loss matrix which measures the smallest size of a prediction space in which it is possible to design a convex surrogate that is calibrated with respect to the loss matrix we derive both upper and lower bounds on this quantity and use these results to analyze various loss matrices in particular we apply our framework to study various subset ranking losses and use the convex calibration dimension as a tool to show both the existence and non existence of various types of convex calibrated surrogates for these losses our results strengthen recent results of duchi et al 2010 and calauzenes et al 2012 on the non existence of certain types of convex calibrated surrogates in subset ranking we anticipate the convex calibration dimension may prove to be a useful tool in the study and design of surrogate losses for general multiclass learning problems

improving the sample complexity using global data we study the sample complexity of proper and improper learning problems with respect to different q loss functions we improve the known estimates for classes which have relatively small covering numbers in empirical l sub 2 spaces e g log covering numbers which are polynomial with exponent p 2 we present several examples of relevant classes which have a small fat shattering dimension and hence fit our setup the most important of which are kernel machines

multicategory ψ learning in binary classification margin based techniques usually deliver high performance as a result a multicategory problem is often treated as a sequence of binary classifications in the absence of a dominating class this treatment may be suboptimal and may yield poor performance such as for support vector machines svms we propose a novel multicategory generalization of learning that treats all classes simultaneously the new generalization eliminates this potential problem while at the same time retaining the desirable properties of its binary counterpart we develop a statistical learning theory for the proposed methodology and obtain fast convergence rates for both linear and nonlinear learning examples we demonstrate the operational characteristics of this method through a simulation our results indicate that the proposed methodology can deliver accurate class prediction and is more robust against extreme observations than its svm counterpart

nodes selection strategy in cooperative tracking problem in this paper a new optimization problem is addressed for node selection that has application potentials in input output switches for sensors in control system design and leader determination in social networks the purpose of the addressed problem is to develop a strategy for selecting a subset of nodes as controlled nodes in order to minimize certain objective function consisting of the convergence speed and the energy of control action over a finite time horizon for networks with fixed controlled nodes an upper bound of the objective function is obtained which is shown to be convex and independent of the time horizon for networks with switched controlled nodes a greedy algorithm is proposed to reduce the computation complexity resulting from the length of the time horizon where the nodes selection is carried out over divided small time intervals the cost gap is also analyzed between the strategy of optimizing over the whole time horizon and the strategy of optimizing over the small intervals finally the proposed nodes selection strategy is validated through simulations and two regions are found in which the number of optimal controlled nodes is determined

controllability of multiplex multi time scale networks the paradigm of layered networks is used to describe many real world systems from biological networks to social organizations and transportation systems while recently there has been much progress in understanding the general properties of multilayer networks our understanding of how to control such systems remains limited one fundamental aspect that makes this endeavor challenging is that each layer can operate at a different time scale thus we cannot directly apply standard ideas from structural control theory of individual networks here we address the problem of controlling multilayer and multi time scale networks focusing on two layer multiplex networks with one to one interlayer coupling we investigate the practically relevant case when the control signal is applied to the nodes of one layer we develop a theory based on disjoint path covers to determine the minimum number of inputs n i necessary for full control we show that if both layers operate on the same time scale then the network structure of both layers equally affect controllability in the presence of time scale separation controllability is enhanced if the controller interacts with the faster layer n i decreases as the time scale difference increases up to a critical time scale difference above which n i remains constant and is completely determined by the faster layer we show that the critical time scale difference is large if layer i is easy and layer ii is hard to control in isolation in contrast control becomes increasingly difficult if the controller interacts with the layer operating on the slower time scale and increasing time scale separation leads to increased n i again up to a critical value above which n i still depends on the structure of both layers this critical value is largely determined by the longest path in the faster layer that does not involve cycles by identifying the underlying mechanisms that connect time scale difference and controllability for a simplified model we provide crucial insight into disentangling how our ability to control real interacting complex systems is affected by a variety of sources of complexity

efficient agnostic learning of neural networks with bounded fan in we show that the class of two layer neural networks with bounded fan in is efficiently learnable in a realistic extension to the probably approximately correct pac learning model in this model a joint probability distribution is assumed to exist on the observations and the learner is required to approximate the neural network which minimizes the expected quadratic error as special cases the model allows learning real valued functions with bounded noise learning probabilistic concepts and learning the best approximation to a target function that cannot be well approximated by the neural network the networks we consider have real valued inputs and outputs an unlimited number of threshold hidden units with bounded fan in and a bound on the sum of the absolute values of the output weights the number of computation steps of the learning algorithm is bounded by a polynomial in 1 spl epsiv 1 spl delta n and b where spl epsiv is the desired accuracy spl delta is the probability that the algorithm fails n is the input dimension and b is the bound on both the absolute value of the target which may be a random variable and the sum of the absolute values of the output weights in obtaining the result we also extended some results on iterative approximation of functions in the closure of the convex hull of a function class and on the sample complexity of agnostic learning with the quadratic loss function

optimal aggregation of classifiers in statistical learning classification can be considered as nonparametric estimation of sets where the risk is defined by means of a specific distance between sets associated with misclassification error it is shown that the rates of convergence of classifiers depend on two parameters the complexity of the class of candidate sets and the margin parameter the dependence is explicitly given indicating that optimal fast rates approaching o n 1 can be attained where n is the sample size and that the proposed classifiers have the property of robustness to the margin the main result of the paper concerns optimal aggregation of classifiers we suggest a classifier that automatically adapts both to the complexity and to the margin and attains the optimal fast rates up to a logarithmic factor

smooth discrimination analysis ties f and g can be based on the estimation of the set g x fx gx we consider applications where it is appropriate to assume that the region g has a smooth boundary or belongs to another nonparametric class of sets in particular this assumption makes sense if discrimination is used as a data analytic tool decision rules based on minimization of empirical risk over the whole class of sets and over sieves are considered their rates of convergence are obtained we show that these rules achieve optimal rates for estimation of g and optimal rates of convergence for bayes risks an interesting conclusion is that the optimal rates for bayes risks can be very fast in particular faster than the parametric root n rate these fast rates cannot be guaranteed for plug in rules

survey generic properties and control of linear structured systems a survey in this survey paper we consider linear structured systems in state space form where a linear system is structured when each entry of its matrices like a b c and d is either a fixed zero or a free parameter the location of the fixed zeros in these matrices constitutes the structure of the system indeed a lot of man made physical systems which admit a linear model are structured a structured system is representative of a class of linear systems in the usual sense it is of interest to investigate properties of structured systems which are true for almost any value of the free parameters therefore also called generic properties interestingly a lot of classical properties of linear systems can be studied in terms of genericity moreover these generic properties can in general be checked by means of directed graphs that can be associated to a structured system in a natural way we review here a number of results concerning generic properties of structured systems expressed in graph theoretic terms by properties we mean here system specific properties like controllability the finite and infinite zero structure and so on as well as solvability issues of certain classical control problems like disturbance rejection input output decoupling and so on in this paper we do not try to be exhaustive but instead by a selection of results we would like to motivate the reader to appreciate what we consider as a wonderful modelling and analysis tool we emphasize the fact that this modelling technique allows us to get a number of important results based on poor information on the system only moreover the graph theoretic conditions are intuitive and are easy to check by hand for small systems and by means of well known polynomially bounded combinatorial techniques for larger systems

twice punctured hyperbolic sphere with a conical singularity and generalized elliptic integral we describe in terms of generalized elliptic integrals the hyperbolic metric of the twice punctured sphere with one conical singularity of prescribed order we also give several monotonicity properties of the metric and a couple of applications

robustness and regularization of support vector machines we consider regularized support vector machines svms and show that they are precisely equivalent to a new robust optimization formulation we show that this equivalence of robust optimization and regularization has implications for both algorithms and analysis in terms of algorithms the equivalence suggests more general svm like algorithms for classification that explicitly build in protection to noise and at the same time control overfitting on the analysis front the equivalence of robustness and regularization provides a robust optimization interpretation for the success of regularized svms we use this new robustness interpretation of svms to give a new proof of consistency of kernelized svms thus establishing robustness as the reason regularized svms generalize well

on p norm linear discrimination abstract we consider a p norm linear discrimination model that generalizes the model of bennett and mangasarian 1992 and reduces to a linear programming problem with p order cone constraints the proposed approach for handling linear programming problems with p order cone constraints is based on reformulation of p order cone optimization problems as second order cone programming socp problems when p is rational since such reformulations typically lead to socp problems with large numbers of second order cones an economical representation that minimizes the number of second order cones is proposed a case study illustrating the developed model on several popular data sets is conducted

robust classification and regression using support vector machines abstract in this paper we investigate the theoretical aspects of robust classification and robust regression using support vector machines given training data x 1 y 1 x l y l where l represents the number of samples x i r n and y i 1 1 for classification or y i r for regression we investigate the training of a support vector machine in the case where bounded perturbation is added to the value of the input x i r n we consider both cases where our training data are either linearly separable and nonlinearly separable respectively we show that we can perform robust classification or regression by using linear or second order cone programming

classification using geometric level sets a variational level set method is developed for the supervised classification problem nonlinear classifier decision boundaries are obtained by minimizing an energy functional that is composed of an empirical risk term with a margin based loss and a geometric regularization term new to machine learning the surface area of the decision boundary this geometric level set classifier is analyzed in terms of consistency and complexity through the calculation of its e entropy for multicategory classification an efficient scheme is developed using a logarithmic number of decision functions in the number of classes rather than the typical linear number of decision functions geometric level set classification yields performance results on benchmark data sets that are competitive with well established methods

distance based margin support vector machine for classification recently the development of machine learning techniques has provided an effective analysis tool for classification problems support vector machine svm is one of the most popular supervised learning techniques however svm may not effectively detect the instance of the minority class and obtain a lower classification performance in the overlap region when learning from complicated data sets complicated data sets with imbalanced and overlapping class distributions are common in most practical applications moreover they negatively affect the classification performances of the svm the present study proposes the use of modified slack variables within the svm ms svm to solve complex data problems including class imbalance and overlapping artificial and uci data sets are provided to evaluate the effectiveness of the ms svm model experimental results indicate that the ms svm performed better than the other methods in terms of accuracy sensitivity and specificity in addition the proposed ms svm is a robust approach for solving different levels of complex data sets

geometric intuition and algorithms for ev svm in this work we address the ev svm model proposed by perez cruz et al as an extension of the traditional v support vector classification model v svm through an enhancement of the range of admissible values for the regularization parameter v the ev svm has been shown to be able to produce a wider variety of decision functions giving rise to a better adaptability to the data however while a clear and intuitive geometric interpretation can be given for the v svm model as a nearest point problem in reduced convex hulls rch npp no previous work has been made in developing such intuition for the ev svm model in this paper we show how ev svm can be reformulated as a geometrical problem that generalizes rch npp providing new insights into this model under this novel point of view we propose the rapminos algorithm able to solve ev svm more efficiently than the current methods furthermore we show how rapminos is able to address the ev svm model for any choice of regularization norm lp 1 seamlessly which further extends the svm model flexibility beyond the usual ev svm models

statistical behavior and consistency of classification methods based on convex risk minimization we study how closely the optimal bayes error rate can be approximately reached using a classification algorithm that computes a classifier by minimizing a convex upper bound of the classification error function the measurement of closeness is characterized by the loss function used in the estimation we show that such a classification scheme can be generally regarded as a nonmaximum likelihood conditional in class probability estimate and we use this analysis to compare various convex loss functions that have appeared in the literature furthermore the theoretical insight allows us to design good loss functions with desirable properties another aspect of our analysis is to demonstrate the consistency of certain classification methods using convex risk minimization this study sheds light on the good performance of some recently proposed linear classification methods including boosting and support vector machines it also shows their limitations and suggests possible improvements

on the bayes risk consistency of regularized boosting methods the probability of error of classification methods based on convex combinations of simple base classifiers by boosting algorithms is investigated the main result of the paper is that certain regularized boosting algorithms provide bayes risk consistent classifiers under the sole assumption that the bayes classifier may be approximated by a convex combination of the base classifiers nonasymptotic distribution free bounds are also developed which offer interesting new insight into how boosting works and help explain its success in practical classification problems

empirical margin distributions and bounding the generalization error of combined classifiers we prove new probabilistic upper bounds on generalization error of complex classifiers that are combinations of simple classifiers such combinations could be implemented by neural networks or by voting methods of combining the classifiers such as boosting and bagging the bounds are in terms of the empirical distribution of the margin of the combined classifier they are based on the methods of the theory of gaussian and empirical processes comparison inequalities symmetrization method concentration inequalities and they improve previous results of bartlett 1998 on bounding the generalization error of neural networks in terms of ell 1 norms of the weights of neurons and of schapire freund bartlett and lee 1998 on bounding the generalization error of boosting we also obtain rates of convergence in levy distance of empirical margin distribution to the true margin distribution uniformly over the classes of classifiers and prove the optimality of these rates

robust convex optimization we study convex optimization problems for which the data is not specified exactly and it is only known to belong to a given uncertainty set u yet the constraints must hold for all possible values of the data from u the ensuing optimization problem is called robust optimization in this paper we lay the foundation of robust convex optimization in the main part of the paper we show that if u is an ellipsoidal uncertainty set then for some of the most important generic convex optimization problems linear programming quadratically constrained programming semidefinite programming and others the corresponding robust convex program is either exactly or approximately a tractable problem which lends itself to efficientalgorithms such as polynomial time interior point methods

convexity classification and risk bounds many of the classification algorithms developed in the machine learning literature including the support vector machine and boosting can be viewed as minimum contrast methods that minimize a convex surrogate of the 0 1 loss function the convexity makes these algorithms computationally efficient the use of a surrogate however has statistical consequences that must be balanced against the computational virtues of convexity to study these issues we provide a general quantitative relationship between the risk as assessed using the 0 1 loss and the risk as assessed using any nonnegative surrogate loss function we show that this relationship gives nontrivial upper bounds on excess risk under the weakest possible condition on the loss function that it satisfies a pointwise form of fisher consistency for classification the relationship is based on a simple variational transformation of the loss function that is easy to compute in many applications we also present a refined version of this result in the

on bregman distances and divergences of probability measures this paper introduces scaled bregman distances of probability distributions which admit nonuniform contributions of observed events they are introduced in a general form covering not only the distances of discrete and continuous stochastic observations but also the distances of random processes and signals it is shown that the scaled bregman distances extend not only the classical ones studied in the previous literature but also the information divergence and the related wider class of convex divergences of probability measures an information processing theorem is established too but only in the sense of invariance w r t statistically sufficient transformations and not in the sense of universal monotonicity pathological situations where coding can increase the classical bregman distance are illustrated by a concrete example in addition to the classical areas of application of the bregman distances and convex divergences such as recognition classification learning and evaluation of proximity of various features and signals the paper mentions a new application in 3 d exploratory data analysis explicit expressions for the scaled bregman distances are obtained in general exponential families with concrete applications in the binomial poisson and rayleigh families and in the families of exponential processes such as the poisson and diffusion processes including the classical examples of the wiener process and geometric brownian motion

a geometric approach to support vector machine svm classification the geometric framework for the support vector machine svm classification problem provides an intuitive ground for the understanding and the application of geometric optimization algorithms leading to practical solutions of real world classification problems in this work the notion of reduced convex hull is employed and supported by a set of new theoretical results these results allow existing geometric algorithms to be directly and practically applied to solve not only separable but also nonseparable classification problems both accurately and efficiently as a practical application of the new theoretical results a known geometric algorithm has been employed and transformed accordingly to solve nonseparable problems successfully

minimum cost input output design for large scale linear structural systems in this paper we provide optimal solutions to two different but related input output design problems involving large scale linear dynamical systems where the cost associated to each directly actuated measured state variable can take different values but is independent of the input output performing the task under these conditions we first aim to determine and characterize the input output placement that incurs in the minimum cost while ensuring that the resulting placement achieves structural controllability observability further we address a constrained variant of the above problem in which we seek to determine the minimum cost placement configuration among all possible input output placement configurations that ensures structural controllability observability with the lowest number of directly actuated measured state variables we develop new graph theoretical characterizations of cost constrained input selections for structural controllability and properties that enable us to address both problems by reduction to a weighted maximum matching problem efficiently addressed by algorithms with polynomial time complexity in the number of state variables finally we illustrate the obtained results with an example

conjugate relation between loss functions and uncertainty sets in classification problems there are two main approaches to binary classification problems the loss function approach and the uncertainty set approach the loss function approach is widely used in real world data analysis statistical decision theory has been used to elucidate its properties such as statistical consistency conditional probabilities can also be estimated by using the minimum solution of the loss function in the uncertainty set approach an uncertainty set is defined for each binary label from training samples the best separating hyperplane between the two uncertainty sets is used as the decision function although the uncertainty set approach provides an intuitive understanding of learning algorithms its statistical properties have not been sufficiently studied in this paper we show that the uncertainty set is deeply connected with the convex conjugate of a loss function on the basis of the conjugate relation we propose a way of revising the uncertainty set approach so that it will have good statistical properties such as statistical consistency we also introduce statistical models corresponding to uncertainty sets in order to estimate conditional probabilities finally we present numerical experiments verifying that the learning with revised uncertainty sets improves the prediction accuracy

training with noise is equivalent to tikhonov regularization it is well known that the addition of noise to the input data of a neural network during training can in some circumstances lead to significant improvements in generalization performance previous work has shown that such training with noise is equivalent to a form of regularization in which an extra term is added to the error function however the regularization term which involves second derivatives of the error function is not bounded below and so can lead to difficulties if used directly in a learning algorithm based on error minimization in this paper we show that for the purposes of network training the regularization term can be reduced to a positive semi definite form that involves only first derivatives of the network mapping for a sum of squares error function the regularization term belongs to the class of generalized tikhonov regularizers direct minimization of the regularized error function provides a practical alternative to training with noise

composite binary losses we study losses for binary classification and class probability estimation and extend the understanding of them from margin losses to general composite losses which are the composition of a proper loss with a link function we characterise when margin losses can be proper composite losses explicitly show how to determine a symmetric loss in full from half of one of its partial losses introduce an intrinsic parametrisation of composite binary losses and give a complete characterisation of the relationship between proper losses and classification calibrated losses we also consider the question of the best surrogate binary loss we introduce a precise notion of best and show there exist situations where two convex surrogate losses are incommensurable we provide a complete explicit characterisation of the convexity of composite binary losses in terms of the link function and the weight function associated with the proper loss which make up the composite loss this characterisation suggests new ways of surrogate tuning as well as providing an explicit characterisation of when bregman divergences on the unit interval are convex in their second argument finally in an appendix we present some new algorithm independent results on the relationship between properness convexity and robustness to misclassification noise for binary losses and show that all convex proper losses are non robust to misclassification noise

a robust approach based on conditional value at risk measure to statistical learning problems in statistical learning problems measurement errors in the observed data degrade the reliability of estimation there exist several approaches to handle those uncertainties in observations in this paper we propose to use the conditional value at risk cvar measure in order to depress influence of measurement errors and investigate the relation between the resulting cvar minimization problems and some existing approaches in the same framework for the cvar minimization problems which include the computation of integration we apply monte carlo sampling method and obtain their approximate solutions the approximation error bound and convergence property of the solution are proved by vapnik and chervonenkis theory numerical experiments show that the cvar minimization problem can achieve fairly good estimation results compared with several support vector machines in the presence of measurement errors

applications of second order cone programming in a second order cone program socp a linear function is minimized over the intersection of an affine set and the product of second order quadratic cones socps are nonlinear convex problems that include linear and convex quadratic programs as special cases but are less general than semidefinite programs sdps several efficient primaldual interior point methods for socp have been developed in the last few years after reviewing the basic theory of socps we describe general families of problems that tan be recast as socps these include robust linear programming and robust leastsquares problems problems involving sums or maxima of norms or with convex hyperbolic constraints we discuss a variety of engineering applications such as filter design antenna array weight design truss design and grasping forte optimization in robotics we describe an efficient primaldual interior point method for solving socps which shares many of the features of primaldual interior point methods for linear program

prescribing curvature on open surfaces in this article we study the problem sometimes called the berger nirenberg problem of prescribing the curvature on a riemann surface that is on an oriented surface equipped with a conformal class of riemannian metrics

process consistency for adaboost recent experiments and theoretical studies show that adaboost can overfit in the limit of large time if running the algorithm forever is suboptimal a natural question is how low can the prediction error be during the process of adaboost we show under general regularity conditions that during the process of adaboost a consistent prediction is generated which has the prediction error approximating the optimal bayes error as the sample size increases this result suggests that while running the algorithm forever can be suboptimal it is reasonable to expect that some regularization method via truncation of the process may lead to a near optimal performance for sufficiently large sample size

additive logistic regression a statistical view of boosting boosting is one of the most important recent developments in classification methodology boosting works by sequentially applying a classification algorithm to reweighted versions of the training data and then taking a weighted majority vote of the sequence of classifiers thus produced for many classification algorithms this simple strategy results in dramatic improvements in performance we show that this seemingly mysterious phenomenon can be understood in terms of well known statistical principles namely additive modeling and maximum likelihood for the two class problem boosting can be viewed as an approximation to additive modeling on the logistic scale using maximum bernoulli likelihood as a criterion we develop more direct approximations and show that they exhibit nearly identical results to boosting direct multiclass generalizations based on multinomial likelihood are derived that exhibit performance comparable to other recently proposed multiclass generalizations of boosting in most situations and far superior in some we suggest a minor modification to boosting that can reduce computation often by factors of 10 to 50 finally we apply these insights to produce an alternative formulation of boosting decision trees this approach based on best first truncated tree induction often leads to better performance and can provide interpretable descriptions of the aggregate decision rule it is also much faster computationally making it more suitable to large scale data mining applications

new support vector algorithms we propose a new class of support vector algorithms for regression and classification in these algorithms a parameter lets one effectively control the number of support vectors while this can be useful in its own right the parameterization has the additional benefit of enabling us to eliminate one of the other free parameters of the algorithm the accuracy parameter epsilon in the regression case and the regularization constant c in the classification case we describe the algorithms give some theoretical results concerning the meaning and the choice of and report experimental results

a unified classification model based on robust optimization a wide variety of machine learning algorithms such as the support vector machine svm minimax probability machine mpm and fisher discriminant analysis fda exist for binary classification the purpose of this letter is to provide a unified classification model that includes these models through a robust optimization approach this unified model has several benefits one is that the extensions and improvements intended for svms become applicable to mpm and fda and vice versa for example we can obtain nonconvex variants of mpm and fda by mimicking perez cruz weston hermann and scholkopf s 2003 extension from convex i svm to nonconvex ei svm another benefit is to provide theoretical results concerning these learning methods at once by dealing with the unified model we give a statistical interpretation of the unified classification model and prove that the model is a good approximation for the worst case minimization of an expected loss with respect to the uncertain probability distribution we also propose a nonconvex optimization algorithm that can be applied to nonconvex variants of existing learning methods and show promising numerical results

mixed integer second order cone programming formulations for variable selection in linear regression abstract this study concerns a method of selecting the best subset of explanatory variables in a multiple linear regression model goodness of fit measures for example adjusted r2 aic and bic are generally used to evaluate a subset regression model although variable selection with regard to these measures is usually performed with a stepwise regression method it does not always provide the best subset of explanatory variables in this paper we propose mixed integer second order cone programming formulations for selecting the best subset of variables with respect to adjusted r2 aic and bic computational experiments show that in terms of these measures the proposed formulations yield better solutions than those provided by common stepwise regression methods

second order cone programming formulations for feature selection this paper addresses the issue of feature selection for linear classifiers given the moments of the class conditional densities the problem is posed as finding a minimal set of features such that the resulting classifier has a low misclassification error using a bound on the misclassification error involving the mean and covariance of class conditional densities and minimizing an l1 norm as an approximate criterion for feature selection a second order programming formulation is derived to handle errors in estimation of mean and covariances a tractable robust formulation is also discussed in a slightly different setting the fisher discriminant is derived feature selection for fisher discriminant is also discussed experimental results on synthetic data sets and on real life microarray data show that the proposed formulations are competitive with the state of the art linear programming formulation

observability preservation under sensor failure this paper is concerned with the study of observability in a structural framework it turns out that the system is structurally observable if and only if the system is output connected and contains no contraction we focus our attention on the observability preservation under sensor failure we consider linear observable systems and we wonder if a given system remains observable in case of sensor failure more precisely we will characterize among the sensors those which are critical i e which failure leads to observability loss those which are useless for observability purpose and the set of those which are useful without being critical using a graph approach we classify the sensors with respect to their importance for output connection preservation contraction avoidance and then observability preservation under sensor failure

a robust minimax approach to classification when constructing a classifier the probability of correct classification of future data points should be maximized we consider a binary classification problem where the mean and covariance matrix of each class are assumed to be known no further assumptions are made with respect to the class conditional distributions misclassification probabilities are then controlled in a worst case setting that is under all possible choices of class conditional densities with given mean and covariance matrix we minimize the worst case maximum probability of misclassification of future data points for a linear decision boundary this desideratum is translated in a very direct way into a convex second order cone optimization problem with complexity similar to a support vector machine problem the minimax problem can be interpreted geometrically as minimizing the maximum of the mahalanobis distances to the two classes we address the issue of robustness with respect to estimation errors in the means and covariances of the classes via a simple modification of the input data we also show how to exploit mercer kernels in this setting to obtain nonlinear decision boundaries yielding a classifier which proves to be competitive with current methods including support vector machines an important feature of this method is that a worst case bound on the probability of misclassification of future data is always obtained explicitly

brief paper leader localization in multi agent systems subject to failure a graph theoretic approach in this paper structural controllability of a leader follower multi agent system with multiple leaders is studied a graphical condition for structural controllability based on the information flow graph of the system is provided the notions of p link and q agent controllability in a multi leader setting are then introduced which provide quantitative measures for the controllability of a system subject to failure in the agents and communication links the problem of leader localization is introduced which is concerned with finding the minimum number of agents whose selection as leaders results in a p link or q agent controllable network polynomial time algorithms are subsequently presented to solve the problem for both cases of undirected and directed information flow graphs

universally consistent vertex classification for latent positions graphs in this work we show that using the eigen decomposition of the adjacency matrix we can consistently estimate feature maps for latent position graphs with positive definite link function provided that the latent positions are i i d from some distribution f we then consider the exploitation task of vertex classification where the link

consistency of support vector machines and other regularized kernel classifiers it is shown that various classifiers that are based on minimization of a regularized risk are universally consistent i e they can asymptotically learn in every classification task the role of the loss functions used in these algorithms is considered in detail as an application of our general framework several types of support vector machines svms as well as regularization networks are treated our methods combine techniques from stochastics approximation theory and functional analysis

distance weighted discrimination high dimension low sample size statistical analysis is becoming increasingly important in a wide range of applied contexts in such situations the popular support vector machine suffers from data piling at the margin which can diminish generalizability this leads naturally to the development of distance weighted discrimination which is based on second order cone programming a modern computationally intensive optimization method

on the complexity of the constrained input selection problem for structural linear systems this paper studies the problem of given the structure of a linear time invariant system and a set of possible inputs finding the smallest subset of input vectors that ensures system s structural controllability we refer to this problem as the minimum constrained input selection mincis problem since the selection has to be performed on an initial given set of possible inputs we prove that the mincis problem is np hard which addresses a recent open question of whether there exist polynomial algorithms in the size of the system plant matrices that solve the mincis problem to this end we show that the associated decision problem to be referred to as the cis of determining whether a subset of a given collection of inputs with a prescribed cardinality exists that ensures structural controllability is np complete further we explore in detail practically important subclasses of the mincis obtained by introducing more specific assumptions either on the system dynamics or the input set instances for which systematic solution methods are provided by constructing explicit reductions to well known computational problems the analytical findings are illustrated through examples in multi agent leader follower type control problems

of quantiles and expectiles consistent scoring functions choquet representations and forecast rankings summary in the practice of point prediction it is desirable that forecasters receive a directive in the form of a statistical functional for example forecasters might be asked to report the mean or a quantile of their predictive distributions when evaluating and comparing competing forecasts it is then critical that the scoring function used for these purposes be consistent for the functional at hand in the sense that the expected score is minimized when following the directive we show that any scoring function that is consistent for a quantile or an expectile functional can be represented as a mixture of elementary or extremal scoring functions that form a linearly parameterized family scoring functions for the mean value and probability forecasts of binary events constitute important examples the extremal scoring functions admit appealing economic interpretations of quantiles and expectiles in the context of betting and investment problems the choquet type mixture representations give rise to simple checks of whether a forecast dominates another in the sense that it is preferable under any consistent scoring function in empirical settings it suffices to compare the average scores for only a finite number of extremal elements plots of the average scores with respect to the extremal scoring functions which we call murphy diagrams permit detailed comparisons of the relative merits of competing forecasts

robust support vector machines for classification with nonconvex and smooth losses this letter addresses the robustness problem when learning a large margin classifier in the presence of label noise in our study we achieve this purpose by proposing robustified large margin support vector machines the robustness of the proposed robust support vector classifiers rsvc which is interpreted from a weighted viewpoint in this work is due to the use of nonconvex classification losses besides the robustness we also show that the proposed rscv is simultaneously smooth which again benefits from using smooth classification losses the idea of proposing rsvc comes from m estimation in statistics since the proposed robust and smooth classification losses can be taken as one sided cost functions in robust statistics its fisher consistency property and generalization ability are also investigated besides the robustness and smoothness another nice property of rsvc lies in the fact that its solution can be obtained by solving weighted squared hinge loss based support vector machine problems iteratively we further show that in each iteration it is a quadratic programming problem in its dual space and can be solved by using state of the art methods we thus propose an iteratively reweighted type algorithm and provide a constructive proof of its convergence to a stationary point effectiveness of the proposed classifiers is verified on both artificial and real data sets

the single input minimal controllability problem for structured systems abstract this paper considers the minimal controllability problem mcp i e the problem of controlling a linear system with an input vector having as few non zero entries as possible we focus on structured systems which represent an interesting class of parameter dependent linear systems and look for structural controllability properties based on the sparsity pattern of the input vector we show first that the mcp is solvable when a rank condition is satisfied and show that generically one non zero entry in the input vector is sufficient to achieve controllability when there is no specific system structure according to the fixed zero non zero pattern of the state matrix entries we give an explicit characterization of the minimum number and the possible location of non zero entries in the input vector to ensure generic controllability the analysis based on graph tools provides with a simple polynomial mcp solution and highlights the structural mechanisms that make it useful to act on some variables to ensure controllability

moduli spaces of weighted pointed stable curves abstract a weighted pointed curve consists of a nodal curve and a sequence of marked smooth points each assigned a number between zero and one a subset of the marked points may coincide if the sum of the corresponding weights is no greater than one we construct moduli spaces for these objects using methods of the log minimal model program and describe the induced birational morphisms between moduli spaces as the weights are varied in the genus zero case we explain the connection to geometric invariant theory quotients of points in the projective line and to compactifications of moduli spaces studied by kapranov keel and losev manin

on the controllability and observability of networked dynamic systems some necessary and sufficient conditions are obtained for the controllability and observability of a networked system with linear time invariant lti dynamics the topology of this system is fixed but arbitrary and every subsystem is permitted to have different dynamics these conditions essentially depend only on transmission zeros of every subsystem and the subsystem connection matrix which makes them attractive in the analysis and synthesis of a large scale networked system as an application these conditions are utilized to characterize systems whose steady estimation accuracy with the distributed predictor of zhou 2013 is equal to that of the lumped kalman filter it has been made clear that to guarantee this equivalence the steady update gain matrix of the kalman filter must be block diagonal

robust support vector regression for uncertain input and output data in this paper a robust support vector regression rsvr method with uncertain input and output data is studied first the data uncertainties are investigated under a stochastic framework and two linear robust formulations are derived linear formulations robust to ellipsoidal uncertainties are also considered from a geometric perspective second kernelized rsvr formulations are established for nonlinear regression problems both linear and nonlinear formulations are converted to second order cone programming problems which can be solved efficiently by the interior point method simulation demonstrates that the proposed method outperforms existing rsvrs in the presence of both input and output data uncertainties

boosting with early stopping convergence and consistency boosting is one of the most significant advances in machine learning for classification and regression in its original and computationally flexible version boosting seeks to minimize empirically a loss function in a greedy fashion the resulting estimator takes an additive function form and is built iteratively by applying a base estimator or learner to updated samples depending on the previous iterations an unusual regularization technique early stopping is employed based on cv or a test set this paper studies numerical convergence consistency and statistical rates of convergence of boosting with early stopping when it is carried out over the linear span of a family of basis functions for general loss functions we prove the convergence of boosting s greedy optimization to the infinimum of the loss function over the linear span using the numerical convergence result we find early stopping strategies under which boosting is shown to be consistent based on i i d samples and we obtain bounds on the rates of convergence for boosting estimators simulation studies are also presented to illustrate the relevance of our theoretical results for providing insights to practical aspects of boosting as a side product these results also reveal the importance of restricting the greedy search step sizes as known in practice through the work of friedman and others moreover our results lead to a rigorous proof that for a linearly separable problem adaboost with e 0 step size becomes an l 1 margin maximizer when left to run to convergence

variation of cone metrics on riemann surfaces we discuss the variational properties of the unique conical metric of constant curvature 1 associated to a compact riemann surface together with a weighted divisor 2005 elsevier inc all rights reserved

population theory for boosting ensembles tree ensembles are looked at in distribution space that is the limit case of infinite sample size it is shown that the simplest kind of trees is complete in d dimensional l 2 p space if the number of terminal nodes t is greater than d for such trees we show that the adaboost algorithm gives an ensemble converging to the bayes risk

on the fixed controllable subspace in linear structured systems in this paper we consider interconnected networks that are described by means of structured linear systems with state and control variables we represent these systems whose matrices contain fixed zeros and free parameters by means of directed graphs and study questions concerning controllability and the controllable subspace we show in this paper that the controllable subspace can have a part that will be present for almost all values of the free parameters it actually is a subspace of the controllable subspace and will be referred to as the fixed controllable subspace the subspace can then be seen as a kind of robustly controllable part of the system indeed it is a subspace in the state space with the generic property that states in it can be steered in an arbitrary way we derive a characterization of the fixed controllable subspace using the graph representation the obtained characterization makes use of well known algorithms from optimization and networks theory to get some more insight in the components in the fixed part we also give a representation of the structured linear systems by means of bipartite graphs using the dulmage mendelsohn decomposition we are able to decompose our structured systems in such a way that in some special cases the fixed controllable subspace can be obtained directly from the decomposition

boosting method for local learning in statistical pattern recognition we propose a local boosting method in classification problems borrowing from an idea of the local likelihood method our proposal local boosting includes a simple device for localization for computational feasibility we proved the bayes risk consistency of the local boosting in the framework of probably approximately correct learning inspection of the proof provides a useful viewpoint for comparing ordinary boosting and local boosting with respect to the estimation error and the approximation error both boosting methods have the bayes risk consistency if their approximation errors decrease to zero compared to ordinary boosting local boosting may perform better by controlling the trade off between the estimation error and the approximation error ordinary boosting with complicated base classifiers or other strong classification methods including kernel machines may have classification performance comparable to local boosting with simple base classifiers for example decision stumps local boosting however has an advantage with respect to interpretability local boosting with simple base classifiers offers a simple way to specify which features are informative and how their values contribute to a classification rule even though locally several numerical studies on real data sets confirm these advantages of local boosting

robust twin support vector machine for pattern classification in this paper we proposed a new robust twin support vector machine called r twsvm via second order cone programming formulations for classification which can deal with data with measurement noise efficiently preliminary experiments confirm the robustness of the proposed method and its superiority to the traditional robust svm in both computation time and classification accuracy remarkably since there are only inner products about inputs in our dual problems this makes us apply kernel trick directly for nonlinear cases simultaneously we does not need to solve the extra inverse of matrices which is totally different with existing twsvms in addition we also show that the twsvms are the special case of our robust model and simultaneously give a new dual form of twsvm by degenerating r twsvm which successfully overcomes the existing shortcomings of twsvm

a framework for structural input output and control configuration selection in large scale systems this paper addresses problems on the structural design of large scale control systems an efficient and unified framework is proposed to select the minimum number of manipulated measured variables to achieve structural controllability observability of the system and to select the minimum number of feedback interconnections between measured and manipulated variables such that the closed loop system has no structural fixed modes global solutions are computed using polynomial complexity algorithms in the number of the state variables of the system finally graph theoretic characterizations are proposed which allow a characterization of all possible solutions

inegalites de concentration pour les processus empiriques de classes de parties we propose new concentration inequalities for maxima of set indexed empirical processes our approach is based either on entropy inequalities or on martingale methods the improvements we get concern the rate function which is exactly the large deviations rate function of a binomial law in most of the cases

on ψ learning the concept of large margins have been recognized as an important principle in analyzing learning methodologies including boosting neural networks and support vector machines svms however this concept alone is not adequate for learning in nonseparable cases we propose a learning methodology called learning that is derived from a direct consideration of generalization errors we provide a theory for learning and show that it essentially attains the optimal rates of convergence in two learning examples finally results from simulation studies and from breast cancer classification confirm the ability of learning to outperform svm in generalization

brief paper state and input observability recovering by additional sensor implementation a graph theoretic approach this paper deals with the problem of additional sensor location in order to recover the state and input observability for structured linear systems the proposed method is based on a graph theoretic approach and assumes only the knowledge of the system s structure it allows one to provide the minimal number of the required sensors and either their pertinent location or a necessary and sufficient condition which allows one to check if a sensor location is adequate or not we obtain a sensor placement procedure based on classical and well known graph theory algorithms which have polynomial complexity orders

local rademacher complexities we propose new bounds on the error of learning algorithms in terms of a data dependent notion of complexity the estimates we establish give optimal rates and are based on a local and empirical version of rademacher averages in the sense that the rademacher averages are computed from the data on a subset of functions with small empirical error we present some applications to classification and prediction with convex function classes and with kernel classes in particular

prediction games and arcing algorithms the theory behind the success of adaptive reweighting and combining algorithms arcing such as adaboost freund schapire 1996a 1997 and others in reducing generalization error has not been well understood by formulating prediction as a game where one player makes a selection from instances in the training set and the other a convex linear combination of predictors from a finite set existing arcing algorithms are shown to be algorithms for finding good game strategies the minimax theorem is an essential ingredient of the convergence proofs an arcing algorithm is described that converges to the optimal strategy a bound on the generalization error for the combined predictors in terms of their maximum error is proven that is sharper than bounds to date schapire freund bartlett and lee 1997 offered an explanation of why adaboost works in terms of its ability to produce generally high margins the empirical comparison of adaboost to the optimal arcing algorithm shows that their explanation is n

prescribing curvature on compact surfaces with conical singularities we study the berger nirenberg problem on surfaces with conical singularities i e we discuss conditions under which a function on a riemann surface is the gaussian curvature of some conformal metric with a prescribed set of singularities of conical types

complexity regularization via localized random penalties in this article model selection via penalized empirical loss minimization in nonparametric classification problems is studied data dependent penalties are constructed which are based on estimates of the complexity of a small subclass of each model class containing only those functions with small empirical loss the penalties are novel since those considered in the literature are typically based on the entire model class oracle inequalities using these penalties are established and the advantage of the new penalties over those based on the complexity of the whole model class is demonstrated