boosting the margin a new explanation for the effectiveness of voting methods one of the surprising recurring phenomena observed in experiments with boosting is that the test error of the generated classifier usually does not increase as its size becomes very large and often is observed to decrease even after the training error reaches zero in this paper we show that this phenomenon is related to the distribution of margins of the training examples with respect to the generated voting classification rule where the margin of an example is simply the difference between the number of correct votes and the maximum number of votes received by any incorrect label we show that techniques used in the analysis of vapnik s support vector classifiers and of neural networks with small weights can be applied to voting methods to relate the margin distribution to the test error we also show theoretically and experimentally that boosting is especially effective at increasing the margins of the training examples finally we compare our explanation to those based on the bias variance decomposition

a general theory of canonical induction formulae abstract so far there exist three independent constructions of two different canonical versions of brauer s induction theorem for complex characters due to v snaith p symonds and the author canonical in this context means functorial with respect to restrictions along group homomorphisms in this article we axiomatize the situation in which the above canonical induction formulae are constructed mackey functors and related structures arise in this way naturally as a convenient language this approach allows us to construct canonical induction formulae for arbitrary mackey functors in particular we obtain canonical induction formulae for the brauer character ring the group of projective characters the ring of trivial source modules and the ring of linear source modules in most cases it is not difficult to construct such formulae over the rational numbers a much more subtle question is whether the constructed formula comes from a canonical induction formula defined over the integers we give a sufficient condition in the general framework of mackey functors for a canonical induction formula to be integral as an application we show how canonical induction formulae allow the construction of functorial maps on representation rings in terms of functorial maps on subrings as for example the span of linear characters in the case of the canonical brauer induction formula this will be used in a subsequent article in the case of adams operations and chern classes

monomial resolutions of trivial source modules abstract for trivial source modules admitting a filtration related to their generalized brauer constructions a resolution in terms of monomial modules is given which provides a categorical interpretation for the canonical induction formula applied to these modules

fibred permutation sets and the idempotents and units of monomial burnside rings we study the units of monomial burnside rings and the idempotents of monomial burnside algebras introducing a tenduction map we realise the unit group and the torsion unit group as a mackey functor

on the rate of convergence of regularized boosting classifiers a regularized boosting method is introduced for which regularization is obtained through a penalization function it is shown through oracle inequalities that this method is model adaptive the rate of convergence of the probability of misclassification is investigated it is shown that for quite a large class of distributions the probability of error converges to the bayes risk at a rate faster than n v 2 4 v 1 where v is the vc dimension of the base class whose elements are combined by boosting methods to obtain an aggregated classifier the dimension independent nature of the rates may partially explain the good behavior of these methods in practical problems under tsybakov s noise condition the rate of convergence is even faster we investigate the conditions necessary to obtain such rates for different base classes the special case of boosting using decision stumps is studied in detail we characterize the class of classifiers realizable by aggregating decision stumps it is shown that some versions of boosting work especially well in high dimensional logistic additive models it appears that adding a limited labelling noise to the training data may in certain cases improve the convergence as has been also suggested by other authors

a view of margin losses as regularizers of probability estimates regularization is commonly used in classifier design to assure good generalization classical regularization enforces a cost on classifier complexity by constraining parameters this is usually combined with a margin loss which favors large margin decision rules a novel and unified view of this architecture is proposed by showing that margin losses act as regularizers of posterior class probabilities in a way that amplifies classical parameter regularization the problem of controlling the regularization strength of a margin loss is considered using a decomposition of the loss in terms of a link and a binding function the link function is shown to be responsible for the regularization strength of the loss while the binding function determines its outlier robustness a large class of losses is then categorized into equivalence classes of identical regularization strength or outlier robustness it is shown that losses in the same regularization class can be parameterized so as to have tunable regularization strength this parameterization is finally used to derive boosting algorithms with loss regularization boostlr three classes of tunable regularization losses are considered in detail canonical losses can implement all regularization behaviors but have no flexibility in terms of outlier modeling shrinkage losses support equally parameterized link and binding functions leading to boosting algorithms that implement the popular shrinkage procedure this offers a new explanation for shrinkage as a special case of loss based regularization finally tunable losses enable the independent parameterization of link and binding functions leading to boosting algorithms of great exibility this is illustrated by the derivation of an algorithm that generalizes both adaboost and logitboost behaving as either one when that best suits the data to classify various experiments provide evidence of the benefits of probability regularization for both classification and estimation of posterior class probabilities

a generalized brauer construction and linear source modules for a complete discrete valuation ring 0 with residue field f a subgroup h of a finite group g and a homomorphism p h ox we define a functor v 4 v h p from the category of og modules to the category of fng h p modules and investigate its behaviour with respect to linear source modules

boosting with the l2 loss this article investigates a computationally simple variant of boosting l2boost which is constructed from a functional gradient descent algorithm with the l2 loss function like other boosting algorithms l2boost uses many times in an iterative fashion a prechosen fitting method called the learner based on the explicit expression of refitting of residuals of l2boost the case with symmetric linear learners is studied in detail in both regression and classification in particular with the boosting iteration m working as the smoothing or regularization parameter a new exponential bias variance trade off is found with the variance complexity term increasing very slowly as m tends to infinity when the learner is a smoothing spline an optimal rate of convergence result holds for both regression and classification and the boosted smoothing spline even adapts to higher order unknown smoothness moreover a simple expansion of a smoothed 0 1 loss function is derived to reveal the importance of the d

multiclass posterior probability support vector machines tao et al have recently proposed the posterior probability support vector machine ppsvm which uses soft labels derived from estimated posterior probabilities to be more robust to noise and outliers tao et al s model uses a window based density estimator to calculate the posterior probabilities and is a binary classifier we propose a neighbor based density estimator and also extend the model to the multiclass case our bias variance analysis shows that the decrease in error by ppsvm is due to a decrease in bias on 20 benchmark data sets we observe that ppsvm obtains accuracy results that are higher or comparable to those of canonical svm using significantly fewer support vectors

training a support vector machine in the primal most literature on support vector machines svms concentrates on the dual optimization problem in this letter we point out that the primal problem can also be solved efficiently for both linear and nonlinear svms and that there is no reason for ignoring this possibility on the contrary from the primal point of view new families of algorithms for large scale svm training can be investigated

convergence and consistency of regularized boosting with weakly dependent observations this paper studies the statistical convergence and consistency of regularized boosting methods where the samples need not be independent and identically distributed but can come from stationary weakly dependent sequences consistency is proven for the composite classifiers that result from a regularization achieved by restricting the 1 norm of the base classifiers weights the less restrictive nature of sampling considered here is manifested in the consistency result through a generalized condition on the growth of the regularization parameter the weaker the sample dependence the faster the regularization parameter is allowed to grow with increasing sample size a consistency result is also provided for data dependent choices of the regularization parameter

on the bayes risk consistency of regularized boosting methods the probability of error of classification methods based on convex combinations of simple base classifiers by boosting algorithms is investigated the main result of the paper is that certain regularized boosting algorithms provide bayes risk consistent classifiers under the sole assumption that the bayes classifier may be approximated by a convex combination of the base classifiers nonasymptotic distribution free bounds are also developed which offer interesting new insight into how boosting works and help explain its success in practical classification problems

statistical behavior and consistency of classification methods based on convex risk minimization we study how closely the optimal bayes error rate can be approximately reached using a classification algorithm that computes a classifier by minimizing a convex upper bound of the classification error function the measurement of closeness is characterized by the loss function used in the estimation we show that such a classification scheme can be generally regarded as a nonmaximum likelihood conditional in class probability estimate and we use this analysis to compare various convex loss functions that have appeared in the literature furthermore the theoretical insight allows us to design good loss functions with desirable properties another aspect of our analysis is to demonstrate the consistency of certain classification methods using convex risk minimization this study sheds light on the good performance of some recently proposed linear classification methods including boosting and support vector machines it also shows their limitations and suggests possible improvements

semi supervised learning via regularized boosting working on multiple semi supervised assumptions semi supervised learning concerns the problem of learning in the presence of labeled and unlabeled data several boosting algorithms have been extended to semi supervised learning with various strategies to our knowledge however none of them takes all three semi supervised assumptions i e smoothness cluster and manifold assumptions together into account during boosting learning in this paper we propose a novel cost functional consisting of the margin cost on labeled data and the regularization penalty on unlabeled data based on three fundamental semi supervised assumptions thus minimizing our proposed cost functional with a greedy yet stagewise functional optimization procedure leads to a generic boosting framework for semi supervised learning extensive experiments demonstrate that our algorithm yields favorite results for benchmark and real world classification tasks in comparison to state of the art semi supervised learning algorithms including newly developed boosting algorithms finally we discuss relevant issues and relate our algorithm to the previous work

ensemble tracking we consider tracking as a binary classification problem where an ensemble of weak classifiers is trained online to distinguish between the object and the background the ensemble of weak classifiers is combined into a strong classifier using adaboost the strong classifier is then used to label pixels in the next frame as either belonging to the object or the background giving a confidence map the peak of the map and hence the new position of the object is found using mean shift temporal coherence is maintained by updating the ensemble with new weak classifiers that are trained online during tracking we show a realization of this method and demonstrate it on several video sequences

convexity classification and risk bounds many of the classification algorithms developed in the machine learning literature including the support vector machine and boosting can be viewed as minimum contrast methods that minimize a convex surrogate of the 0 1 loss function the convexity makes these algorithms computationally efficient the use of a surrogate however has statistical consequences that must be balanced against the computational virtues of convexity to study these issues we provide a general quantitative relationship between the risk as assessed using the 0 1 loss and the risk as assessed using any nonnegative surrogate loss function we show that this relationship gives nontrivial upper bounds on excess risk under the weakest possible condition on the loss function that it satisfies a pointwise form of fisher consistency for classification the relationship is based on a simple variational transformation of the loss function that is easy to compute in many applications we also present a refined version of this result in the

composite binary losses we study losses for binary classification and class probability estimation and extend the understanding of them from margin losses to general composite losses which are the composition of a proper loss with a link function we characterise when margin losses can be proper composite losses explicitly show how to determine a symmetric loss in full from half of one of its partial losses introduce an intrinsic parametrisation of composite binary losses and give a complete characterisation of the relationship between proper losses and classification calibrated losses we also consider the question of the best surrogate binary loss we introduce a precise notion of best and show there exist situations where two convex surrogate losses are incommensurable we provide a complete explicit characterisation of the convexity of composite binary losses in terms of the link function and the weight function associated with the proper loss which make up the composite loss this characterisation suggests new ways of surrogate tuning as well as providing an explicit characterisation of when bregman divergences on the unit interval are convex in their second argument finally in an appendix we present some new algorithm independent results on the relationship between properness convexity and robustness to misclassification noise for binary losses and show that all convex proper losses are non robust to misclassification noise

in defense of one vs all classification we consider the problem of multiclass classification our main thesis is that a simple one vs all scheme is as accurate as any other approach assuming that the underlying binary classifiers are well tuned regularized classifiers such as support vector machines this thesis is interesting in that it disagrees with a large body of recent published work on multiclass classification we support our position by means of a critical review of the existing literature a substantial collection of carefully controlled experimental work and theoretical arguments

process consistency for adaboost recent experiments and theoretical studies show that adaboost can overfit in the limit of large time if running the algorithm forever is suboptimal a natural question is how low can the prediction error be during the process of adaboost we show under general regularity conditions that during the process of adaboost a consistent prediction is generated which has the prediction error approximating the optimal bayes error as the sample size increases this result suggests that while running the algorithm forever can be suboptimal it is reasonable to expect that some regularization method via truncation of the process may lead to a near optimal performance for sufficiently large sample size

additive logistic regression a statistical view of boosting boosting is one of the most important recent developments in classification methodology boosting works by sequentially applying a classification algorithm to reweighted versions of the training data and then taking a weighted majority vote of the sequence of classifiers thus produced for many classification algorithms this simple strategy results in dramatic improvements in performance we show that this seemingly mysterious phenomenon can be understood in terms of well known statistical principles namely additive modeling and maximum likelihood for the two class problem boosting can be viewed as an approximation to additive modeling on the logistic scale using maximum bernoulli likelihood as a criterion we develop more direct approximations and show that they exhibit nearly identical results to boosting direct multiclass generalizations based on multinomial likelihood are derived that exhibit performance comparable to other recently proposed multiclass generalizations of boosting in most situations and far superior in some we suggest a minor modification to boosting that can reduce computation often by factors of 10 to 50 finally we apply these insights to produce an alternative formulation of boosting decision trees this approach based on best first truncated tree induction often leads to better performance and can provide interpretable descriptions of the aggregate decision rule it is also much faster computationally making it more suitable to large scale data mining applications

early stopping and non parametric regression an optimal data dependent stopping rule early stopping is a form of regularization based on choosing when to stop running an iterative algorithm focusing on non parametric regression in a reproducing kernel hilbert space we analyze the early stopping strategy for a form of gradient descent applied to the least squares loss function we propose a data dependent stopping rule that does not involve hold out or cross validation data and we prove upper bounds on the squared error of the resulting function estimate measured in either the l2 p and l2 pn norm these upper bounds lead to minimax optimal rates for various kernel classes including sobolev smoothness classes and other forms of reproducing kernel hilbert spaces we show through simulation that our stopping rule compares favorably to two other stopping rules one based on hold out data and the other based on stein s unbiased risk estimate we also establish a tight connection between our early stopping strategy and the solution path of a kernel ridge regression estimator

quantitative a priori estimates for fast diffusion equations with caffarelli kohn nirenberg weights harnack inequalities and holder continuity abstract we study a priori estimates for a class of non negative local weak solution to the weighted fast diffusion equation u t x x u m with 0 m 1 posed on cylinders of 0 t r n the weights x and x with n and 2 n 2 n can be both degenerate and singular and need not belong to the class a 2 a typical assumption for this kind of problems this range of parameters is optimal for the validity of a class of caffarelli kohn nirenberg inequalities which play the role of the standard sobolev inequalities in this more complicated weighted setting the weights that we consider are not translation invariant and this causes a number of extra difficulties and a variety of scenarios for instance the scaling properties of the equation change when considering the problem around the origin or far from it we therefore prove quantitative with computable constants upper and lower estimates for local weak solutions focussing our attention where a change of geometry appears such estimates fairly combine into forms of harnack inequalities of forward backward and elliptic type as a consequence we obtain holder continuity of the solutions with a quantitative even if non optimal exponent our results apply to a quite large variety of solutions and problems the proof of the positivity estimates requires a new method and represents the main technical novelty of this paper our techniques are flexible and can be adapted to more general settings for instance to a wider class of weights or to similar problems posed on riemannian manifolds possibly with unbounded curvature in the linear case m 1 we also prove quantitative estimates recovering known results in some cases and extending such results to a wider class of weights

boosting as a regularized path to a maximum margin classifier in this paper we study boosting methods from a new perspective we build on recent work by efron et al to show that boosting approximately and in some cases exactly minimizes its loss criterion with an l1 constraint on the coefficient vector this helps understand the success of boosting with early stopping as regularized fitting of the loss criterion for the two most commonly used criteria exponential and binomial log likelihood we further show that as the constraint is relaxed or equivalently as the boosting iterations proceed the solution converges in the separable case to an l1 optimal separating hyper plane we prove that this l1 optimal separating hyper plane has the property of maximizing the minimal l1 margin of the training data as defined in the boosting literature an interesting fundamental similarity between boosting and kernel support vector machines emerges as both can be described as methods for regularized optimization in high dimensional predictor space using a computational trick to make the calculation practical and converging to margin maximizing solutions while this statement describes svms exactly it applies to boosting only approximately

boosting with early stopping convergence and consistency boosting is one of the most significant advances in machine learning for classification and regression in its original and computationally flexible version boosting seeks to minimize empirically a loss function in a greedy fashion the resulting estimator takes an additive function form and is built iteratively by applying a base estimator or learner to updated samples depending on the previous iterations an unusual regularization technique early stopping is employed based on cv or a test set this paper studies numerical convergence consistency and statistical rates of convergence of boosting with early stopping when it is carried out over the linear span of a family of basis functions for general loss functions we prove the convergence of boosting s greedy optimization to the infinimum of the loss function over the linear span using the numerical convergence result we find early stopping strategies under which boosting is shown to be consistent based on i i d samples and we obtain bounds on the rates of convergence for boosting estimators simulation studies are also presented to illustrate the relevance of our theoretical results for providing insights to practical aspects of boosting as a side product these results also reveal the importance of restricting the greedy search step sizes as known in practice through the work of friedman and others moreover our results lead to a rigorous proof that for a linearly separable problem adaboost with e 0 step size becomes an l 1 margin maximizer when left to run to convergence

greedy function approximation a gradient boosting machine function estimation approximation is viewed from the perspective of numerical optimization in function space rather than parameter space a connection is made between stagewise additive expansions and steepest descent minimization a general gradient descent boosting paradigm is developed for additive expansions based on any fitting criterion specific algorithms are presented for least squares least absolute deviation and huber m loss functions for regression and multiclass logistic likelihood for classification special enhancements are derived for the particular case where the individual additive components are regression trees and tools for interpreting such treeboost models are presented gradient boosting of regression trees produces competitive highly robust interpretable procedures for both regression and classification especially appropriate for mining less than clean data connections between this approach and the boosting methods of freund and shapire and friedman hastie and tibshirani are discussed

support vector machine classifier with pinball loss traditionally the hinge loss is used to construct support vector machine svm classifiers the hinge loss is related to the shortest distance between sets and the corresponding classifier is hence sensitive to noise and unstable for re sampling in contrast the pinball loss is related to the quantile distance and the result is less sensitive the pinball loss has been deeply studied and widely applied in regression but it has not been used for classification in this paper we propose a svm classifier with the pinball loss called pin svm and investigate its properties including noise insensitivity robustness and misclassification error besides insensitive zone is applied to the pin svm for a sparse model compared to the svm with the hinge loss the proposed pin svm has the same computational complexity and enjoys noise insensitivity and re sampling stability

strictly proper scoring rules prediction and estimation scoring rules assess the quality of probabilistic forecasts by assigning a numerical score based on the predictive distribution and on the event or value that materializes a scoring rule is proper if the forecaster maximizes the expected score for an observation drawn from the distributionf if he or she issues the probabilistic forecast f rather than g f it is strictly proper if the maximum is unique in prediction problems proper scoring rules encourage the forecaster to make careful assessments and to be honest in estimation problems strictly proper scoring rules provide attractive loss and utility functions that can be tailored to the problem at hand this article reviews and develops the theory of proper scoring rules on general probability spaces and proposes and discusses examples thereof proper scoring rules derive from convex functions and relate to information measures entropy functions and bregman divergences in the case of categorical variables we prove a rigorous version of the

some theory for generalized boosting algorithms we give a review of various aspects of boosting clarifying the issues through a few simple results and relate our work and that of others to the minimax paradigm of statistics we consider the population version of the boosting algorithm and prove its convergence to the bayes classifier as a corollary of a general result about gauss southwell optimization in hilbert space we then investigate the algorithmic convergence of the sample version and give bounds to the time until perfect separation of the sample we conclude by some results on the statistical optimality of the l2 boosting