consistency of multiclass empirical risk minimization methods based on convex loss the consistency of classification algorithm plays a central role in statistical learning theory a consistent algorithm guarantees us that taking more samples essentially suffices to roughly reconstruct the unknown distribution we consider the consistency of erm scheme over classes of combinations of very simple rules base classifiers in multiclass classification our approach is under some mild conditions to establish a quantitative relationship between classification errors and convex risks in comparison with the related previous work the feature of our result is that the conditions are mainly expressed in terms of the differences between some values of the convex function

boosting the margin a new explanation for the effectiveness of voting methods one of the surprising recurring phenomena observed in experiments with boosting is that the test error of the generated classifier usually does not increase as its size becomes very large and often is observed to decrease even after the training error reaches zero in this paper we show that this phenomenon is related to the distribution of margins of the training examples with respect to the generated voting classification rule where the margin of an example is simply the difference between the number of correct votes and the maximum number of votes received by any incorrect label we show that techniques used in the analysis of vapnik s support vector classifiers and of neural networks with small weights can be applied to voting methods to relate the margin distribution to the test error we also show theoretically and experimentally that boosting is especially effective at increasing the margins of the training examples finally we compare our explanation to those based on the bias variance decomposition

consistency and convergence rates of one class svms and related algorithms we determine the asymptotic behaviour of the function computed by support vector machines svm and related algorithms that minimize a regularized empirical convex loss function in the reproducing kernel hilbert space of the gaussian rbf kernel in the situation where the number of examples tends to infinity the bandwidth of the gaussian kernel tends to 0 and the regularization parameter is held fixed non asymptotic convergence bounds to this limit in the l2 sense are provided together with upper bounds on the classification error that is shown to converge to the bayes risk therefore proving the bayes consistency of a variety of methods although the regularization term does not vanish these results are particularly relevant to the one class svm for which the regularization can not vanish by construction and which is shown for the first time to be a consistent density level set estimator

nonparametric modeling of neural point processes via stochastic gradient boosting regression statistical nonparametric modeling tools that enable the discovery and approximation of functional forms e g tuning functions relating neural spiking activity to relevant covariates are desirable tools in neuroscience in this article we show how stochastic gradient boosting regression can be successfully extended to the modeling of spiking activity data while preserving their point process nature thus providing a robust nonparametric modeling tool we formulate stochastic gradient boosting in terms of approximating the conditional intensity function of a point process in discrete time and use the standard likelihood of the process to derive the loss function for the approximation problem to illustrate the approach we apply the algorithm to the modeling of primary motor and parietal spiking activity as a function of spiking history and kinematics during a two dimensional reaching task model selection goodness of fit via the time rescaling theorem model interpretation via partial dependence plots ranking of covariates according to their relative importance and prediction of peri event time histograms are illustrated and discussed additionally we use the tenfold cross validated log likelihood of the modeled neural processes 67 cells to compare the performance of gradient boosting regression to two alternative approaches standard generalized linear models glms and bayesian p splines with markov chain monte carlo mcmc sampling in our data set gradient boosting outperformed both bayesian p splines in approximately 90 of the cells and glms 100 because of its good performance and computational efficiency we propose stochastic gradient boosting regression as an off the shelf nonparametric tool for initial analyses of large neural data sets e g more than 50 cells more than 105 samples per cell with corresponding multidimensional covariate spaces e g more than four covariates in the cases where a functional form might be amenable to a more compact representation gradient boosting might also lead to the discovery of simpler parametric models

statistical analysis of some multi category large margin classification methods the purpose of this paper is to investigate statistical properties of risk minimization based multi category classification methods these methods can be considered as natural extensions of binary large margin classification we establish conditions that guarantee the consistency of classifiers obtained in the risk minimization framework with respect to the classification error examples are provided for four specific forms of the general formulation which extend a number of known methods using these examples we show that some risk minimization formulations can also be used to obtain conditional probability estimates for the underlying problem such conditional probability information can be useful for statistical inferencing tasks beyond classification

existence of a positive solution to kirchhoff type problems without compactness conditions abstract the existence of a positive solution to a kirchhoff type problem on r n is proved by using variational methods and the new result does not require usual compactness conditions a cut off functional is utilized to obtain the bounded palais smale sequences

asymptotics in empirical risk minimization in this paper we study a two category classification problem we indicate the categories by labels y 1 and y 1 we observe a covariate or feature x x d consider a collection ha of classifiers indexed by a finite dimensional parameter a and the classifier ha that minimizes the prediction error over this class the parameter a is estimated by the empirical risk minimizer n over the class where the empirical risk is calculated on a training sample of size n we apply the kim pollard theorem to show that under certain differentiability assumptions n converges to a with rate n 1 3 and also present the asymptotic distribution of the renormalized estimator for example let v0 denote the set of x on which given x x the label y 1 is more likely than the label y 1 if x is one dimensional the set v0 is the union of disjoint intervals the problem is then to estimate the thresholds of the intervals we obtain the asymptotic distribution of the empirical risk minimizer when the classifiers have k thresholds where k is fixed we furthermore consider an extension to higher dimensional x assuming basically that v0 has a smooth boundary in some given parametric class we also discuss various rates of convergence when the differentiability conditions are possibly violated here we again restrict ourselves to one dimensional x we show that the rate is n 1 in certain cases and then also obtain the asymptotic distribution for the empirical prediction error

on the rate of convergence of regularized boosting classifiers a regularized boosting method is introduced for which regularization is obtained through a penalization function it is shown through oracle inequalities that this method is model adaptive the rate of convergence of the probability of misclassification is investigated it is shown that for quite a large class of distributions the probability of error converges to the bayes risk at a rate faster than n v 2 4 v 1 where v is the vc dimension of the base class whose elements are combined by boosting methods to obtain an aggregated classifier the dimension independent nature of the rates may partially explain the good behavior of these methods in practical problems under tsybakov s noise condition the rate of convergence is even faster we investigate the conditions necessary to obtain such rates for different base classes the special case of boosting using decision stumps is studied in detail we characterize the class of classifiers realizable by aggregating decision stumps it is shown that some versions of boosting work especially well in high dimensional logistic additive models it appears that adding a limited labelling noise to the training data may in certain cases improve the convergence as has been also suggested by other authors

projective subvarieties having large green lazarsfeld index abstract let x p n c be a nondegenerate projective irreducible subvariety of degree d and codimension c 1 the green lazarsfeld index of x denoted by index x is defined as the largest p such that the homogeneous ideal of x is generated by quadrics and the syzygies among them are generated by linear syzygies until the p 1 th stage thus index x is an important invariant in order to describe the minimal free resolution of x recently it is shown that d c 1 if and only if index x c and x is a del pezzo variety if and only if index x c 1 in this paper we prove that index x c 2 c 3 if and only if x is either a complete intersection of three quadrics or else an arithmetically cohen macaulay variety with d c 3 theorem 1 1 also we classify x with index x c 3 c 4 for the cases when d c 2 theorem 4 1 and when x is a smooth surface theorem 4 3

online regularized classification algorithms this paper considers online classification learning algorithms based on regularization schemes in reproducing kernel hilbert spaces associated with general convex loss functions a novel capacity independent approach is presented it verifies the strong convergence of the algorithm under a very weak assumption of the step sizes and yields satisfactory convergence rates for polynomially decaying step sizes explicit learning rates with respect to the misclassification error are given in terms of the choice of step sizes and the regularization parameter depending on the sample size error bounds associated with the hinge loss the least square loss and the support vector machine q norm loss are presented to illustrate our method

are loss functions all the same in this letter we investigate the impact of choosing different loss functions from the viewpoint of statistical learning theory we introduce a convexity assumption which is met by all loss functions commonly used in the literature and study how the bound on the estimation error changes with the loss we also derive a general result on the minimizer of the expected risk for a convex loss function in the case of classification the main outcome of our analysis is that for classification the hinge loss appears to be the loss of choice other things being equal the hinge loss leads to a convergence rate practically indistinguishable from the logistic loss rate and much better than the square loss rate furthermore if the hypothesis space is sufficiently rich the bounds obtained for the hinge loss are not loosened by the thresholding stage

small schemes and varieties of minimal degree we prove that if x p r is any 2 regular scheme in the sense of castelnuovo mumford then x is small this means that if l is a linear space and y l x is finite then y is linearly independent in the sense that the dimension of the linear span of y is deg y 1 the converse is true and well known for finite schemes but false in general the main result of this paper is that the converse small implies 2 regular is also true for reduced schemes algebraic sets this is proven by means of a delicate geometric analysis leading to a complete classification we show that the components of a small algebraic set are varieties of minimal degree meeting in a particularly simple way from the classification one can show that if x p r is 2 regular then so is xred and so also is the projection of x from any point of x our results extend the del pezzo bertini classification of varieties of minimal degree the charac terization of these as the varieties of regularity 2 by eisenbud goto and the construction of 2 regular square free monomial ideals by froberg 0 introduction throughout this paper we will work with projective schemes x p r over an algebraically closed field k the castelnuovo mumford regularity of x p r is a basic homological measure of the complexity of x and its embedding in p r that gives a bound for the degrees of the generators of the defining ideal ix of x and for many other invariants the only schemes of regular ity 1 are the linear spaces but no classification is known for projective schemes of regularity 2 in this paper we prove a structure theorem for reduced 2 regular schemes showing that their irreducible components are varieties of minimal degree and characterizing how these components can meet we also show that the reduced structure on any 2 regular scheme is 2 regular and thus we obtain a complete description of the reduced structures on 2 regular schemes since a high veronese re embedding of any zero dimensional scheme is 2 regular one cannot hope to characterize the isomorphism types of all 2 regular nonreduced schemes before stating our results we review some basic notions for any subscheme x p r we write span x for the smallest linear subspace of p r containing x recall that every variety reduced irreducible scheme x p r satisfies the condition deg x 1 codim x span x

a view of margin losses as regularizers of probability estimates regularization is commonly used in classifier design to assure good generalization classical regularization enforces a cost on classifier complexity by constraining parameters this is usually combined with a margin loss which favors large margin decision rules a novel and unified view of this architecture is proposed by showing that margin losses act as regularizers of posterior class probabilities in a way that amplifies classical parameter regularization the problem of controlling the regularization strength of a margin loss is considered using a decomposition of the loss in terms of a link and a binding function the link function is shown to be responsible for the regularization strength of the loss while the binding function determines its outlier robustness a large class of losses is then categorized into equivalence classes of identical regularization strength or outlier robustness it is shown that losses in the same regularization class can be parameterized so as to have tunable regularization strength this parameterization is finally used to derive boosting algorithms with loss regularization boostlr three classes of tunable regularization losses are considered in detail canonical losses can implement all regularization behaviors but have no flexibility in terms of outlier modeling shrinkage losses support equally parameterized link and binding functions leading to boosting algorithms that implement the popular shrinkage procedure this offers a new explanation for shrinkage as a special case of loss based regularization finally tunable losses enable the independent parameterization of link and binding functions leading to boosting algorithms of great exibility this is illustrated by the derivation of an algorithm that generalizes both adaboost and logitboost behaving as either one when that best suits the data to classify various experiments provide evidence of the benefits of probability regularization for both classification and estimation of posterior class probabilities

convex calibration dimension for multiclass loss matrices we study consistency properties of surrogate loss functions for general multiclass learning problems defined by a general multiclass loss matrix we extend the notion of classification calibration which has been studied for binary and multiclass 0 1 classification problems and for certain other specific learning problems to the general multiclass setting and derive necessary and sufficient conditions for a surrogate loss to be calibrated with respect to a loss matrix in this setting we then introduce the notion of convex calibration dimension of a multiclass loss matrix which measures the smallest size of a prediction space in which it is possible to design a convex surrogate that is calibrated with respect to the loss matrix we derive both upper and lower bounds on this quantity and use these results to analyze various loss matrices in particular we apply our framework to study various subset ranking losses and use the convex calibration dimension as a tool to show both the existence and non existence of various types of convex calibrated surrogates for these losses our results strengthen recent results of duchi et al 2010 and calauzenes et al 2012 on the non existence of certain types of convex calibrated surrogates in subset ranking we anticipate the convex calibration dimension may prove to be a useful tool in the study and design of surrogate losses for general multiclass learning problems

statistical analysis of bayes optimal subset ranking the ranking problem has become increasingly important in modern applications of statistical methods in automated decision making systems in particular we consider a formulation of the statistical ranking problem which we call subset ranking and focus on the discounted cumulated gain dcg criterion that measures the quality of items near the top of the rank list similar to error minimization for binary classification direct optimization of natural ranking criteria such as dcg leads to a nonconvex optimization problems that can be np hard therefore a computationally more tractable approach is needed we present bounds that relate the approximate optimization of dcg to the approximate minimization of certain regression errors these bounds justify the use of convex learning formulations for solving the subset ranking problem the resulting estimation methods are not conventional in that we focus on the estimation quality in the top portion of the rank list we further investigate the asymptotic statistical behavior of these formulations under appropriate conditions the consistency of the estimation schemes with respect to the dcg metric can be derived

boosting with the l2 loss this article investigates a computationally simple variant of boosting l2boost which is constructed from a functional gradient descent algorithm with the l2 loss function like other boosting algorithms l2boost uses many times in an iterative fashion a prechosen fitting method called the learner based on the explicit expression of refitting of residuals of l2boost the case with symmetric linear learners is studied in detail in both regression and classification in particular with the boosting iteration m working as the smoothing or regularization parameter a new exponential bias variance trade off is found with the variance complexity term increasing very slowly as m tends to infinity when the learner is a smoothing spline an optimal rate of convergence result holds for both regression and classification and the boosted smoothing spline even adapts to higher order unknown smoothness moreover a simple expansion of a smoothed 0 1 loss function is derived to reveal the importance of the d

convergence and consistency of regularized boosting with weakly dependent observations this paper studies the statistical convergence and consistency of regularized boosting methods where the samples need not be independent and identically distributed but can come from stationary weakly dependent sequences consistency is proven for the composite classifiers that result from a regularization achieved by restricting the 1 norm of the base classifiers weights the less restrictive nature of sampling considered here is manifested in the consistency result through a generalized condition on the growth of the regularization parameter the weaker the sample dependence the faster the regularization parameter is allowed to grow with increasing sample size a consistency result is also provided for data dependent choices of the regularization parameter

positive solutions for a quasilinear elliptic equation of kirchhoff type this paper is concerned with the existence of positive solutions to the class of nonlocal boundary value problems of the type m w u 2dx du f x u email protected u 0 email protected email protected where w is a smooth bounded domain of n m is a positive function and f has subcritical growth

on the doubt about margin explanation of boosting margin theory provides one of the most popular explanations to the success of adaboost where the central point lies in the recognition that margin is the key for characterizing the performance of adaboost this theory has been very influential e g it has been used to argue that adaboost usually does not overfit since it tends to enlarge the margin even after the training error reaches zero previously the minimum margin bound was established for adaboost however breiman 1999 9 pointed out that maximizing the minimum margin does not necessarily lead to a better generalization later reyzin and schapire 2006 37 emphasized that the margin distribution rather than minimum margin is crucial to the performance of adaboost in this paper we first present the kth margin bound and further study on its relationship to previous work such as the minimum margin bound and emargin bound then we improve the previous empirical bernstein bounds audibert et al 2009 maurer and pontil 2009 2 30 and based on such findings we defend the margin based explanation against breiman s doubts by proving a new generalization error bound that considers exactly the same factors as schapire et al 1998 39 but is sharper than breiman s 1999 9 minimum margin bound by incorporating factors such as average margin and variance we present a generalization error bound that is heavily related to the whole margin distribution we also provide margin distribution bounds for generalization error of voting classifiers in finite vc dimension space

on the bayes risk consistency of regularized boosting methods the probability of error of classification methods based on convex combinations of simple base classifiers by boosting algorithms is investigated the main result of the paper is that certain regularized boosting algorithms provide bayes risk consistent classifiers under the sole assumption that the bayes classifier may be approximated by a convex combination of the base classifiers nonasymptotic distribution free bounds are also developed which offer interesting new insight into how boosting works and help explain its success in practical classification problems

statistical behavior and consistency of classification methods based on convex risk minimization we study how closely the optimal bayes error rate can be approximately reached using a classification algorithm that computes a classifier by minimizing a convex upper bound of the classification error function the measurement of closeness is characterized by the loss function used in the estimation we show that such a classification scheme can be generally regarded as a nonmaximum likelihood conditional in class probability estimate and we use this analysis to compare various convex loss functions that have appeared in the literature furthermore the theoretical insight allows us to design good loss functions with desirable properties another aspect of our analysis is to demonstrate the consistency of certain classification methods using convex risk minimization this study sheds light on the good performance of some recently proposed linear classification methods including boosting and support vector machines it also shows their limitations and suggests possible improvements

prediction games and arcing algorithms the theory behind the success of adaptive reweighting and combining algorithms arcing such as adaboost freund schapire 1996a 1997 and others in reducing generalization error has not been well understood by formulating prediction as a game where one player makes a selection from instances in the training set and the other a convex linear combination of predictors from a finite set existing arcing algorithms are shown to be algorithms for finding good game strategies the minimax theorem is an essential ingredient of the convergence proofs an arcing algorithm is described that converges to the optimal strategy a bound on the generalization error for the combined predictors in terms of their maximum error is proven that is sharper than bounds to date schapire freund bartlett and lee 1997 offered an explanation of why adaboost works in terms of its ability to produce generally high margins the empirical comparison of adaboost to the optimal arcing algorithm shows that their explanation is n

learning gradients by a gradient descent algorithm we propose a stochastic gradient descent algorithm for learning the gradient of a regression function from random samples of function values this is a learning algorithm involving mercer kernels by a detailed analysis in reproducing kernel hilbert spaces we provide some error bounds to show that the gradient estimated by the algorithm converges to the true gradient under some natural conditions on the regression function and suitable choices of the step size and regularization parameters

svm soft margin classifiers linear programming versus quadratic programming support vector machine svm soft margin classifiers are important learning algorithms for classification problems they can be stated as convex optimization problems and are suitable for a large data setting linear programming svm classifiers are especially efficient for very large size samples but little is known about their convergence compared with the well understood quadratic programming svm classifier in this article we point out the difficulty and provide an error analysis our analysis shows that the convergence behavior of the linear programming svm is almost the same as that of the quadratic programming svm this is implemented by setting a stepping stone between the linear programming svm and the classical 1 norm soft margin classifier an upper bound for the misclassification error is presented for general probability distributions explicit learning rates are derived for deterministic and weakly separable distributions and for distributions satisfying some tsybakov noise condition

empirical margin distributions and bounding the generalization error of combined classifiers we prove new probabilistic upper bounds on generalization error of complex classifiers that are combinations of simple classifiers such combinations could be implemented by neural networks or by voting methods of combining the classifiers such as boosting and bagging the bounds are in terms of the empirical distribution of the margin of the combined classifier they are based on the methods of the theory of gaussian and empirical processes comparison inequalities symmetrization method concentration inequalities and they improve previous results of bartlett 1998 on bounding the generalization error of neural networks in terms of ell 1 norms of the weights of neurons and of schapire freund bartlett and lee 1998 on bounding the generalization error of boosting we also obtain rates of convergence in levy distance of empirical margin distribution to the true margin distribution uniformly over the classes of classifiers and prove the optimality of these rates

partial elimination ideals and secant cones for any k n0 we show that the cone of k 1 secant lines of a closed subscheme z pkn over an algebraically closed field k running through a closed point p pkn is defined by the k th partial elimination ideal of z with respect to p we use this fact to give an algorithm for computing secant cones also we show that under certain conditions partial elimination ideals describe the length of the fibres of a multiple projection in a way similar to the way they do for simple projections finally we study some examples illustrating these results computed by means of singular

convexity classification and risk bounds many of the classification algorithms developed in the machine learning literature including the support vector machine and boosting can be viewed as minimum contrast methods that minimize a convex surrogate of the 0 1 loss function the convexity makes these algorithms computationally efficient the use of a surrogate however has statistical consequences that must be balanced against the computational virtues of convexity to study these issues we provide a general quantitative relationship between the risk as assessed using the 0 1 loss and the risk as assessed using any nonnegative surrogate loss function we show that this relationship gives nontrivial upper bounds on excess risk under the weakest possible condition on the loss function that it satisfies a pointwise form of fisher consistency for classification the relationship is based on a simple variational transformation of the loss function that is easy to compute in many applications we also present a refined version of this result in the

the nehari manifold for a kirchhoff type problem involving sign changing weight functions abstract this paper examines a class of kirchhoff type equations that involve sign changing weight functions using nehari manifold and fibering map the existence of multiple positive solutions is established

ground state sign changing solutions for kirchhoff type problems in bounded domains abstract in the present paper we consider the existence of ground state sign changing solutions for a class of kirchhoff type problems 0 1 a b u 2 d x u f u x u 0 x where r n is a bounded domain with a smooth boundary n 1 2 3 a 0 b 0 and f c r r under some weak assumptions on f with the aid of some new analytical skills and non nehari manifold method we prove that 0 1 possesses one ground state sign changing solution u b and its energy is strictly larger than twice that of the ground state solutions of nehari type furthermore we establish the convergence property of u b as the parameter b 0 our results improve and generalize some results obtained by w shuai 2015 34

sharp bounds for higher linear syzygies and classifications of projective varieties in the present paper we consider upper bounds of higher linear syzygies i e graded betti numbers in the first linear strand of the minimal free resolutions of projective varieties in arbitrary characteristic for this purpose we first remind partial elimination ideals peis theory and introduce a new framework in which one can study the syzygies of embedded projective varieties well using peis theory and the reduction method via inner projections next we establish fundamental inequalities which govern the relations between the graded betti numbers in the first linear strand of an algebraic set x and those of its inner projection x q using these results we obtain some natural sharp upper bounds for higher linear syzygies of any nondegenerate projective variety in terms of the codimension with respect to its own embedding and classify what the extremal case and the next to extremal case are this is a generalization of castelnuovo and fano s results on the number of quadrics containing a given variety and another characterization of varieties of minimal degree and del pezzo varieties from the viewpoint of syzygies note that our method could also be applied to get similar results for more general categories e g connected in codimension one algebraic sets

existence and asymptotic behavior of nodal solutions for the kirchhoff type problems in r3 abstract in this paper we study the existence and asymptotic behavior of nodal solutions to the following kirchhoff problem a b r 3 u 2 d x u v x u f x u in r 3 u h 1 r 3 where v x is a smooth function a b are positive constants because the so called nonlocal term r 3 u 2 d x u is involved in the equation the variational functional of the equation has totally different properties from the case of b 0 under suitable construction conditions we prove that for any positive integer k the problem has a sign changing solution u k b which changes signs exactly k times moreover the energy of u k b is strictly increasing in k and for any sequence b n 0 n there is a subsequence b n s such that u k b n s converges in h 1 r 3 to w k as s where w k also changes signs exactly k times and solves the following equation a u v x u f x u in r 3 u h 1 r 3

some theory for generalized boosting algorithms we give a review of various aspects of boosting clarifying the issues through a few simple results and relate our work and that of others to the minimax paradigm of statistics we consider the population version of the boosting algorithm and prove its convergence to the bayes classifier as a corollary of a general result about gauss southwell optimization in hilbert space we then investigate the algorithmic convergence of the sample version and give bounds to the time until perfect separation of the sample we conclude by some results on the statistical optimality of the l2 boosting

complexities of convex combinations and bounding the generalization error in classification we introduce and study several measures of complexity of functions from the convex hull of a given base class these complexity measures take into account the sparsity of the weights of a convex combination as well as certain clustering properties of the base functions involved in it we prove new upper confidence bounds on the generalization error of ensemble voting classification algorithms that utilize the new complexity measures along with the empirical distributions of classification margins providing a better explanation of generalization performance of large margin classification methods

boosted classification trees and class probability quantile estimation the standard by which binary classifiers are usually judged misclassification error assumes equal costs of misclassifying the two classes or equivalently classifying at the 1 2 quantile of the conditional class probability function p y 1 x boosted classification trees are known to perform quite well for such problems in this article we consider the use of standard off the shelf boosting for two more general problems 1 classification with unequal costs or equivalently classification at quantiles other than 1 2 and 2 estimation of the conditional class probability function p y 1 x we first examine whether the latter problem estimation of p y 1 x can be solved with logitboost and with adaboost when combined with a natural link function the answer is negative both approaches are often ineffective because they overfit p y 1 x even though they perform well as classifiers a major negative point of the present article is the disconnect between class probability estimation and classification next we consider the practice of over under sampling of the two classes we present an algorithm that uses adaboost in conjunction with over under sampling and jittering of the data jous boost this algorithm is simple yet successful and it preserves the advantage of relative protection against overfitting but for arbitrary misclassification costs and equivalently arbitrary quantile boundaries we then use collections of classifiers obtained from a grid of quantiles to form estimators of class probabilities the estimates of the class probabilities compare favorably to those obtained by a variety of methods across both simulated and real data sets

process consistency for adaboost recent experiments and theoretical studies show that adaboost can overfit in the limit of large time if running the algorithm forever is suboptimal a natural question is how low can the prediction error be during the process of adaboost we show under general regularity conditions that during the process of adaboost a consistent prediction is generated which has the prediction error approximating the optimal bayes error as the sample size increases this result suggests that while running the algorithm forever can be suboptimal it is reasonable to expect that some regularization method via truncation of the process may lead to a near optimal performance for sufficiently large sample size

additive logistic regression a statistical view of boosting boosting is one of the most important recent developments in classification methodology boosting works by sequentially applying a classification algorithm to reweighted versions of the training data and then taking a weighted majority vote of the sequence of classifiers thus produced for many classification algorithms this simple strategy results in dramatic improvements in performance we show that this seemingly mysterious phenomenon can be understood in terms of well known statistical principles namely additive modeling and maximum likelihood for the two class problem boosting can be viewed as an approximation to additive modeling on the logistic scale using maximum bernoulli likelihood as a criterion we develop more direct approximations and show that they exhibit nearly identical results to boosting direct multiclass generalizations based on multinomial likelihood are derived that exhibit performance comparable to other recently proposed multiclass generalizations of boosting in most situations and far superior in some we suggest a minor modification to boosting that can reduce computation often by factors of 10 to 50 finally we apply these insights to produce an alternative formulation of boosting decision trees this approach based on best first truncated tree induction often leads to better performance and can provide interpretable descriptions of the aggregate decision rule it is also much faster computationally making it more suitable to large scale data mining applications

best subset selection persistence in high dimensional statistical learning and optimization under l1 constraint let y x 1 x m be a random vector it is desired to predict y based on x 1 x m examples of prediction methods are regression classification using logistic regression or separating hyperplanes and so on we consider the problem of best subset selection and study it in the context m n a 1 where n is the number of observations we investigate procedures that are based on empirical risk minimization it is shown that in common cases we should aim to find the best subset among those of size which is of order o n log n it is also shown that in some asymptotic sense when assuming a certain sparsity condition there is no loss in letting m be much larger than n for example m n a 1 this is in comparison to starting with the best subset of size smaller than n and regardless of the value of a we then study conditions under which empirical risk minimization subject to l 1 constraint yields nearly the best subset these results extend some recent results obtained by greenshtein and ritov finally we present a high dimensional simulation study of a boosting type classification procedure

projective invariants of quadratic embeddings x sx s2x sx sk1 1x s1x p where k1 k1 x min k skx pk and sk 1x 6 skx for 1 k k1 for example if r a 1 b 1 1 pr is the projectivization of the linear space of a 1 b 1 matrices and x pa pb pr is the segre variety corresponding to the a 1 b 1 matrices of rank one then skx corresponds to the matrices whose rank does not exceed k 1 and k1 x min a b the number k1 and the dimensions of the secant varieties sk x dimsk x yield a natural collection of invariants of the projective variety x there are however two serious drawbacks

existence and multiplicity of non trivial solutions for schrodinger kirchhoff type equations with radial potential abstract in the present paper the following schrodinger kirchhoff type problem p a b r n u 2 d x u v x u q x f u in r n u x 0 as x is studied four existence theorems of nontrivial solutions and a sequence of high energy solutions for problem p are obtained by the mountain pass theorem and symmetric mountain pass theorem

arcing classifier with discussion and a rejoinder by the author recent work has shown that combining multiple versions of unstable classifiers such as trees or neural nets results in reduced test set error one of the more effective is bagging here modified training sets are formed by resampling from the original training set classifiers constructed using these training sets and then combined by voting freund and schapire propose an algorithm the basis of which is to adaptively resample and combine hence the acronym arcing so that the weights in the resampling are increased for those cases most often misclassified and the combining is done by weighted voting arcing is more successful than bagging in test set error reduction we explore two arcing algorithms compare them to each other and to bagging and try to understand how arcing works we introduce the definitions of bias and variance for a classifier as components of the test set error unstable classifiers can have low bias on a large range of data sets their problem is high variance combining multiple versions either through bagging or arcing reduces variance significantly

a refined margin analysis for boosting algorithms via equilibrium margin much attention has been paid to the theoretical explanation of the empirical success of adaboost the most influential work is the margin theory which is essentially an upper bound for the generalization error of any voting classifier in terms of the margin distribution over the training data however important questions were raised about the margin explanation breiman 1999 proved a bound in terms of the minimum margin which is sharper than the margin distribution bound he argued that the minimum margin would be better in predicting the generalization error grove and schuurmans 1998 developed an algorithm called lp adaboost which maximizes the minimum margin while keeping all other factors the same as adaboost in experiments however lp adaboost usually performs worse than adaboost putting the margin explanation into serious doubt in this paper we make a refined analysis of the margin theory we prove a bound in terms of a new margin measure called the equilibrium margin emargin the emargin bound is uniformly sharper than breiman s minimum margin bound thus our result suggests that the minimum margin may be not crucial for the generalization error we also show that a large emargin and a small empirical error at emargin imply a smaller bound of the generalization error experimental results on benchmark data sets demonstrate that adaboost usually has a larger emargin and a smaller test error than lp adaboost which agrees well with our theory

boosting for high dimensional linear models we prove that boosting with the squared error loss l 2 boosting is consistent for very high dimensional linear models where the number of predictor variables is allowed to grow essentially as fast as o exp sample size assuming that the true underlying regression function is sparse in terms of the 11 norm of the regression coefficients in the language of signal processing this means consistency for de noising using a strongly overcomplete dictionary if the underlying signal is sparse in terms of the l 1 norm we also propose here an aic based method for tuning namely for choosing the number of boosting iterations this makes l 2 boosting computationally attractive since it is not required to run the algorithm multiple times for cross validation as commonly used so far we demonstrate l 2 boosting for simulated data in particular where the predictor dimension is large in comparison to sample size and for a difficult tumor classification problem with gene expression microarray data

on ranking and generalization bounds the problem of ranking is to predict or to guess the ordering between objects on the basis of their observed features in this paper we consider ranking estimators that minimize the empirical convex risk we prove generalization bounds for the excess risk of such estimators with rates that are faster than 1 n we apply our results to commonly used ranking algorithms for instance boosting or support vector machines moreover we study the performance of considered estimators on real data sets

infinitely many solutions for kirchhoff problems with lack of compactness abstract in this paper we consider the following kirchhoff problem 1 b r 3 u 2 d x u v x u f u x r 3 u h 1 r 3 where b is a positive constant assume that f u is an odd function of u under some appropriate assumptions on v but without radial symmetry or compactness hypotheses we establish the existence of infinitely many solutions to the above problem by using an approximation method employed by sato and shibata 2018 and some new arguments

on surrogate loss functions and f divergences the goal of binary classification is to estimate a discriminant function y from observations of covariate vectors and corresponding binary labels we consider an elaboration of this problem in which the covariates are not available directly but are transformed by a dimensionality reducing quantizer q we present conditions on loss functions such that empirical risk minimization yields bayes consistency when both the discriminant function and the quantizer are estimated these conditions are stated in terms of a general correspondence between loss functions and a class of functionals known as ali silvey or divergence functionals whereas this correspondence was established by blackwell proc 2nd berkeley symp probab statist 1 1951 93 102 univ california press berkeley for the 0 1 loss we extend the correspondence to the broader class of surrogate loss functions that play a key role in the general theory of bayes consistency for binary classification our result makes it possible to pick out the strict subset of surrogate loss functions that yield bayes consistency for joint estimation of the discriminant function and the quantizer

boosting with early stopping convergence and consistency boosting is one of the most significant advances in machine learning for classification and regression in its original and computationally flexible version boosting seeks to minimize empirically a loss function in a greedy fashion the resulting estimator takes an additive function form and is built iteratively by applying a base estimator or learner to updated samples depending on the previous iterations an unusual regularization technique early stopping is employed based on cv or a test set this paper studies numerical convergence consistency and statistical rates of convergence of boosting with early stopping when it is carried out over the linear span of a family of basis functions for general loss functions we prove the convergence of boosting s greedy optimization to the infinimum of the loss function over the linear span using the numerical convergence result we find early stopping strategies under which boosting is shown to be consistent based on i i d samples and we obtain bounds on the rates of convergence for boosting estimators simulation studies are also presented to illustrate the relevance of our theoretical results for providing insights to practical aspects of boosting as a side product these results also reveal the importance of restricting the greedy search step sizes as known in practice through the work of friedman and others moreover our results lead to a rigorous proof that for a linearly separable problem adaboost with e 0 step size becomes an l 1 margin maximizer when left to run to convergence

local rademacher complexities and oracle inequalities in risk minimization let be a class of measurable functions f s 0 1 defined on a probability space s mathcal a p given a sample x1 xn of i i d random variables taking values in s with common distribution p let pn denote the empirical measure based on x1 xn we study an empirical risk minimization problem pnf min f given a solution fn of this problem the goal is to obtain very general upper bounds on its excess risk mathcal e p hat f n p hat f n inf f in mathcal f pf expressed in terms of relevant geometric parameters of the class using concentration inequalities and other empirical processes tools we obtain both distribution dependent and data dependent upper bounds on the excess risk that are of asymptotically correct order in many examples the bounds involve localized sup norms of empirical and rademacher processes indexed by functions from the class we use these bounds to develop model selection techniques in abstract risk minimization problems that can be applied to more specialized frameworks of regression and classification

sign changing solutions for a class of kirchhoff type problem in bounded domains abstract we are interested in the existence of least energy sign changing solutions for a class of kirchhoff type problem in bounded domains because the so called nonlocal term b u 2 d x u is involving in the equation the variational functional of the equation has totally different properties from the case of b 0 combining constraint variational method and quantitative deformation lemma we prove that the problem possesses one least energy sign changing solution u b moreover we show that the energy of u b is strictly larger than the ground state energy finally we regard b as a parameter and give a convergence property of u b as b 0

risk bounds for cart classifiers under a margin condition non asymptotic risk bounds for classification and regression trees cart classifiers are obtained in the binary supervised classification framework under a margin assumption on the joint distribution of the covariates and the labels these risk bounds are derived conditionally on the construction of the maximal binary tree and allow to prove that the linear penalty used in the cart pruning algorithm is valid under the margin condition it is also shown that conditionally on the construction of the maximal tree the final selection by test sample does not alter dramatically the estimation accuracy of the bayes classifier

a hybrid loss for multiclass and structured prediction we propose a novel hybrid loss for multiclass and structured prediction problems that is a convex combination of a log loss for conditional random fields crfs and a multiclass hinge loss for support vector machines svms we provide a sufficient condition for when the hybrid loss is fisher consistent for classification this condition depends on a measure of dominance between labels specifically the gap between the probabilities of the best label and the second best label we also prove fisher consistency is necessary for parametric consistency when learning models such as crfs we demonstrate empirically that the hybrid loss typically performs least as well as and often better than both of its constituent losses on a variety of tasks such as human action recognition in doing so we also provide an empirical comparison of the efficacy of probabilistic and margin based approaches to multiclass and structured prediction

population theory for boosting ensembles tree ensembles are looked at in distribution space that is the limit case of infinite sample size it is shown that the simplest kind of trees is complete in d dimensional l 2 p space if the number of terminal nodes t is greater than d for such trees we show that the adaboost algorithm gives an ensemble converging to the bayes risk

analysis on some infinite modules inner projection and applications a projective scheme x is called quadratic if x is scheme theoretically cut out by homogeneous equations of degree 2 furthermore we say that x satisfies property n2 p if it is quadratic and the quadratic ideal has only linear syzygies up to the first p th steps in the present paper we compare the linear syzygies of the inner projections with those of x and obtain a theorem on embedded linear syzygies as one of our main results this is the natural projection analogue of restricting linear syzygies in the linear section case as an immediate corollary we show that the inner projections of x satisfy property n2 p 1 for any reduced scheme x with property n2 p moreover we also obtain the neccessary lower bound codimx p p p 1 2 which is sharp on the number of quadrics vanishing on x in order to satisfy n2 p and show that the arithmetic depths of inner projections are equal to that of the quadratic scheme x these results admit an interesting syzygetic rigidity theorem on property n2 p which leads the classifications of extremal and next to extremal cases for these results we develop the elimination mapping cone theorem for infinitely generated graded modules and improve the partial elimination ideal theory initiated by m green this new method allows us to treat a wider class of projective schemes which cannot be covered by the koszul cohomology techniques because these are not projectively normal in general

boosting method for local learning in statistical pattern recognition we propose a local boosting method in classification problems borrowing from an idea of the local likelihood method our proposal local boosting includes a simple device for localization for computational feasibility we proved the bayes risk consistency of the local boosting in the framework of probably approximately correct learning inspection of the proof provides a useful viewpoint for comparing ordinary boosting and local boosting with respect to the estimation error and the approximation error both boosting methods have the bayes risk consistency if their approximation errors decrease to zero compared to ordinary boosting local boosting may perform better by controlling the trade off between the estimation error and the approximation error ordinary boosting with complicated base classifiers or other strong classification methods including kernel machines may have classification performance comparable to local boosting with simple base classifiers for example decision stumps local boosting however has an advantage with respect to interpretability local boosting with simple base classifiers offers a simple way to specify which features are informative and how their values contribute to a classification rule even though locally several numerical studies on real data sets confirm these advantages of local boosting

existence and multiplicity of solutions for kirchhoff type equations abstract in this paper the existence and multiplicity of weak solutions are obtained for a class of kirchhoff type problems with dirichlet boundary value conditions by using the mountain pass theorem the local linking theorem the fountain theorem and the symmetric mountain pass lemma in critical point theory

linear normality of general linear sections and some graded betti numbers of 3 regular projective schemes in this paper we study graded betti numbers of any nondegenerate 3 regular algebraic set x in a projective space pn more concretely via generic initial ideals gins method we mainly consider tailing betti numbers whose homological index is at least codim x pn for this purpose we first introduce a key definition nd 1 property which provides a suitable ground where one can generalize the concepts such as being nondegenerate or of minimal degree from the case of varieties to the case of more general closed subschemes and give a clear interpretation on the tailing betti numbers next we recall basic notions and facts on gins theory and we analyze the generation structure of the reverse lexicographic rlex gins of 3 regular nd 1 subschemes as a result we present exact formulae for these tailing betti numbers which connect them with linear normality of general linear sections of x with a linear subspace of dimension at least codim x pn finally we consider some applications and related examples

simultaneous adaptation to the margin and to complexity in classification we consider the problem of adaptation to the margin and to complexity in binary classification we suggest a learning method with a numerically easy aggregation step adaptivity both to the margin and complexity in classification usually involves empirical risk minimization or rademacher complexities which lead to numerical difficulties on the other hand there exist classifiers that are easy to compute and that converge with fast rates but are not adaptive combining these classifiers by our aggregation procedure we get numerically realizable adaptive classifiers that converge with fast rates

universally consistent vertex classification for latent positions graphs in this work we show that using the eigen decomposition of the adjacency matrix we can consistently estimate feature maps for latent position graphs with positive definite link function provided that the latent positions are i i d from some distribution f we then consider the exploitation task of vertex classification where the link

the critical problem of kirchhoff type elliptic equations in dimension four abstract we study the following kirchhoff type elliptic problem p a b u 2 d x u u q u 3 u 0 in u 0 on where r 4 is a bounded domain with smooth boundary moreover we assume a 0 b 0 and 1 q 3 in this paper we prove the existence of solutions of p our tools are the variational method and the concentration compactness argument for ps sequences