boosting the margin a new explanation for the effectiveness of voting methods one of the surprising recurring phenomena observed in experiments with boosting is that the test error of the generated classifier usually does not increase as its size becomes very large and often is observed to decrease even after the training error reaches zero in this paper we show that this phenomenon is related to the distribution of margins of the training examples with respect to the generated voting classification rule where the margin of an example is simply the difference between the number of correct votes and the maximum number of votes received by any incorrect label we show that techniques used in the analysis of vapnik s support vector classifiers and of neural networks with small weights can be applied to voting methods to relate the margin distribution to the test error we also show theoretically and experimentally that boosting is especially effective at increasing the margins of the training examples finally we compare our explanation to those based on the bias variance decomposition

first passage percolation network flows and electrical resistances we show that the first passage times of first passage percolation on 2 are such that p 0n n decay geometrically as n where may represent any of the four usual first passage time processes the former estimate requires no moment condition on the time coordinates but there exists a geometrically decaying estimate for the latter quantity if and only if the time coordinate distribution has finite moment generating function near the origin here is the time constant and 0 we study the line to line first passage times and describe applications to the maximum network flow through a randomly capacitated subsection of 2 and to the asymptotic behaviour of the electrical resistance of a subsection of 2 when the edges of the subsection are wires in an electrical network with random resistances in the latter case we show for example that if each edge resistance equals 1 or ohms with probabilities p and 1 p respectively then the effective resistance rn across opposite faces of an n by n box satisfies the following a if p 1 2 then p rn 1 as n b if p 1 2 then there exists p such that p p 1 underset raise0 3em hbox smash scriptscriptstyle leqslant mathop lim inf r n limits n to infty underset raise0 3em hbox smash scriptscriptstyle leqslant mathop lim sup r n limits n to infty underset raise0 3em hbox smash scriptscriptstyle leqslant v p 1

square root penalty adaptation to the margin in classification and in edge estimation we consider the problem of adaptation to the margin in binary classification we suggest a penalized empirical risk minimization classifier that adaptively attains up to a logarithmic factor fast optimal rates of convergence for the excess risk that is rates that can be faster than n 1 2 where n is the sample size we show that our method also gives adaptive estimators for the problem of edge estimation

risk bounds for embedded variable selection in classification trees the problems of model and variable selections for classification trees are jointly considered a penalized criterion is proposed which explicitly takes into account the number of variables and a risk bound inequality is provided for the tree classifier minimizing this criterion this penalized criterion is compared to the one used during the pruning step of the cart algorithm it is shown that the two criteria are similar under some specific margin assumptions in practice the tuning parameter of the cart penalty has to be calibrated by hold out or cross validation a simulation study is performed to compare the form of the theoretical penalized criterion we propose with the form obtained after tuning the regularization parameter via cross validation

domination by product measures we consider families of 0 1 valued random variables indexed by the vertices of countable graphs with bounded degree first we show that if these random variables satisfy the property that conditioned on what happens outside of the neighborhood of each given site the probability of seeing a 1 at this site is at least a value p which is large enough then this random field dominates a product measure with positive density moreover the density of this dominated product measure can be made arbitrarily close to 1 provided that p is close enough to 1 next we address the issue of obtaining the critical value of p defined as the threshold above which the domination by positive density product measures is assured for the graphs which have as vertices the integers and edges connecting vertices which are separated by no more than k units this critical value is shown to be 1 k k k 1 k 1 and a discontinuous transition is shown to occur similar critical values of p are found for other classes of probability measures on 0 1 z for the class of k dependent measures the critical value is again 1 k k k 1 k 1 with a discontinuous transition for the class of two block factors the critical value is shown to be 1 2 and a continuous transition is shown to take place in this case thus both the critical value and the nature of the transition are different in the two block factor and 1 dependent cases

overall properties of a discrete membrane with randomly distributed defects a prototype for variational percolation problems with surface energies is considered the description of the macroscopic properties of a two dimensional discrete membrane with randomly distributed defects in the spirit of the weak membrane model of blake and zisserman quadratic springs that may break at a critical length of the elongation after introducing energies depending on suitable independent identically distributed random variables this is done by exhibiting an equivalent continuum energy computed using convergence geometric measure theory and percolation arguments we show that below a percolation threshold the effect of the defects is negligible and the continuum description is given by the dirichlet integral while above that threshold an additional griffith fracture term appears in the energy which depends only on the defect probability through the chemical distance on the weak cluster of defects

the speed of a biased random walk on a percolation cluster at high density we study the speed of a biased random walk on a percolation cluster on z d in function of the percolation parameter p we obtain a first order expansion of the speed at p 1 which proves that percolating slows down the random walk at least in the case where the drift is along a component of the lattice

variational problems with percolation dilute spin systems at zero temperature we study the asymptotic behavior of dilute spin lattice energies by exhibiting a continuous interfacial limit energy computed using the notion of convergence and techniques mixing geometric measure theory and percolation while scaling to zero the lattice spacing the limit is not trivial above a percolation threshold since the lattice energies are not equi coercive a suitable notion of limit magnetization must be defined which can be characterized by two phases separated by an interface the macroscopic surface tension at this interface is characterized through a first passage percolation formula which highlights interesting connections between variational problems and percolation issues a companion result on the asymptotic description on energies defined on paths in a dilute environment is also given

stopping criterion for boosting based data reduction techniques from binary to multiclass problem so far boosting has been used to improve the quality of moderately accurate learning algorithms by weighting and combining many of their weak hypotheses into a final classifier with theoretically high accuracy in a recent work sebban nock and lallich 2001 we have attempted to adapt boosting properties to data reduction techniques in this particular context the objective was not only to improve the success rate but also to reduce the time and space complexities due to the storage requirements of some costly learning algorithms such as nearest neighbor classifiers in that framework each weak hypothesis which is usually built and weighted from the learning set is replaced by a single learning instance the weight given by boosting defines in that case the relevance of the instance and a statistical test allows one to decide whether it can be discarded without damaging further classification tasks in sebban nock and lallich 2001 we addressed problems with two classes it is the aim of the present paper to relax the class constraint and extend our contribution to multiclass problems beyond data reduction experimental results are also provided on twenty three datasets showing the benefits that our boosting derived weighting rule brings to weighted nearest neighbor classifiers

risk bounds for statistical learning we propose a general theorem providing upper bounds for the risk of an empirical risk minimizer erm we essentially focus on the binary classification framework we extend tsybakov s analysis of the risk of an erm under margin type conditions by using concentration inequalities for conveniently weighted empirical processes this allows us to deal with ways of measuring the size of a class of classifiers other than entropy with bracketing as in tsybakov s work in particular we derive new risk bounds for the erm when the classification rules belong to some vc class under margin conditions and discuss the optimality of these bounds in a minimax sense

optimal aggregation of classifiers in statistical learning classification can be considered as nonparametric estimation of sets where the risk is defined by means of a specific distance between sets associated with misclassification error it is shown that the rates of convergence of classifiers depend on two parameters the complexity of the class of candidate sets and the margin parameter the dependence is explicitly given indicating that optimal fast rates approaching o n 1 can be attained where n is the sample size and that the proposed classifiers have the property of robustness to the margin the main result of the paper concerns optimal aggregation of classifiers we suggest a classifier that automatically adapts both to the complexity and to the margin and attains the optimal fast rates up to a logarithmic factor

smooth discrimination analysis ties f and g can be based on the estimation of the set g x fx gx we consider applications where it is appropriate to assume that the region g has a smooth boundary or belongs to another nonparametric class of sets in particular this assumption makes sense if discrimination is used as a data analytic tool decision rules based on minimization of empirical risk over the whole class of sets and over sieves are considered their rates of convergence are obtained we show that these rules achieve optimal rates for estimation of g and optimal rates of convergence for bayes risks an interesting conclusion is that the optimal rates for bayes risks can be very fast in particular faster than the parametric root n rate these fast rates cannot be guaranteed for plug in rules

efficient agnostic learning of neural networks with bounded fan in we show that the class of two layer neural networks with bounded fan in is efficiently learnable in a realistic extension to the probably approximately correct pac learning model in this model a joint probability distribution is assumed to exist on the observations and the learner is required to approximate the neural network which minimizes the expected quadratic error as special cases the model allows learning real valued functions with bounded noise learning probabilistic concepts and learning the best approximation to a target function that cannot be well approximated by the neural network the networks we consider have real valued inputs and outputs an unlimited number of threshold hidden units with bounded fan in and a bound on the sum of the absolute values of the output weights the number of computation steps of the learning algorithm is bounded by a polynomial in 1 spl epsiv 1 spl delta n and b where spl epsiv is the desired accuracy spl delta is the probability that the algorithm fails n is the input dimension and b is the bound on both the absolute value of the target which may be a random variable and the sum of the absolute values of the output weights in obtaining the result we also extended some results on iterative approximation of functions in the closure of the convex hull of a function class and on the sample complexity of agnostic learning with the quadratic loss function

a simple lemma on greedy approximation in hilbert space and convergence rates for projection pursuit regression and neural network training a general convergence criterion for certain iterative sequences in hilbert space is presented for an important subclass of these sequences estimates of the rate of convergence are given under very mild assumptions these results establish an 0 1 f4n nonsampling convergence rate for projection pursuit regression and neural network training where n represents the number of ridge functions neurons or coefficients in a greedy basis expansion

classification by pairwise coupling we discuss a strategy for polychotomous classification that involves estimating class probabilities for each pair of classes and then coupling the estimates together the coupling model is similar to the bradley terry method for paired comparisons we study the nature of the class probability estimates that arise and examine the performance of the procedure in real and simulated data sets classifiers used include linear discriminants nearest neighbors adaptive nonlinear methods and the support vector machine

classification algorithms using adaptive partitioning algorithms for binary classification based on adaptive tree partitioning are formulated and analyzed for both their risk performance and their friendliness to numerical implementation the algorithms can be viewed as generating a set approximation to the bayes set and thus fall into the general category of set estimators in contrast with the most studied tree based algorithms which utilize piecewise constant approximation on the generated partition ieee trans inform theory52 2006 1335 1353 mach learn 66 2007 209 242 we consider decorated trees which allow us to derive higher order methods convergence rates for these methods are derived in terms the parameter alpha of margin conditions and a rate s of best approximation of the bayes set by decorated adaptive partitions they can also be expressed in terms of the besov smoothness beta of the regression function that governs its approximability by piecewise polynomials on adaptive partition the execution of the algorithms does not require knowledge of the smoothness or margin conditions besov smoothness conditions are weaker than the commonly used holder conditions which govern approximation by nonadaptive partitions and therefore for a given regression function can result in a higher rate of convergence this in turn mitigates the compatibility conflict between smoothness and margin parameters

surface order large deviations for high density percolation we derive surface order large deviation estimates for the volume of the largest cluster and for the volume of the largest region surrounded by a cluster of a bernoulli percolation process restricted to a big finite box with sufficiently large parameter we also establish a useful version of the isoperimetric inequality which is the main tool of our proofs

the supercritical phase of percolation is well behaved we prove a general result concerning the critical probabilities of subsets of a lattice l it is a consequence of this result that the critical probability of a percolation process on l equals the limit of the critical probability of a slice of l as the thickness of the slice tends to infinity this verification of one of the standard hypotheses of the subject settles many questions concerning supercritical percolation

large deviations for the chemical distance in supercritical bernoulli percolation the chemical distance d x y is the length of the shortest open path between two points x and y in an infinite bernoulli percolation cluster in this work we study the asymptotic behavior of this random metric and we prove that for an appropriate norm depending on the dimension and the percolation parameter the probability of the event 0 x d 0 x x 1 e 1 e exponentially decreases when x 1 tends to infinity from this bound we also derive a large deviation inequality for the corresponding asymptotic shape result

large deviations for the contact process and two dimensional percolation summary the following results are proved 1 for the upper invariant measure of the basic one dimensional supercritical contact process the density of l s has the usual large deviation behavior the probability of a large deviation decays exponentially with the number of sites considered 2 for supercritical two dimensional nearest neighbor site or bond percolation the density ya of sites inside a square a which belong to the infinite cluster has the following large deviation properties the probability that ya deviates from its expected value by a positive amount decays exponentially with the area of a while the probability that it deviates from its expected value by a negative amount decays exponentially with the perimeter of a these two problems are treated together in this paper because similar techniques renormalization are used for both

self organized segregation on the grid we consider an agent based model with exponentially distributed waiting times in which two types of agents interact locally over a graph and based on this interaction and on the value of a common intolerance threshold tau decide whether to change their types this is equivalent to a zero temperature ising model with glauber dynamics an asynchronous cellular automaton with extended moore neighborhoods or a schelling model of self organized segregation in an open system and has applications in the analysis of social and biological networks and spin glasses systems some rigorous results were recently obtained in the theoretical computer science literature and this work provides several extensions we enlarge the intolerance interval leading to the expected formation of large segregated regions of agents of a single type from the known size epsilon 0 to size approx 0 134 namely we show that for 0 433 tau 1 2 and by symmetry 1 2 tau 0 567 the expected size of the largest segregated region containing an arbitrary agent is exponential in the size of the neighborhood we further extend the interval leading to expected large segregated regions to size approx 0 312 considering almost segregated regions namely regions where the ratio of the number of agents of one type and the number of agents of the other type vanishes quickly as the size of the neighborhood grows in this case we show that for 0 344 tau le 0 433 and by symmetry for 0 567 le tau 0 656 the expected size of the largest almost segregated region containing an arbitrary agent is exponential in the size of the neighborhood this behavior is reminiscent of supercritical percolation where small clusters of empty sites can be observed within any sufficiently large region of the occupied percolation cluster the exponential bounds that we provide also imply that complete segregation where agents of a single type cover the whole grid does not occur with high probability for p 1 2 and the range of intolerance considered

reducing multiclass to binary a unifying approach for margin classifiers we present a unifying framework for studying the solution of multiclass categorization problems by reducing them to multiple binary problems that are then solved using a margin based binary learning algorithm the proposed framework unifies some of the most popular approaches in which each class is compared against all others or in which all pairs of classes are compared to each other or in which output codes with error correcting properties are used we propose a general method for combining the classifiers generated on the binary problems and we prove a general empirical multiclass loss bound given the empirical loss of the individual binary learning algorithms the scheme and the corresponding bounds apply to many popular classification learning algorithms including support vector machines adaboost regression logistic regression and decision tree algorithms we also give a multiclass generalization error analysis for general output codes with adaboost as the binary learner experimental results with svm and adaboost show that our scheme provides a viable alternative to the most commonly used multiclass algorithms

on the chemical distance for supercritical bernoulli percolation we prove large deviation estimates at the correct order for the graph distance of two sites lying in the same cluster of an independent percolation process we improve earlier results of gartner and molchanov and grimmett and marstrand and answer affirmatively a conjecture of kozlov

homogenization of surface and length energies for spin systems abstract we study the homogenization of lattice energies related to ising systems of the form e e u i j c i j e u i u j with u i a spin variable indexed on the portion of a cubic lattice e z d by computing their limit in the framework of surface energies in a bv setting we introduce a notion of homogenizability of the system c i j e that allows to treat periodic almost periodic and random statistically homogeneous models the latter in dimension two when the coefficients are positive ferromagnetic energies in which case the limit energy is finite on bv 1 and takes the form f u u 1 d h d 1 is the normal to u 1 where is characterized by an asymptotic formula in the random case can be expressed in terms of first passage percolation characteristics the result is extended to coefficients with varying sign under the assumption that the areas where the energies are antiferromagnetic are well separated finally we prove a dual result for discrete curves

on the rate of convergence of local averaging plug in classification rules under a margin condition the rates of convergence of plug in kernel partitioning and nearest neighbors classification rules are analyzed a margin condition which measures how quickly the a posteriori probabilities cross the decision boundary smoothness conditions on the a posteriori probabilities and boundedness of the feature vector are imposed the rates of convergence of the plug in classifiers shown in this paper are faster than previously known

on the asymptotic probability of error in nonparametric discrimination let x y x 1 y 1 cdots x n y n be independent identically distributed random vectors from r d times 0 1 and let hat y be the k nearest neighbor estimate of y from x and the x i y i s we show that for all distributions of x y the limit of l n p hat y neq y exists and satisfies lim n rightarrow infty l n leq 1 a k r ast a k frac alpha sqrt k k 3 25 big 1 frac beta sqrt k 3 big k text odd k geq 5 where r ast is the bayes probability of error and alpha 0 3399 cdots and beta 0 9749 cdots are universal constants this bound is shown to be best possible in a certain sense

fast learning rates for plug in classifiers it has been recently shown that under the margin or low noise assumption there exist classifiers attaining fast rates of convergence of the excess bayes risk that is rates faster than n 1 2 the work on this subject has suggested the following two conjectures i the best achievable fast rate is of the order n 1 and ii the plug in classifiers generally converge more slowly than the classifiers based on empirical risk minimization we show that both conjectures are not correct in particular we construct plug in classifiers that can achieve not only fast but also super fast rates that is rates faster than n 1 we establish minimax lower bounds showing that the obtained rates cannot be improved

risk bounds for cart classifiers under a margin condition non asymptotic risk bounds for classification and regression trees cart classifiers are obtained in the binary supervised classification framework under a margin assumption on the joint distribution of the covariates and the labels these risk bounds are derived conditionally on the construction of the maximal binary tree and allow to prove that the linear penalty used in the cart pruning algorithm is valid under the margin condition it is also shown that conditionally on the construction of the maximal tree the final selection by test sample does not alter dramatically the estimation accuracy of the bayes classifier

the correlation length for the high density phase of bernoulli percolation l auteur examine 2 types standards de fonctions de connectivite dans la phase de densite elevee de filtration de bernoulli du plus proche voisin il demontre que les 2 quantites decroissent exponentiellement avec le meme taux constant la reciproque de cette constante definit donc une longueur de correlation pour laquelle on n arrive pas a montrer qu elle est finie au dessus d une densite critique associee a la transition de phases des proprietes de semicontinuite et de convexite sont montrees pour cette longueur de correlation et on obtient des resultats analogues a ceux concernant le modele d ising en deux dimensions

spectral connectivity analysis spectral kernel methods are techniques for mapping data into a coordinate system that efficiently reveals the geometric structure in particular the connectivity of the data these methods depend on tuning parameters we analyze the dependence of the method on these tuning parameters we focus on one particular technique diffusion maps but our analysis can be used for other spectral methods as well we identify the key population quantities we define an appropriate risk function for analyzing the estimators and we explain how these methods relate to classical kernel smoothing we also show that in some cases fast rates of convergence are possible even in high dimensions the appendix of the article is available online as supplementary materials

the rate of convergence of k_n nn regression estimates and classification rules corresp the rate of convergence of k n nn regression estimates and the corresponding multiple classification error are calculated without assuming the existence of the density of the observations

on the strong universal consistency of nearest neighbor regression function estimates two results are presented concerning the consistency of the k nearest neighbor regression estimate we show that all modes of convergence in l 1 in probability almost sure complete are equivalent if the regression variable is bounded under the additional condition k log n we also obtain the strong universal consistency of the estimate

necessary and sufficient conditions for the pointwise convergence of nearest neighbor regression function estimates where v 1 v is a given probability vector and xt x yl x x x y x is a permutation of x1 i71 x y according to increasing values of ilxi x i x e r a when ixi xll h x j x l but i 1 and some constant c 0