refined error bounds for several learning algorithms this article studies the achievable guarantees on the error rates of certain learning algorithms with particular focus on refining logarithmic factors many of the results are based on a general technique for obtaining bounds on the error rates of sample consistent classifiers with monotonic error regions in the realizable case we prove bounds of this type expressed in terms of either the vc dimension or the sample compression size this general technique also enables us to derive several new bounds on the error rates of general sample consistent learning algorithms as well as refined bounds on the label complexity of the cal active learning algorithm additionally we establish a simple necessary and sufficient condition for the existence of a distribution free bound on the error rates of all sample consistent learning rules converging at a rate inversely proportional to the sample size we also study learning in the presence of classification noise deriving a new excess error rate guarantee for general vc classes under tsybakov s noise condition and establishing a simple and general necessary and sufficient condition for the minimax excess risk under bounded noise to converge at a rate inversely proportional to the sample size

boosting the margin a new explanation for the effectiveness of voting methods one of the surprising recurring phenomena observed in experiments with boosting is that the test error of the generated classifier usually does not increase as its size becomes very large and often is observed to decrease even after the training error reaches zero in this paper we show that this phenomenon is related to the distribution of margins of the training examples with respect to the generated voting classification rule where the margin of an example is simply the difference between the number of correct votes and the maximum number of votes received by any incorrect label we show that techniques used in the analysis of vapnik s support vector classifiers and of neural networks with small weights can be applied to voting methods to relate the margin distribution to the test error we also show theoretically and experimentally that boosting is especially effective at increasing the margins of the training examples finally we compare our explanation to those based on the bias variance decomposition

stabilized nearest neighbor classifier and its statistical properties abstractthe stability of statistical analysis is an important indicator for reproducibility which is one main principle of the scientific method it entails that similar statistical conclusions can be reached based on independent samples from the same underlying population in this article we introduce a general measure of classification instability cis to quantify the sampling variability of the prediction made by a classification method interestingly the asymptotic cis of any weighted nearest neighbor classifier turns out to be proportional to the euclidean norm of its weight vector based on this concise form we propose a stabilized nearest neighbor snn classifier which distinguishes itself from other nearest neighbor classifiers by taking the stability into consideration in theory we prove that snn attains the minimax optimal convergence rate in risk and a sharp convergence rate in cis the latter rate result is established for general plug in classifiers under a low noise condition ext

aggregation of svm classifiers using sobolev spaces this paper investigates statistical performances of support vector machines svm and considers the problem of adaptation to the margin parameter and to complexity in particular we provide a classifier with no tuning parameter it is a combination of svm classifiers our contribution is two fold 1 we propose learning rates for svm using sobolev spaces and build a numerically realizable aggregate that converges with same rate 2 we present practical experiments of this method of aggregation for svm using both sobolev spaces and gaussian kernels

risk bounds for embedded variable selection in classification trees the problems of model and variable selections for classification trees are jointly considered a penalized criterion is proposed which explicitly takes into account the number of variables and a risk bound inequality is provided for the tree classifier minimizing this criterion this penalized criterion is compared to the one used during the pruning step of the cart algorithm it is shown that the two criteria are similar under some specific margin assumptions in practice the tuning parameter of the cart penalty has to be calibrated by hold out or cross validation a simulation study is performed to compare the form of the theoretical penalized criterion we propose with the form obtained after tuning the regularization parameter via cross validation

the multi armed bandit problem with covariates we consider a multi armed bandit problem in a setting where each arm produces a noisy reward realization which depends on an observable random covariate as opposed to the traditional static multi armed bandit problem this setting allows for dynamically changing rewards that better describe applications where side information is available we adopt a nonparametric model where the expected rewards are smooth functions of the covariate and where the hardness of the problem is captured by a margin parameter to maximize the expected cumulative reward we introduce a policy called adaptively binned successive elimination abse that adaptively decomposes the global problem into suitably localized static bandit problems this policy constructs an adaptive partition using a variant of the successive elimination se policy our results include sharper regret bounds for the se policy in a static bandit problem and minimax optimal regret bounds for the abse policy in the dynamic problem

linearity measure for curve segments in this paper we consider linearity measure for a bounded length curves first we define a new linearity measure for open curve segments and then extend method to closed curves contours the derived measures for both open curve segments and closed curves are invariant with respect to similarity transformations the linearity measure for open curve segments picks the value 1 if and only if the measured open line segment is a perfect straight line segment while the established linearity measures for closed curves never reach 1 as preferred

minimax analysis of active learning this work establishes distribution free upper and lower bounds on the minimax label complexity of active learning with general hypothesis classes under various noise models the results reveal a number of surprising facts in particular under the noise model of tsybakov 2004 the minimax label complexity of active learning with a vc class is always asymptotically smaller than that of passive learning and is typically signi cantly smaller than the best previously published upper bounds in the active learning literature in high noise regimes it turns out that all active learning problems of a given vc dimension have roughly the same minimax label complexity which contrasts with well known results for bounded noise in low noise regimes we find that the label complexity is well characterized by a simple combinatorial complexity measure we call the star number interestingly we find that almost all of the complexity measures previously explored in the active learning literature have worst case values exactly equal to the star number we also propose new active learning strategies that nearly achieve these minimax label complexities

data driven calibration of penalties for least squares regression penalization procedures often suffer from their dependence on multiplying factors whose optimal values are either unknown or hard to estimate from data we propose a completely data driven calibration algorithm for these parameters in the least squares regression framework without assuming a particular shape for the penalty our algorithm relies on the concept of minimal penalty recently introduced by birge and massart 2007 in the context of penalized least squares for gaussian homoscedastic regression on the positive side the minimal penalty can be evaluated from the data themselves leading to a data driven estimation of an optimal penalty which can be used in practice on the negative side their approach heavily relies on the homoscedastic gaussian nature of their stochastic framework the purpose of this paper is twofold stating a more general heuristics for designing a data driven penalty the slope heuristics and proving that it works for penalized least squares regression with a random design even for heteroscedastic non gaussian data for technical reasons some exact mathematical results will be proved only for regressogram bin width selection this is at least a first step towards further results since the approach and the method that we use are indeed general

asymptotics in empirical risk minimization in this paper we study a two category classification problem we indicate the categories by labels y 1 and y 1 we observe a covariate or feature x x d consider a collection ha of classifiers indexed by a finite dimensional parameter a and the classifier ha that minimizes the prediction error over this class the parameter a is estimated by the empirical risk minimizer n over the class where the empirical risk is calculated on a training sample of size n we apply the kim pollard theorem to show that under certain differentiability assumptions n converges to a with rate n 1 3 and also present the asymptotic distribution of the renormalized estimator for example let v0 denote the set of x on which given x x the label y 1 is more likely than the label y 1 if x is one dimensional the set v0 is the union of disjoint intervals the problem is then to estimate the thresholds of the intervals we obtain the asymptotic distribution of the empirical risk minimizer when the classifiers have k thresholds where k is fixed we furthermore consider an extension to higher dimensional x assuming basically that v0 has a smooth boundary in some given parametric class we also discuss various rates of convergence when the differentiability conditions are possibly violated here we again restrict ourselves to one dimensional x we show that the rate is n 1 in certain cases and then also obtain the asymptotic distribution for the empirical prediction error

generalized density clustering we study generalized density based clustering in which sharply defined clusters such as clusters on lower dimensional manifolds are allowed we show that accurate clustering is possible even in high dimensions we propose two data based methods for choosing the bandwidth and we study the stability properties of density clusters we show that a simple graph based algorithm successfully approximates the high density clusters

rates of convergence for minimum contrast estimators we shall present here a general study of minimum contrast estimators in a nonparametric setting although our results are also valid in the classical parametric case for independent observations these estimators include many of the most popular estimators in various situations such as maximum likelihood estimators least squares and other estimators of the regression function estimators for mixture models or deconvolution the main theorem relates the rate of convergence of those estimators to the entropy structure of the space of parameters optimal rates depending on entropy conditions are already known at least for some of the models involved and they agree with what we get for minimum contrast estimators as long as the entropy counts are not too large but under some circumstances large entropies or changes in the entropy structure due to local perturbations the resulting the rates are only suboptimal counterexamples are constructed which show that the phenomenon is real for non parametric maximum likelihood or regression this proves that under purely metric assumptions our theorem is optimal and that minimum contrast estimators happen to be suboptimal

quantization via empirical divergence maximization empirical divergence maximization edm refers to a recently proposed strategy for estimating f divergences and likelihood ratio functions this paper extends the idea to empirical vector quantization where one seeks to empirically derive quantization rules that maximize the kullback leibler divergence between two statistical hypotheses we analyze the estimator s error convergence rate leveraging tsybakov s margin condition and show that rates as fast as n 1 are possible where n equals the number of training samples we also show that the flynn and gray algorithm can be used to efficiently compute edm estimates and show that they can be efficiently and accurately represented by recursive dyadic partitions the edm formulation have several advantages first the formulation gives access to the tools and results of empirical process theory that quantify the estimator s error convergence rate second the formulation provides a previously unknown derivation for the flynn and gray algorithm third the flexibility it affords allows one to avoid a small cell assumption common in other approaches finally we illustrate the potential use of the method through an example

regression level set estimation via cost sensitive classification regression level set estimation is an important yet understudied learning task it lies somewhere between regression function estimation and traditional binary classification and in many cases is a more appropriate setting for questions posed in these more common frameworks this note explains how estimating the level set of a regression function from training examples can be reduced to cost sensitive classification we discuss the theoretical and algorithmic benefits of this learning reduction demonstrate several desirable properties of the associated risk and report experimental results for histograms support vector machines and nearest neighbor rules on synthetic and real data

artificial fractal dimension obtained by using perimeter area relationship on digitalized images perimeter area relation is a widely used method to obtain the perimeter fractal dimension of self similar shapes embedded into a 2 dimensional space in this short paper we would like to demonstrate that the orientation anisotropy of the shapes which cannot be excluded in a natural sample can cause an error in the calculation of the perimeters and areas of the digitalized images in this way one might obtain artificial fractal dimension therefore the applicability of the perimeter area method seems to be very limited mited

risk bounds for statistical learning we propose a general theorem providing upper bounds for the risk of an empirical risk minimizer erm we essentially focus on the binary classification framework we extend tsybakov s analysis of the risk of an erm under margin type conditions by using concentration inequalities for conveniently weighted empirical processes this allows us to deal with ways of measuring the size of a class of classifiers other than entropy with bracketing as in tsybakov s work in particular we derive new risk bounds for the erm when the classification rules belong to some vc class under margin conditions and discuss the optimality of these bounds in a minimax sense

adaptive noisy clustering the problem of adaptive noisy clustering is investigated given a set of noisy observations z i x i epsilon i i 1 ldots n the goal is to design clusters associated with the law of x i s with unknown density f with respect to the lebesgue measure since we observe a corrupted sample a direct approach as the popular k means is not suitable in this case in this paper we propose a noisy k means minimization which is based on the k means loss function and a deconvolution estimator of the density f in particular this approach suffers from the dependence on a bandwidth involved in the deconvolution kernel fast rates of convergence for the excess risk are proposed for a particular choice of the bandwidth which depends on the smoothness of the density f then we turn out into the main issue of this paper the data driven choice of the bandwidth we state an adaptive upper bound using a modified version of lespki s method called empirical risk comparison where empirical risks associated with different bandwidths are compared eventually we illustrate that the selection rule can be used in many statistical problems of m estimation where the empirical risk depends on a nuisance parameter

probability inequalities for empirical processes and a law of the iterated logarithm given a class f of functions on x we can view vn as a stochastic process indexed by f and consider limit theorems for this process to prove such theorems it is often helpful to have bounds on the tail of the r v sup w i n f i see for example dudley 1978 kuelbs and dudley 1980 dudley and phillip 1983 or gine and zinn 1983 our main question of interest is for what classes f can best possible bounds be obtained so first we must ask what are these best possible bounds of course

on estimation of monotone and concave frontier functions abstract when analyzing the productivity of firms one may want to compare how the firms transform a set of inputs x typically labor energy or capital into an output y typically a quantity of goods produced the economic efficiency of a firm is then defined in terms of its ability to operate close to or on the production frontier the boundary of the production set the frontier function gives the maximal level of output attainable by a firm for a given combination of its inputs the efficiency of a firm may then be estimated via the distance between the attained production level and the optimal level given by the frontier function from a statistical viewpoint the frontier function may be viewed as the upper boundary of the support of the population of firms density in the input and output space it is often reasonable to assume that the production frontier is a concave monotone function then a famous estimator in the univariate input and output case is the data envelopment analysis dea estimato

minimum clinically important difference in medical studies summary in clinical trials minimum clinically important difference mcid has attracted increasing interest as an important supportive clinical and statistical inference tool many estimation methods have been developed based on various intuitions while little theoretical justification has been established this article proposes a new estimation framework of the mcid using both diagnostic measurements and patient reported outcomes pros the framework first formulates the population based mcid as a large margin classification problem and then extends to the personalized mcid to allow individualized thresholding value for patients whose clinical profiles may affect their pro responses more importantly the proposed estimation framework is showed to be asymptotically consistent and a finite sample upper bound is established for its prediction accuracy compared against the ideal mcid the advantage of our proposed method is also demonstrated in a variety of simulated experiments as well as two phase 3 clinical trials

classification with gaussians and convex loss this paper considers binary classification algorithms generated from tikhonov regularization schemes associated with general convex loss functions and varying gaussian kernels our main goal is to provide fast convergence rates for the excess misclassification error allowing varying gaussian kernels in the algorithms improves learning rates measured by regularization error and sample error special structures of gaussian kernels enable us to construct by a nice approximation scheme with a fourier analysis technique uniformly bounded regularizing functions achieving polynomial decays of the regularization error under a sobolev smoothness condition the sample error is estimated by using a projection operator and a tight bound for the covering numbers of reproducing kernel hilbert spaces generated by gaussian kernels the convexity of the general loss function plays a very important role in our analysis

synchronization and transient stability in power networks and nonuniform kuramoto oscillators motivated by recent interest for multiagent systems and smart grid architectures we discuss the synchronization problem for the network reduced model of a power system with nontrivial transfer conductances our key insight is to exploit the relationship between the power network model and a first order model of coupled oscillators assuming overdamped generators possibly due to local excitation controllers a singular perturbation analysis shows the equivalence between the classic swing equations and a nonuniform kuramoto model here nonuniform kuramoto oscillators are characterized by multiple time constants nonhomogeneous coupling and nonuniform phase shifts extending methods from transient stability synchronization theory and consensus protocols we establish sufficient conditions for synchronization of nonuniform kuramoto oscillators these conditions reduce to necessary and sufficient tests for the standard kuramoto model combining our singular perturbation and kuramoto analyses we derive

stability of density based clustering high density clusters can be characterized by the connected components of a level set l x p x of the underlying probability density function p generating the data at some appropriate level 0 the complete hierarchical clustering can be characterized by a cluster tree t l in this paper we study the behavior of a density level set estimate l and cluster tree estimate t based on a kernel density estimator with kernel bandwidth h we define two notions of instability to measure the variability of l and t as a function of h and investigate the theoretical properties of these instability measures

multicategory ψ learning in binary classification margin based techniques usually deliver high performance as a result a multicategory problem is often treated as a sequence of binary classifications in the absence of a dominating class this treatment may be suboptimal and may yield poor performance such as for support vector machines svms we propose a novel multicategory generalization of learning that treats all classes simultaneously the new generalization eliminates this potential problem while at the same time retaining the desirable properties of its binary counterpart we develop a statistical learning theory for the proposed methodology and obtain fast convergence rates for both linear and nonlinear learning examples we demonstrate the operational characteristics of this method through a simulation our results indicate that the proposed methodology can deliver accurate class prediction and is more robust against extreme observations than its svm counterpart

rademacher penalties and structural risk minimization we suggest a penalty function to be used in various problems of structural risk minimization this penalty is data dependent and is based on the sup norm of the so called rademacher process indexed by the underlying class of functions sets the standard complexity penalties used in learning problems and based on the vc dimensions of the classes are conservative upper bounds in a probabilistic sense uniformly over the set of all underlying distributions for the penalty we suggest thus for a particular distribution of training examples one can expect better performance of learning algorithms with the data driven rademacher penalties we obtain oracle inequalities for the theoretical risk of estimators obtained by structural minimization of the empirical risk with rademacher penalties the inequalities imply some form of optimality of the empirical risk minimizers we also suggest an iterative approach to structural risk minimization with rademacher penalties in which the hierarchy of classes is not given in advance but is determined in the data driven iterative process of risk minimization we prove probabilistic oracle inequalities for the theoretical risk of the estimators based on this approach as well

asymptotical minimax recovery of sets with smooth boundaries in this paper optimal rates of convergence are derived for estimates of sets in n dimensional black and white pictures under smoothness conditions it is assumed that the boundaries of the black regions have a smooth parameterisation that is that the boundaries are given by smooth functions from the sphere s n 1 into r n furthermore classes of convex regions are considered two models are studied edge estimation models motivated by image segmentation problems and density support estimation

optimal aggregation of classifiers in statistical learning classification can be considered as nonparametric estimation of sets where the risk is defined by means of a specific distance between sets associated with misclassification error it is shown that the rates of convergence of classifiers depend on two parameters the complexity of the class of candidate sets and the margin parameter the dependence is explicitly given indicating that optimal fast rates approaching o n 1 can be attained where n is the sample size and that the proposed classifiers have the property of robustness to the margin the main result of the paper concerns optimal aggregation of classifiers we suggest a classifier that automatically adapts both to the complexity and to the margin and attains the optimal fast rates up to a logarithmic factor

smooth discrimination analysis ties f and g can be based on the estimation of the set g x fx gx we consider applications where it is appropriate to assume that the region g has a smooth boundary or belongs to another nonparametric class of sets in particular this assumption makes sense if discrimination is used as a data analytic tool decision rules based on minimization of empirical risk over the whole class of sets and over sieves are considered their rates of convergence are obtained we show that these rules achieve optimal rates for estimation of g and optimal rates of convergence for bayes risks an interesting conclusion is that the optimal rates for bayes risks can be very fast in particular faster than the parametric root n rate these fast rates cannot be guaranteed for plug in rules

a plug in approach to neyman pearson classification the neyman pearson np paradigm in binary classification treats type i and type ii errors with different priorities it seeks classifiers that minimize type ii error subject to a type i error constraint under a user specified level in this paper plug in classifiers are developed under the np paradigm based on the fundamental neyman pearson lemma we propose two related plug in classifiers which amount to thresholding respectively the class conditional density ratio and the regression function these two classifiers handle different sampling schemes this work focuses on theoretical properties of the proposed classifiers in particular we derive oracle inequalities that can be viewed as finite sample versions of risk bounds np classification can be used to address anomaly detection problems where asymmetry in errors is an intrinsic property as opposed to a common practice in anomaly detection that consists of thresholding normal class density our approach does not assume a specific form for anomaly distributions such consideration is particularly necessary when the anomaly class density is far from uniformly distributed

combining pac bayesian and generic chaining bounds there exist many different generalization error bounds in statistical learning theory each of these bounds contains an improvement over the others for certain situations or algorithms our goal is first to underline the links between these bounds and second to combine the different improvements into a single bound in particular we combine the pac bayes approach introduced by mcallester 1998 which is interesting for randomized predictions with the optimal union bound provided by the generic chaining technique developed by fernique and talagrand see talagrand 1996 in a way that also takes into account the variance of the combined functions we also show how this connects to rademacher based bounds

local likelihood tracking of fault lines and boundaries summary we suggest locally parametric methods for estimating curves such as boundaries of density supports or fault lines in response surfaces in a variety of spatial problems the methods are based on spatial approximations to the local likelihood that the curve passes through a given point in the plane as a function of that point the local likelihood might be a regular likelihood computed locally with kernel weights e g in the case of support boundary estimation or a local version of a likelihood ratio statistic e g in fault line estimation in either case the local likelihood surface represents a function which is relatively large near the target curve and relatively small elsewhere therefore the curve may be estimated as a ridge line of the surface we require only a numerical algorithm for tracking the projection of a ridge into the plane this approach offers several potential advantages over alternative methods first the local log likelihood surface can be graphed and the degree of ridginess assessed visually to determine how the level of local smoothing should be varied in different spatial locations in order to emphasize the ridge and hence the curve adequately secondly the local likelihood surface does not need to be computed in anything like its entirety once we have a reasonable approximation to a point on the curve we may track it by numerically walking along the ridge line thirdly the method is appropriate without change for many different types of spatial explanatory vanables gridded stochastic or otherwise three examples are explored in detail fault lines in response surfaces and in intensity or density surfaces and boundaries of supports of probability densities

minimax estimation of the volume of a set under the rolling ball condition abstractwe consider the problem of estimating the volume of a compact domain in a euclidean space based on a uniform sample from the domain we assume that the domain has a boundary with positive reach we propose a data splitting approach to correct the bias of the plug in estimator based on the sample convex hull we show that this simple estimator achieves a minimax lower bound that we derive some numerical experiments corroborate our theoretical findings

rademacher complexities and bounding the excess risk in active learning sequential algorithms of active learning based on the estimation of the level sets of the empirical risk are discussed in the paper localized rademacher complexities are used in the algorithms to estimate the sample sizes needed to achieve the required accuracy of learning in an adaptive way probabilistic bounds on the number of active examples have been proved and several applications to binary classification problems are considered

representation and compression of multidimensional piecewise functions using surflets we study the representation approximation and compression of functions in m dimensions that consist of constant or smooth regions separated by smooth m 1 dimensional discontinuities examples include images containing edges video sequences of moving objects and seismic data containing geological horizons for both function classes we derive the optimal asymptotic approximation and compression rates based on kolmogorov metric entropy for piecewise constant functions we develop a multiresolution predictive coder that achieves the optimal rate distortion performance for piecewise smooth functions our coder has near optimal rate distortion performance our coder for piecewise constant functions employs surflets a new multiscale geometric tiling consisting of m dimensional piecewise constant atoms containing polynomial discontinuities our coder for piecewise smooth functions uses surfprints which wed surflets to wavelets for piecewise smooth approximation both of these schemes achieve the optimal asymptotic approximation performance key features of our algorithms are that they carefully control the potential growth in surflet parameters at higher smoothness and do not require explicit estimation of the discontinuity we also extend our results to the corresponding discrete function spaces for sampled data we provide asymptotic performance results for both discrete function spaces and relate this asymptotic performance to the sampling rate and smoothness orders of the underlying functions and discontinuities for approximation of discrete data we propose a new scale adaptive dictionary that contains few elements at coarse and fine scales but many elements at medium scales simulation results on synthetic signals provide a comparison between surflet based coders and previously studied approximation schemes based on wedgelets and wavelets

empirical margin distributions and bounding the generalization error of combined classifiers we prove new probabilistic upper bounds on generalization error of complex classifiers that are combinations of simple classifiers such combinations could be implemented by neural networks or by voting methods of combining the classifiers such as boosting and bagging the bounds are in terms of the empirical distribution of the margin of the combined classifier they are based on the methods of the theory of gaussian and empirical processes comparison inequalities symmetrization method concentration inequalities and they improve previous results of bartlett 1998 on bounding the generalization error of neural networks in terms of ell 1 norms of the weights of neurons and of schapire freund bartlett and lee 1998 on bounding the generalization error of boosting we also obtain rates of convergence in levy distance of empirical margin distribution to the true margin distribution uniformly over the classes of classifiers and prove the optimality of these rates

minimax optimal level set estimation this paper describes a new methodology and associated theoretical analysis for rapid and accurate extraction of level sets of a multivariate function from noisy data the identification of the boundaries of such sets is an important theoretical problem with applications for digital elevation maps medical imaging and pattern recognition this problem is significantly different from classical segmentation because level set boundaries may not correspond to singularities or edges in the underlying function as a result segmentation methods which rely upon detecting boundaries would be potentially ineffective in this regime this issue is addressed in this paper through a novel error metric sensitive to both the error in the location of the level set estimate and the deviation of the function from the critical level hoeffding s inequality is used to derive a novel regularization term that is distinctly different from regularization methods used in conventional image denoising settings building upon this foundation it is possible to derive error performance bounds for the proposed estimator and demonstrate that it exhibits near minimax optimal error decay rates for large classes of level set problems the proposed method automatically adapts to the spatially varying regularity of both the boundary of the level set and the underlying function

fast rates in statistical and online learning the speed with which a learning algorithm converges as it is presented with more data is a central problem in machine learning a fast rate of convergence means less data is needed for the same level of performance the pursuit of fast rates in online and statistical learning has led to the discovery of many conditions in learning theory under which fast learning is possible we show that most of these conditions are special cases of a single unifying condition that comes in two forms the central condition for proper learning algorithms that always output a hypothesis in the given model and stochastic mixability for online algorithms that may make predictions outside of the model we show that under surprisingly weak assumptions both conditions are in a certain sense equivalent the central condition has a re interpretation in terms of convexity of a set of pseudoprobabilities linking it to density estimation under misspecification for bounded losses we show how the central condition enables a direct proof of fast rates and we prove its equivalence to the bernstein condition itself a generalization of the tsybakov margin condition both of which have played a central role in obtaining fast rates in statistical learning yet while the bernstein condition is two sided the central condition is one sided making it more suitable to deal with unbounded losses in its stochastic mixability form our condition generalizes both a stochastic exp concavity condition identified by juditsky rigollet and tsybakov and vovk s notion of mixability our unifying conditions thus provide a substantial step towards a characterization of fast rates in statistical learning similar to how classical mixability characterizes constant regret in the sequential prediction with expert advice setting

adaptive estimation of convex and polytopal density support we estimate the support of a uniform density when it is assumed to be a convex polytope or more generally a convex body in mathbb r d in the polytopal case we construct an estimator achieving a rate which does not depend on the dimension d unlike the other estimators that have been proposed so far for d ge 3 our estimator has a better risk than the previous ones and it is nearly minimax up to a logarithmic factor we also propose an estimator which is adaptive with respect to the structure of the boundary of the unknown support

logistic classification with varying gaussians this paper is a continuation of the study of classification learning algorithms generated by regularization schemes associated with gaussian kernels and general convex loss functions in previous papers xiang and zhou 2009 5 xiang 2010 7 it is assumed that the convex loss f has a zero this excludes some useful loss functions without zero such as the logistic loss t log 1 exp t the main purpose of this paper is to conduct error analysis for the classification learning algorithms associated with such loss functions the learning rates are derived by a novel application of projection operators to overcome the technical difficulty

clustering based on pairwise distances when the data is of mixed dimensions in the context of clustering we consider a generative model in a euclidean ambient space with clusters of different shapes dimensions sizes and densities in an asymptotic setting where the number of points becomes large we obtain theoretical guaranties for some emblematic methods based on pairwise distances a simple algorithm based on the extraction of connected components in a neighborhood graph hierarchical clustering with single linkage and the spectral clustering method of ng jordan and weiss the methods are shown to enjoy some near optimal properties in terms of separation between clusters and robustness to outliers the local scaling method of zelnik manor and perona is shown to lead to a near optimal choice for the scale in the first and third methods we also provide a lower bound on the spectral gap to consistently choose the correct number of clusters in the spectral method

recovering convex boundaries from blurred and noisy observations we consider the problem of estimating convex boundaries from blurred and noisy observations in our model the convolution of an intensity function f is observed with additive gaussian white noise the function f is assumed to have convex support g whose boundary is to be recovered rather than directly estimating the intensity function we develop a procedure which is based on estimating the support function of the set g this approach is closely related to the method of geometric hyperplane probing a well known technique in computer vision applications we establish bounds that reveal how the estimation accuracy depends on the ill posedness of the convolution operator and the behavior of the intensity function near the boundary short title recovering convex boundaries

new methods for bias correction at endpoints and boundaries we suggest two new translation based methods for estimating and correcting for bias when estimating the edge of a distribution the first uses an empirical translation applied to the argument of the kernel in order to remove the main effects of the asymmetries that are inherent when constructing estimators at boundaries placing the translation inside the kernel is in marked contrast to traditional approaches such as the use of high order kernels which are related to the jackknife and in effect apply the translation outside the kernel our approach has the advantage of producing bias estimators that while enjoying a high order of accuracy are guaranteed to respect the sign of bias our second method is a new bootstrap technique it involves translating an initial boundary estimate toward the body of the dataset constructing repeated boundary estimates from data that lie below the respective translations and employing averages of the resulting empirical bias approximations to estimate the bias of the original estimator the first of the two methods is most appropriate in univariate cases and is studied there the second approach may be used to bias correct estimates of boundaries of multivariate distributions and is explored in the bivariate case

minimax bounds for active learning this paper analyzes the potential advantages and theoretical challenges of active learning algorithms active learning involves sequential sampling procedures that use information gleaned from previous samples in order to focus the sampling and accelerate the learning process relative to passive learning algorithms which are based on nonadaptive usually random samples there are a number of empirical and theoretical results suggesting that in certain situations active learning can be significantly more effective than passive learning however the fact that active learning algorithms are feedback systems makes their theoretical analysis very challenging this paper aims to shed light on achievable limits in active learning using minimax analysis techniques we study the achievable rates of classification error convergence for broad classes of distributions characterized by decision boundary regularity and noise conditions the results clearly indicate the conditions under which one can expect significant gains through active learning furthermore we show that the learning rates derived are tight for boundary fragment classes in d dimensional feature spaces when the feature marginal density is bounded from above and below

finite dimensional projection for classification and statistical learning in this paper a new method for the binary classification problem is studied it relies on empirical minimization of the hinge risk over an increasing sequence of finite dimensional spaces a suitable dimension is picked by minimizing the regularized risk where the regularization term is proportional to the dimension an oracle type inequality is established for the excess generalization risk i e regret to bayes of the procedure which ensures adequate convergence properties of the method we suggest to select the considered sequence of subspaces by applying kernel principal components analysis kpca in this case the asymptotical convergence rate of the method can be better than what is known for the support vector machine svm exemplary experiments are presented on benchmark data sets where the practical results of the method are comparable to the svm

likelihood based confidence bands for fault lines in response surfaces we consider the problem of constructing asymptotic confidence bands both pointwise and simultaneous for a smooth fault line in a response surface when the design is represented by a point process either deterministic or stochastic with intensity n diverging to infinity the estimator of the fault line is defined as the ridge line on the likelihood surface which arises from locally fitting a model that employs a linear approximation to the fault line to a kernel smooth of the data the construction is based on analysis of the limiting behaviour of perpendicular distance from a point on the true fault line to the nearest point on the ridge we derive asymptotic properties of bias and the limiting distribution of stochastic error this distribution is given by the location of the maximum of a gaussian process with quadratic drift although the majority of attention is focused on the regression problem the limiting distribution is shown to have wider relevance to local likelihood approaches to fault line estimation for density or intensity surfaces

a plug in approach to support estimation we suggest a new approach based on the use of density estimators for the problem of estimating the compact support of a multivariate density this subject motivated in terms of pattern analysis by grenander has interesting connections with detection and clustering a natural class of density based estimators is defined universal consistency results and convergence rates are established for these estimators with respect to the usual measure based metric d between sets further convergence rates with respect to both d and the hausdorff metric d h are also obtained under some fairly intuitive shape restrictions

recovering convex edges of an image from noisy tomographic data we consider the problem of recovering edges of an image from noisy tomographic data the original image is assumed to have a discontinuity jump edge along the boundary of a compact convex set the radon transform of the image is observed with noise and the problem is to estimate the edge we develop an estimation procedure which is based on recovering the support function of the edge it is shown that the proposed estimator is nearly optimal in order in a minimax sense numerical examples illustrate reasonable practical behavior of the estimation procedure

calibrating the excess mass and dip tests of modality summary nonparametric tests of modality are a distribution free way of assessing evidence about inhomogeneity in a population provided that the potential subpopulations are sufficiently well separated they include the excess mass and dip tests which are equivalent in univariate settings and are alternatives to the bandwidth test only very conservative forms of the excess mass and dip tests are available at present however and for that reason they are generally not competitive with the bandwidth test in the present paper we develop a practical approach to calibrating the excess mass and dip tests to improve their level accuracy and power substantially our method exploits the fact that the limiting distribution of the excess mass statistic under the null hypothesis depends on unknowns only through a constant which may be estimated our calibrated test exploits this fact and is shown to have greater power and level accuracy than the bandwidth test has the latter tends to be quite conservative even in an asymptotic sense moreover the calibrated test avoids difficulties that the bandwidth test has with spurious modes in the tails which often must be discounted through subjective intervention of the experimenter

a moment bound for multi hinge classifiers the success of support vector machines in binary classification relies on the fact that hinge loss employed in the risk minimization targets the bayes rule recent research explores some extensions of this large margin based method to the multicategory case we show a moment bound for the socalled multi hinge loss minimizers based on two kinds of complexity constraints entropy with bracketing and empirical entropy obtaining such a result based on the latter is harder than finding one based on the former we obtain fast rates of convergence that adapt to the unknown margin

fractal dimension of time indexed paths abstract measuring animal track or other trajectories for example the track of a flowing particle by divider method to obtain the fractal dimension without knowing the chronology of the points of the path one can obtain erroneous results the importance of time indexing of the track s point will be demonstrated by a simply geometrical example neglecting these indices the error can be high enough to mask relevant differences between various movement paths

discussion local rademacher complexities and oracle inequalities in risk minimization in this magnificent paper professor koltchinskii offers general and powerful performance bounds for empirical risk minimization a fundamental principle of statistical learning theory since the elegant pioneering work of vapnik and chervonenkis in the early 1970s various such bounds have been known that relate the performance of empirical risk minimizers to combinatorial and geometrical features of the class over which the minimization is performed this area of research has been a rich source of motivation and a major field of applications of empirical process theory the appearance of advanced concentration inequalities in the 1990s primarily thanks to talagrand s influential work provoked major advances in both empirical process theory and statistical learning theory and led to a much deeper understanding of some of the basic phenomena in the discussed paper professor koltchinskii develops a powerful new methodology iterative localization which with the help of concentration inequalities is able to explain most of the recent results and go significantly beyond them in many cases the main motivation behind professor koltchinskii s paper is based on classical problems of statistical learning theory such as binary classification and regression in which given a sample xi y i i 1 n of independent and identically distributed pairs of random variables where the xi take their values in some feature space x and the yi are say real valued the goal is to find a function f x r whose risk defined in terms of the expected value of an appropriately chosen loss function is as small as possible in the remaining part of this discussion we point out how the performance bounds of professor koltchinskii s paper can be used to study a seemingly different model motivated by nonparametric ranking problems which has received increasing attention both in the statistical and machine learning literature indeed in several applications such as the search engine problem or credit risk screening the goal is to learn how to rank or to score observations rather than just classify them in this case performance measures involve pairs of observations as can be seen for instance with the auc area under an roc curve criterion in this

learning kernel based halfspaces with the 0 1 loss we describe and analyze a new algorithm for agnostically learning kernel based halfspaces with respect to the 0 1 loss function unlike most of the previous formulations which rely on surrogate convex loss functions e g hinge loss in support vector machines svms and log loss in logistic regression we provide finite time sample guarantees with respect to the more natural 0 1 loss function the proposed algorithm can learn kernel based halfspaces in worst case time poly exp l log l epsilon for any distribution where l is a lipschitz constant which can be thought of as the reciprocal of the margin and the learned classifier is worse than the optimal halfspace by at most epsilon we also prove a hardness result showing that under a certain cryptographic assumption no algorithm can learn kernel based halfspaces in time polynomial in l

alternative area perimeter ratios for measurement of 2d shape compactness of habitats area perimeter ratios are often used to quantify 2d shape compactness shape compactness is of main importance to evaluate the effect of external disturbance on natural habitats reference values a m i n p a m a x p p m i n a p m a x a for area a and perimeter p are defined and proven the calculation of these reference values is based on the characteristics of the object studied and on pixel geometry the cases for 4 and 8 connectivity are discussed using these references alternative area perimeter ratios are composed examples are elaborated to illustrate the use of the indices and their performance

a stability theorem for networks containing synchronous generators abstract we revisit a by now well known stability result for power networks containing several synchronous generators due to f dorfler and f bullo siam journal on control and optimization 2012 the approximate or reduced model that we work with is called the network reduced power system nrps model the mentioned result from 2012 is that under certain conditions the state trajectory of the nrps model is close to the trajectory of the corresponding kuramoto model that is obtained by setting the inertias of the generators to zero if both start from the same initial angles the kuramoto model is locally exponentially stable so that this gives us useful indications on the behavior of the power network we show that the mentioned result can be generalized and strengthened in several aspects for instance we show that the kuramoto model has a unique asymptotically stable equilibrium point and we give an explicit characterization of its domain of attraction we also characterize the region in the state space of the nrps model where the asymptotic approximation by the state trajectories of the kuramoto model is valid we also significantly enlarge the class of nrps models for which the result applies and we prove the local exponential stability of the nrps model for small enough inertia constants we briefly indicate how this result can be useful for analyzing the stability of microgrids containing virtual synchronous machines

on robustness properties of convex risk minimization methods for pattern recognition the paper brings together methods from two disciplines machine learning theory and robust statistics we argue that robustness is an important aspect and we show that many existing machine learning methods based on the convex risk minimization principle have besides other good properties also the advantage of being robust robustness properties of machine learning methods based on convex risk minimization are investigated for the problem of pattern recognition assumptions are given for the existence of the influence function of the classifiers and for bounds on the influence function kernel logistic regression support vector machines least squares and the adaboost loss function are treated as special cases some results on the robustness of such methods are also obtained for the sensitivity curve and the maxbias which are two other robustness criteria a sensitivity analysis of the support vector machine is given

a neyman pearson approach to statistical learning the neyman pearson np approach to hypothesis testing is useful in situations where different types of error have different consequences or a priori probabilities are unknown for any spl alpha 0 the np lemma specifies the most powerful test of size spl alpha but assumes the distributions for each hypothesis are known or in some cases the likelihood ratio is monotonic in an unknown parameter this paper investigates an extension of np theory to situations in which one has no knowledge of the underlying distributions except for a collection of independent and identically distributed i i d training examples from each hypothesis building on a fundamental lemma of cannon et al we demonstrate that several concepts from statistical learning theory have counterparts in the np context specifically we consider constrained versions of empirical risk minimization np erm and structural risk minimization np srm and prove performance guarantees for both general conditions are given under which np srm leads to strong universal consistency we also apply np srm to dyadic decision trees to derive rates of convergence finally we present explicit algorithms to implement np srm for histograms and dyadic decision trees

on the properties of variational approximations of gibbs posteriors the pac bayesian approach is a powerful set of techniques to derive nonasymptotic risk bounds for random estimators the corresponding optimal distribution of estimators usually called the gibbs posterior is unfortunately often intractable one may sample from it using markov chain monte carlo but this is usually too slow for big datasets we consider instead variational approximations of the gibbs posterior which are fast to compute we undertake a general study of the properties of such approximations our main finding is that such a variational approximation has often the same rate of convergence as the original pac bayesian procedure it approximates in addition we show that when the risk function is convex a variational approximation can be obtained in polynomial time using a convex solver we give finite sample oracle inequalities for the corresponding estimator we specialize our results to several learning tasks classification ranking matrix completion discuss how to implement a variational approximation in each case and illustrate the good properties of said approximation on real datasets

minimax optimal classification with dyadic decision trees decision trees are among the most popular types of classifiers with interpretability and ease of implementation being among their chief attributes despite the widespread use of decision trees theoretical analysis of their performance has only begun to emerge in recent years in this paper it is shown that a new family of decision trees dyadic decision trees ddts attain nearly optimal in a minimax sense rates of convergence for a broad range of classification problems furthermore ddts are surprisingly adaptive in three important respects they automatically 1 adapt to favorable conditions near the bayes decision boundary 2 focus on data distributed on lower dimensional manifolds and 3 reject irrelevant features ddts are constructed by penalized empirical risk minimization using a new data dependent penalty and may be computed exactly with computational complexity that is nearly linear in the training sample size ddts comprise the first classifiers known to achieve nearly optimal rates for the diverse class of distributions studied here while also being practical and implementable this is also the first study of which we are aware to consider rates for adaptation to intrinsic data dimension and relevant features

tracking a smooth fault line in a response surface we suggest a sequential or tracking algorithm for estimating a smooth fault line in a response surface the method starts with an approximation to a point on the line and from there the line is tracked as it meanders through the plane the technique differs from recent approaches in that it does not require a large part of the plane to be searched for evidence of a fault line this offers potential computational savings and produces a method that is invariant under rotations of coordinate axes except insofar as a rotation might affect the estimated starting point and the relative orientation of the grid on which calculations are done that feature is important if design points are not located on a regular grid we investigate properties of the method under very general conditions on the design allowing poisson cluster processes jiggled grid processes and deterministic regular lattices uniform rates of convergence are derived in all these settings for the case of noisy data and shown to be within logarithmic factors of optimal pointwise convergence rates in the no noise setting

surrogate regret bounds for bipartite ranking via strongly proper losses the problem of bipartite ranking where instances are labeled positive or negative and the goal is to learn a scoring function that minimizes the probability of mis ranking a pair of positive and negative instances or equivalently that maximizes the area under the roc curve has been widely studied in recent years a dominant theoretical and algorithmic framework for the problem has been to reduce bipartite ranking to pairwise classification in particular it is well known that the bipartite ranking regret can be formulated as a pairwise classification regret which in turn can be upper bounded using usual regret bounds for classification problems recently kotlowski et al 2011 showed regret bounds for bipartite ranking in terms of the regret associated with balanced versions of the standard non pairwise logistic and exponential losses in this paper we show that such nonpairwise surrogate regret bounds for bipartite ranking can be obtained in terms of a broad class of proper composite losses that we term as strongly proper our proof technique is much simpler than that of kotlowski et al 2011 and relies on properties of proper composite losses as elucidated recently by reid and williamson 2010 2011 and others our result yields explicit surrogate bounds with no hidden balancing terms in terms of a variety of strongly proper losses including for example logistic exponential squared and squared hinge losses as special cases an important consequence is that standard algorithms minimizing a non pairwise strongly proper loss such as logistic regression and boosting algorithms assuming a universal function class and appropriate regularization are in fact consistent for bipartite ranking moreover our results allow us to quantify the bipartite ranking regret in terms of the corresponding surrogate regret we also obtain tighter surrogate bounds under certain low noise conditions via a recent result of clemencon and robbiano 2011

image denoising pointwise adaptive approach a new method of pointwise adaptation has been proposed and studied in spokoiny 1998 ann statist 26 1356 1378 in the context of estimation of piecewise smooth univariate functions the present paper extends that method to estimation of bivariate grey scale images composed of large homogeneous regions with smooth edges and observed with noise on a gridded design the proposed estimator hat f x at a point x is simply the average of observations over a window widehat u x selected in a data driven way the theoretical properties of the procedure are studied for the case of piecewise constant images we present a nonasymptotic bound for the accuracy of estimation at a specific grid point x as a function of the number of pixels n of the distance from the point of estimation to the closest boundary and of smoothness properties and orientation of this boundary it is also shown that the proposed method provides a near optimal rate of estimation near edges and inside homogeneous regions we briefly discuss algorithmic aspects and the complexity of the procedure the numerical examples demonstrate a reasonable performance of the method and they are in agreement with the theoretical issues an example from satellite sar imaging illustrates the applicability of the method

a note on support vector machines with polynomial kernels we present a better theoretical foundation of support vector machines with polynomial kernels the sample error is estimated under tsybakov s noise assumption in bounding the approximation error we take advantage of a geometric noise assumption that was introduced to analyze gaussian kernels compared with the previous literature the error analysis in this note does not require any regularity of the marginal distribution or smoothness of bayes rule we thus establish the learning rates for polynomial kernels for a wide class of distributions

rates of growth and sample moduli for weighted empirical processes indexed by sets probability inequalities are obtained for the supremum of a weighted empirical process indexed by a vapnik cervonenkis class c of sets these inequalities are particularly useful under the assumption p c c p c t 0 as t 0 they are used to obtain almost sure bounds on the rate of growth of the process as the sample size approaches infinity to find an asymptotic sample modulus for the unweighted empirical process and to study the ratio pn p of the empirical measure to the actual measure

conditions for almost global attractivity of a synchronous generator connected to an infinite bus conditions for existence and global attractivity of the equilibria of a realistic model of a synchronous generator with constant field current connected to an infinite bus are derived first necessary and sufficient conditions for existence and uniqueness of equilibrium points are provided then sufficient conditions for local asymptotic stability and almost global attractivity of one of these equilibria are given the analysis is carried out by employing a new lyapunov like function to establish convergence of bounded trajectories while the latter is proven using the powerful theoretical framework of cell structures pioneered by leonov and noldus the efficiency of the derived sufficient conditions is illustrated via extensive numerical experiments based on two benchmark examples taken from the literature

rbf nets for approximating an object s boundary by image random sampling abstract our aim is to discuss the problem of approximating the boundary of an object in a binary image in contrast to earlier papers on similar topics we avoid finding contours and then their approximations instead we sample an image in a relatively small number of points 1 3 of pixels at random the collected data is used for imposing constraints on parameters of a radial basis functions rbf neural net it is proved that if the rbf net structure is sufficiently rich to approximate an object boundary then estimates of rbf net parameters tend to their true values as the number of samples approaches to infinity then a seemingly linear algorithm of estimating linear weights is proposed and its consistency is also proved its implementation which is based on an iterative use of the linear programming is briefly discussed and the results of its testing are shortly reported

minimax optimal bounds for detectors based on estimated prior probabilities in many signal detection and classification problems we have knowledge of the distribution under each hypothesis but not the prior probabilities this paper is aimed at providing theory to quantify the performance of detection via estimating prior probabilities from either labeled or unlabeled training data the error or risk is considered as a function of the prior probabilities we show that the risk function is locally lipschitz in the vicinity of the true prior probabilities and the error of detectors based on estimated prior probabilities depends on the behavior of the risk function in this locality in general we show that the error of detectors based on the maximum likelihood estimate mle of the prior probabilities converges to the bayes error at a rate of n 1 2 where n is the number of training data if the behavior of the risk function is more favorable then detectors based on the mle have errors converging to the corresponding bayes errors at optimal rates of the form n 1 alpha 2 where alpha 0 is a parameter governing the behavior of the risk function with a typical value alpha 1 the limit alpha rightarrow infty corresponds to a situation where the risk function is flat near the true probabilities and thus insensitive to small errors in the mle in this case the error of the detector based on the mle converges to the bayes error exponentially fast with n we show that the bounds are achievable no matter given labeled or unlabeled training data and are minimax optimal in the labeled case

change curve estimation via wavelets abstract the recently developed theory of wavelets has a remarkable ability to zoom in on very short lived frequency phenomena such as transients in signals and singularities in functions and hence provides an ideal tool to study localized changes this article proposes a wavelet method for estimating jump and sharp cusp curves of a function in the plane the method involves first computing wavelet transformation of data and then estimating jump and sharp cusp curves by wavelet transformation across fine scales asymptotic theory is established and simulations are carried out to lend some credence to the asymptotic theory the wavelet estimate is nearly optimal and can be computed by fast algorithms the method is applied to a real image

a classification framework for anomaly detection one way to describe anomalies is by saying that anomalies are not concentrated this leads to the problem of finding level sets for the data generating density we interpret this learning problem as a binary classification problem and compare the corresponding classification risk with the standard performance measure for the density level problem in particular it turns out that the empirical classification risk can serve as an empirical performance measure for the anomaly detection problem this allows us to compare different anomaly detection algorithms empirically i e with the help of a test set furthermore by the above interpretation we can give a strong justification for the well known heuristic of artificially sampling labeled samples provided that the sampling plan is well chosen in particular this enables us to propose a support vector machine svm for anomaly detection for which we can easily establish universal consistency finally we report some experiments which compare our svm to other commonly used methods including the standard one class svm

error analysis of coefficient based regularized algorithm for density level detection in this letter we consider a density level detection dld problem by a coefficient based classification framework with regularizer and data dependent hypothesis spaces although the data dependent characteristic of the algorithm provides flexibility and adaptivity for dld it leads to difficulty in generalization error analysis to overcome this difficulty an error decomposition is introduced from an established classification framework on the basis of this decomposition the estimate of the learning rate is obtained by using rademacher average and stepping stone techniques in particular the estimate is independent of the capacity assumption used in the previous literature

learning rates for classification with gaussian kernels this letter aims at refined error analysis for binary classification using support vector machine svm with gaussian kernel and convex loss our first result shows that for some loss functions su

adaptive model selection using empirical complexities given n independent replicates of a jointly distributed pair x y rd r we wish to select from a fixed sequence of model classes f1 f2 a deterministic prediction rule f rd r whose risk is small we investigate the possibility of empirically assessing the complexity of each model class that is the actual difficulty of the estimation problem within each class the estimated complexities are in turn used to define an adaptive model selection procedure which is based on complexity penalized empirical risk the available data are divided into two parts the first is used to form an empirical cover of each model class and the second is used to select a candidate rule from each cover based on empirical risk the covering radii are determined empirically to optimize a tight upper bound on the estimation error an estimate is chosen from the list of candidates in order to minimize the sum of class complexity and empirical risk a distinguishing feature of the approach is that the complexity of each model class is assessed empirically based on the size of its empirical cover finite sample performance bounds are established for the estimates and these bounds are applied to several non parametric estimation problems the estimates are shown to achieve a favorable tradeoff between approximation and estimation error and to perform as well as if the distribution dependent complexities of the model classes were known beforehand in addition it is shown that the estimate can be consistent and even possess near optimal rates of convergence when each model class has an infinite vc or pseudo dimension for regression estimation with squared loss we modify our estimate to achieve a faster rate of convergence appears in annals of statistics vol 27 1830 1864 ams 1991 subject classifications primary 62g07 62g20 secondary 62h30

simultaneous adaptation to the margin and to complexity in classification we consider the problem of adaptation to the margin and to complexity in binary classification we suggest a learning method with a numerically easy aggregation step adaptivity both to the margin and complexity in classification usually involves empirical risk minimization or rademacher complexities which lead to numerical difficulties on the other hand there exist classifiers that are easy to compute and that converge with fast rates but are not adaptive combining these classifiers by our aggregation procedure we get numerically realizable adaptive classifiers that converge with fast rates

ranking and empirical minimization of u statistics the problem of ranking ordering instances instead of simply classifying them has recently gained much attention in machine learning in this paper we formulate the ranking problem in a rigorous statistical framework the goal is to learn a ranking rule for deciding among two instances which one is better with minimum ranking risk since the natural estimates of the risk are of the form of a u statistic results of the theory of u processes are required for investigating the consistency of empirical risk minimizers we establish in particular a tail inequality for degenerate u processes and apply it for showing that fast rates of convergence may be achieved under specific noise assumptions just like in classification convex risk minimization methods are also studied

a high dimensional wilks phenomenon a theorem by wilks asserts that in smooth parametric density estimation the difference between the maximum likelihood and the likelihood of the sampling distribution converges toward a chi square distribution where the number of degrees of freedom coincides with the model dimension this observation is at the core of some goodness of fit testing procedures and of some classical model selection methods this paper describes a non asymptotic version of the wilks phenomenon in bounded contrast optimization procedures using concentration inequalities for general functions of independent random variables it proves that in bounded contrast minimization as for example in statistical learning theory the difference between the empirical risk of the minimizer of the true risk in the model and the minimum of the empirical risk the excess empirical risk satisfies a bernstein like inequality where the variance term reflects the dimension of the model and the scale term reflects the noise conditions from a mathematical statistics viewpoint the significance of this result comes from the recent observation that when using model selection via penalization the excess empirical risk represents a minimum penalty if non asymptotic guarantees concerning prediction error are to be provided from the perspective of empirical process theory this paper describes a concentration inequality for the supremum of a bounded non centered actually non positive empirical process combining the now classical analysis of m estimation building on talagrand s inequality for suprema of empirical processes and versatile moment inequalities for functions of independent random variables this paper develops a genuine bernstein like inequality that seems beyond the reach of traditional tools

some theory for generalized boosting algorithms we give a review of various aspects of boosting clarifying the issues through a few simple results and relate our work and that of others to the minimax paradigm of statistics we consider the population version of the boosting algorithm and prove its convergence to the bayes classifier as a corollary of a general result about gauss southwell optimization in hilbert space we then investigate the algorithmic convergence of the sample version and give bounds to the time until perfect separation of the sample we conclude by some results on the statistical optimality of the l2 boosting

a bernstein type inequality for some mixing processes and dynamical systems with an application to learning we establish a bernstein type inequality for a class of stochastic processes that include the classical geometrically mixing processes rio s generalization of these processes as well as many time discrete dynamical systems modulo a logarithmic factor and some constants our bernsteintype inequality coincides with the classical bernstein inequality for i i d data we further use this new bernstein type inequality to derive an oracle inequality for generic regularized empirical risk minimization algorithms and data generated by such processes applying this oracle inequality to support vector machines using the gaussian kernels for both least squares and quantile regression it turns out that the resulting learning rates match up to some arbitrarily small extra term in the exponent the optimal rates for i i d processes