refined error bounds for several learning algorithms this article studies the achievable guarantees on the error rates of certain learning algorithms with particular focus on refining logarithmic factors many of the results are based on a general technique for obtaining bounds on the error rates of sample consistent classifiers with monotonic error regions in the realizable case we prove bounds of this type expressed in terms of either the vc dimension or the sample compression size this general technique also enables us to derive several new bounds on the error rates of general sample consistent learning algorithms as well as refined bounds on the label complexity of the cal active learning algorithm additionally we establish a simple necessary and sufficient condition for the existence of a distribution free bound on the error rates of all sample consistent learning rules converging at a rate inversely proportional to the sample size we also study learning in the presence of classification noise deriving a new excess error rate guarantee for general vc classes under tsybakov s noise condition and establishing a simple and general necessary and sufficient condition for the minimax excess risk under bounded noise to converge at a rate inversely proportional to the sample size

square root penalty adaptation to the margin in classification and in edge estimation we consider the problem of adaptation to the margin in binary classification we suggest a penalized empirical risk minimization classifier that adaptively attains up to a logarithmic factor fast optimal rates of convergence for the excess risk that is rates that can be faster than n 1 2 where n is the sample size we show that our method also gives adaptive estimators for the problem of edge estimation

irreversibility implies the occurrence of nonmonotonic power spectra both continuous time and discrete time systems driven by l2 exponential ergodic stationary markov processes are investigated we prove that in both cases irreversibility implies the occurrence of nonmonotonic power spectra of some observables this result combined with the work of jiang and zhang j math phys 44 4681 4689 2003 tells us that in the continuous time case reversibility can be characterized by monotonicity in all power spectra while this is not true in the discrete time case

learning by mirror averaging given a collection of m different estimators or classifiers we study the problem of model selection type aggregation i e we construct a new estimator or classifier called aggregate which is nearly as good as the best among them with respect to a given risk criterion we define our aggregate by a simple recursive procedure which solves an auxiliary stochastic linear programming problem related to the original non linear one and constitutes a special case of the mirror averaging algorithm we show that the aggregate satisfies sharp oracle inequalities under some general assumptions the results allow one to construct in an easy way sharp adaptive nonparametric estimators for several problems including regression classification and density estimation

rates of convergence for minimum contrast estimators we shall present here a general study of minimum contrast estimators in a nonparametric setting although our results are also valid in the classical parametric case for independent observations these estimators include many of the most popular estimators in various situations such as maximum likelihood estimators least squares and other estimators of the regression function estimators for mixture models or deconvolution the main theorem relates the rate of convergence of those estimators to the entropy structure of the space of parameters optimal rates depending on entropy conditions are already known at least for some of the models involved and they agree with what we get for minimum contrast estimators as long as the entropy counts are not too large but under some circumstances large entropies or changes in the entropy structure due to local perturbations the resulting the rates are only suboptimal counterexamples are constructed which show that the phenomenon is real for non parametric maximum likelihood or regression this proves that under purely metric assumptions our theorem is optimal and that minimum contrast estimators happen to be suboptimal

analytic and numerical study of stochastic resonance a detailed analytic and numerical study of stochastic resonance in the regime of small amplitude periodic modulation is presented for the double well potential it has been suggested that the first passage time for hopping from one potential minimum to the other is exactly half the modulation period at the value of the noise strength that maximizes the stochastic resonance profile this explanation of the phenomenon is critically assessed we establish a criterion for the noise strength at which the stochastic resonance profile is maximal

risk bounds for statistical learning we propose a general theorem providing upper bounds for the risk of an empirical risk minimizer erm we essentially focus on the binary classification framework we extend tsybakov s analysis of the risk of an erm under margin type conditions by using concentration inequalities for conveniently weighted empirical processes this allows us to deal with ways of measuring the size of a class of classifiers other than entropy with bracketing as in tsybakov s work in particular we derive new risk bounds for the erm when the classification rules belong to some vc class under margin conditions and discuss the optimality of these bounds in a minimax sense

improving the sample complexity using global data we study the sample complexity of proper and improper learning problems with respect to different q loss functions we improve the known estimates for classes which have relatively small covering numbers in empirical l sub 2 spaces e g log covering numbers which are polynomial with exponent p 2 we present several examples of relevant classes which have a small fat shattering dimension and hence fit our setup the most important of which are kernel machines

hellinger consistency of certain nonparametric maximum likelihood estimators consider a class mathscr p p theta theta in theta of probability measures on a measurable space mathscr x mathscr a dominated by a sigma finite measure mu let f theta dp theta d mu theta in theta and let theta n be a maximum likelihood estimator based on n independent observations from p theta 0 theta 0 in theta we use results from empirical process theory to obtain convergence for the hellinger distance h f hat theta n f theta 0 under certain entropy conditions on the class of densities f theta theta in theta the examples we present are a model with interval censored observations smooth densities monotone densities and convolution models in most examples the convexity of the class of densities is of special importance

efficient agnostic learning of neural networks with bounded fan in we show that the class of two layer neural networks with bounded fan in is efficiently learnable in a realistic extension to the probably approximately correct pac learning model in this model a joint probability distribution is assumed to exist on the observations and the learner is required to approximate the neural network which minimizes the expected quadratic error as special cases the model allows learning real valued functions with bounded noise learning probabilistic concepts and learning the best approximation to a target function that cannot be well approximated by the neural network the networks we consider have real valued inputs and outputs an unlimited number of threshold hidden units with bounded fan in and a bound on the sum of the absolute values of the output weights the number of computation steps of the learning algorithm is bounded by a polynomial in 1 spl epsiv 1 spl delta n and b where spl epsiv is the desired accuracy spl delta is the probability that the algorithm fails n is the input dimension and b is the bound on both the absolute value of the target which may be a random variable and the sum of the absolute values of the output weights in obtaining the result we also extended some results on iterative approximation of functions in the closure of the convex hull of a function class and on the sample complexity of agnostic learning with the quadratic loss function

optimal aggregation of classifiers in statistical learning classification can be considered as nonparametric estimation of sets where the risk is defined by means of a specific distance between sets associated with misclassification error it is shown that the rates of convergence of classifiers depend on two parameters the complexity of the class of candidate sets and the margin parameter the dependence is explicitly given indicating that optimal fast rates approaching o n 1 can be attained where n is the sample size and that the proposed classifiers have the property of robustness to the margin the main result of the paper concerns optimal aggregation of classifiers we suggest a classifier that automatically adapts both to the complexity and to the margin and attains the optimal fast rates up to a logarithmic factor

smooth discrimination analysis ties f and g can be based on the estimation of the set g x fx gx we consider applications where it is appropriate to assume that the region g has a smooth boundary or belongs to another nonparametric class of sets in particular this assumption makes sense if discrimination is used as a data analytic tool decision rules based on minimization of empirical risk over the whole class of sets and over sieves are considered their rates of convergence are obtained we show that these rules achieve optimal rates for estimation of g and optimal rates of convergence for bayes risks an interesting conclusion is that the optimal rates for bayes risks can be very fast in particular faster than the parametric root n rate these fast rates cannot be guaranteed for plug in rules

smooth dynamics and new theoretical ideas in nonequilibrium statistical mechanics this paper reviews various applications of the theory of smooth dynamical systems to conceptual problems of nonequilibrium statistical mecanics we adopt a new point of view which has emerged progressively in recent years and which takes seriously into account the chaotic character of the microscopic time evolution the emphasis is on nonequilibrium steady states rather than the traditional approach to equilibrium point of view of boltzmann the nonequilibrium steady states in presence of a gaussian thermostat are described by srb measures in terms of these one can prove the gallavotti cohen fluctuation theorem one can also prove a general linear response formula and study its consequences which are not restricted to near equilibrium situations at equilibrium one recovers in particular the onsager reciprocity relations under suitable conditions the nonequilibrium steady states satisfy the pairing theorem of dettmann and morriss the results just mentioned hold so far only for classical systems they do not involve large size i e they hold without a thermodynamic limit

the estimation of prediction error having constructed a data based estimation rule perhaps a logistic regression or a classification tree the statistician would like to know its performance as a predictor of future cases there are two main theories concerning prediction error 1 penalty methods such as cp akaike s information criterion and stein s unbiased risk estimate that depend on the covariance between data points and their corresponding predictions and 2 cross validation and related nonparametric bootstrap techniques this article concerns the connection between the two theories a rao blackwell type of relation is derived in which nonparametric methods such as cross validation are seen to be randomized versions of their covariance penalty counterparts the model based penalty methods offer substantially better accuracy assuming that the model is believable

new symmetry solution techniques for first order non linear pdes using recent extensions of work of s lie vorlesungen uber differentialgleichungen mit bekannten infinitesimalen transformationen leipzig teubner 1891 and e cartan lecons sur les invariants integreaux hermann paris 1922 les systems differentials exterieures et leurs applications geometrique hermann paris 1945 for integrating frobenius integrable vector field distributions via symmetry we examine some symmetry techniques for finding local solutions of first order non linear partial differential equations pdes in the language of exterior differential systems we develop a technique for solving first order quasilinear pdes that is then applied to general first order non linear partial differential equations our results are significant inasmuch as we give algorithms for solving first order pdes in the presence of symmetry that are essentially mechanical in nature and done in the original coordinate system

convergence rate of sieve estimates in this paper we develop a general theory for the convergence rate of sieve estimates maximum likelihood estimates mle s and related estimates obtained by optimizing certain empirical criteria in general parameter spaces in many cases especially when the parameter space is infinite dimensional maximization over the whole parameter space is undesirable in such cases one has to perform maximization over an approximating space sieve of the original parameter space and allow the size of the approximating space to grow as the sample size increases this method is called the method of sieves in the case of the maximum likelihood estimation an mle based on a sieve is called a sieve mle we found that the convergence rate of a sieve estimate is governed by a the local expected values variances and l 2 entropy of the criterion differences and b the approximation error of the sieve a robust nonparametric regression problem a mixture problem and a nonparametric regression problem are discussed as illustrations of the theory we also found that when the underlying space is too large the estimate based on optimizing over the whole parameter space may not achieve the best possible rates of convergence whereas the sieve estimate typically does not suffer from this difficulty

convexity classification and risk bounds many of the classification algorithms developed in the machine learning literature including the support vector machine and boosting can be viewed as minimum contrast methods that minimize a convex surrogate of the 0 1 loss function the convexity makes these algorithms computationally efficient the use of a surrogate however has statistical consequences that must be balanced against the computational virtues of convexity to study these issues we provide a general quantitative relationship between the risk as assessed using the 0 1 loss and the risk as assessed using any nonnegative surrogate loss function we show that this relationship gives nontrivial upper bounds on excess risk under the weakest possible condition on the loss function that it satisfies a pointwise form of fisher consistency for classification the relationship is based on a simple variational transformation of the loss function that is easy to compute in many applications we also present a refined version of this result in the

minimal penalties for gaussian model selection this paper is mainly devoted to a precise analysis of what kind of penalties should be used in order to perform model selection via the minimiza tion of a penalized least squares type criterion within some general gaussian framework including the classical ones as compared to our previous paper on this topic birge and massart in j eur math soc 3 203 268 2001 more elaborate forms of the penalties are given which are shown to be in some sense optimal we indeed provide more precise upper bounds for the risk of the penalized estimators and lower bounds for the penalty terms showing that the use of smaller penalties may lead to disastrous results these lower bounds may also be used to design a practical strategy that allows to estimate the penalty from the data when the amount of noise is unknown we provide an illustra tion of the method for the problem of estimating a piecewise constant signal in gaussian noise when neither the number nor the location of the change points are known

the green kubo formula for general markov processes with a continuous time parameter for general markov processes the green kubo formula is shown to be valid under a mild condition a class of stochastic evolution equations on a separable hilbert space and three typical infinite systems of locally interacting diffusions on irreversible in most cases are shown to satisfy the green kubo formula and the einstein relations for these stochastic evolution equations are shown explicitly as a corollary

moment inequalities for functions of independent random variables a general method for obtaining moment inequalities for functions of independent random variables is presented it is a generalization of the entropy method which has been used to derive concentration inequalities for such functions boucheron lugosi and massart ann probab 31 2003 1583 1614 and is based on a generalized tensorization inequality due to latala and oleszkiewicz lecture notes in math 1745 2000 147 168 the new inequalities prove to be a versatile tool in a wide range of applications we illustrate the power of the method by showing how it can be used to effortlessly re derive classical inequalities including rosenthal and kahane khinchine type inequalities for sums of independent random variables moment inequalities for suprema of empirical processes and moment inequalities for rademacher chaos and u statistics some of these corollaries are apparently new in particular we generalize talagrand s exponential inequality for rademacher chaos of order 2 to any order we also discuss applications for other complex functions of independent random variables such as suprema of boolean polynomials which include as special cases subgraph counting problems in random graphs

probability inequalities for likelihood ratios and convergence rates of sieve mles let y 1 y n be independent identically distributed with density p 0 and let f be a space of densities we show that the supremum of the likelihood ratios i 1 n p y i p 0 y i where the supremum is over p f with p l 2 p 0 1 2 2 e is exponentially small with probability exponentially close to 1 the exponent is proportional to ne 2 the only condition required for this to hold is that e exceeds a value determined by the bracketing hellinger entropy of f a similar inequality also holds if we replace f by f n and p 0 by q n where q n is an approximation to p 0 in a suitable sense these results are applied to establish rates of convergence of sieve mles furthermore weak conditions are given under which the optimal rate e n defined by h e n f ne n 2 where h f is the hellinger entropy of f is nearly achievable by sieve estimators

some comments on cp i study the typical configuration of a cp plot when the number of variables in the regression problem is large and there are many weak effects i show that a particular configuration that is very commonly seen can arise in a simple way i give a formula by means of which the risk incurred by the minimum cp rule can be estimated

on the imaginary real ratio rule of power spectra the power spectra of real observables for continuous time systems driven by l2 exponential ergodic stationary markov processes are investigated we find the necessary and sufficient condition for the power spectra to be monotonic nonmonotonic which includes the famous imaginary real ratio rule as a special case we also give some examples as applications

discussion local rademacher complexities and oracle inequalities in risk minimization in this magnificent paper professor koltchinskii offers general and powerful performance bounds for empirical risk minimization a fundamental principle of statistical learning theory since the elegant pioneering work of vapnik and chervonenkis in the early 1970s various such bounds have been known that relate the performance of empirical risk minimizers to combinatorial and geometrical features of the class over which the minimization is performed this area of research has been a rich source of motivation and a major field of applications of empirical process theory the appearance of advanced concentration inequalities in the 1990s primarily thanks to talagrand s influential work provoked major advances in both empirical process theory and statistical learning theory and led to a much deeper understanding of some of the basic phenomena in the discussed paper professor koltchinskii develops a powerful new methodology iterative localization which with the help of concentration inequalities is able to explain most of the recent results and go significantly beyond them in many cases the main motivation behind professor koltchinskii s paper is based on classical problems of statistical learning theory such as binary classification and regression in which given a sample xi y i i 1 n of independent and identically distributed pairs of random variables where the xi take their values in some feature space x and the yi are say real valued the goal is to find a function f x r whose risk defined in terms of the expected value of an appropriately chosen loss function is as small as possible in the remaining part of this discussion we point out how the performance bounds of professor koltchinskii s paper can be used to study a seemingly different model motivated by nonparametric ranking problems which has received increasing attention both in the statistical and machine learning literature indeed in several applications such as the search engine problem or credit risk screening the goal is to learn how to rank or to score observations rather than just classify them in this case performance measures involve pairs of observations as can be seen for instance with the auc area under an roc curve criterion in this

an application of solvable structures to classical and nonclassical similarity solutions using exterior differential systems we extend work by harrison and estabrook for deriving similarity solutions of hyperbolic and parabolic partial differential equations pdes we use exterior calculus results to show that a symmetry isovector of the differential ideal corresponding to some hyperbolic or parabolic pde can be used to generate a cauchy characteristic vector field of a restricted exterior differential system defined on some four dimensional regular submanifold of the first jet bundle we then show that this restricted differential ideal has a frobenius integrable annihilating space which can be used to yield a similarity solution of the pde by applying results from lie and cartan on integrating frobenius integrable vector field distributions via symmetry we also give an extension to conditional symmetries

the green kubo formula and power spectrum of reversible markov processes as is known the entropy production rate of a stationary markov process vanishes if and only if the process is reversible in this paper we discuss the reversibility of a stationary markov process from a functional analysis point of view it is shown that the process is reversible if and only if it has a symmetric markov semigroup equivalently a self adjoint infinitesimal generator applying this fact we prove that the green kubo formula holds for reversible markov processes by demonstrating that the power spectrum of each reversible markov process is lorentz typed we show that it is impossible for stochastic resonance to occur in systems with zero entropy production

stochastic resonance on a circle without excitation physical investigation and peak frequency formula in this article the existence of stochastic resonance sr without external force in a simplified circular system for different values of the control parameter b is considered the average power spectra are calculated as well as the signal to noise ratio as a measure for stochastic resonance it is shown that in the monostable and semistable b 1 the system is oscillatory and noise plays only a destructive role therefore no sr occurs the rotation number of the system is calculated and compared to the peak frequency of the power spectrum although the coincidence in the noisy case is not as good as that in the deterministic case we can derive an empirical formula between the peak frequency of the power spectrum and the rotation number of the system which is in good agreement with results of numerical simulations

optimal model assessment selection and combination central to statistical theory and application is statistical modeling which typically involves choosing a single model or combining a number of models of different sizes and from different sources whereas model selection seeks a single best modeling procedure model combination combines the strength of different modeling procedures in this article we look at several key issues and argue that model assessment is the key to model selection and combination most important we introduce a general technique of optimal model assessment based on data perturbation thus yielding optimal selection in particular model selection and combination from a frequentist perspective we advocate model combination over a selected subset of modeling procedures because it controls bias while reducing variability hence yielding better performance in terms of the accuracy of estimation and prediction to realize the potential of model combination we develop methodologies for determining the optimal tuning parameter such

the green kubo formula autocorrelation function and fluctuation spectrum for finite markov chains with continuous time a general form of the green kubo formula which describes the fluctuations pertaining to all the steady states whether equilibrium or non equilibrium for a system driven by a finite markov chain with continuous time briefly mc is shown the equivalence of different forms of the green kubo formula is exploited we also look at the differences in terms of the autocorrelation function and the fluctuation spectrum between the equilibrium state and the non equilibrium steady state also if the mc is in the non equilibrium steady state we can always find a complex function such that the fluctuation spectrum of is non monotonous in 0

ranking and empirical minimization of u statistics the problem of ranking ordering instances instead of simply classifying them has recently gained much attention in machine learning in this paper we formulate the ranking problem in a rigorous statistical framework the goal is to learn a ranking rule for deciding among two instances which one is better with minimum ranking risk since the natural estimates of the risk are of the form of a u statistic results of the theory of u processes are required for investigating the consistency of empirical risk minimizers we establish in particular a tail inequality for degenerate u processes and apply it for showing that fast rates of convergence may be achieved under specific noise assumptions just like in classification convex risk minimization methods are also studied

on characterization of reversible markov processes by monotonicity of the fluctuation spectral density for the system driven by a stationary markov process t t r if it is in nonequilibrium steady state i e irreversible then there exists a function such that the fluctuation spectrum or say power spectrum density of t is nonmonotonic in 0 under mild conditions which means that there exist nonzero spectrum peaks of the fluctuation spectrum for the system driven by a markov chain with discrete time t t z even if it is in equilibrium state i e reversible one cannot distinguish the equilibrium and nonequilibrium steady state in terms of the monotonicity of the fluctuation spectrum any more

green kubo formula derived from large deviation statistics abstract numerical calculations for many physical systems including chaos and turbulence have been greatly developed in recent years we often investigate the characteristics of the time series for chaotic systems and study the essential laws underlying its time series we show the important characteristics which are sometimes not pointed out one of them is the green kubo formula which holds for the time series that satisfies the large deviation statistics extended version for the relation is shown in this paper and a simple example is given the relationship between the green kubo formula and the large deviation statistics is emphasized

more comments on c p we discuss the interpretation of c p plots and show how they can be calibrated in several ways we comment on the practice of using the display as a basis for formal selection of a subset regression model and extend the range of application of the device to encompass arbitrary linear estimates of the regression coefficients for example ridge estimates

on the definition of entropy production via examples we present a definition of entropy production rate for classes of deterministic and stochastic dynamics the point of departure is a gibbsian representation of the steady state path space measure for which the density is determined with respect to the time reversed process the gibbs formalism is used as a unifying algorithm capable of incorporating basic properties of entropy production in nonequilibrium systems our definition is motivated by recent work on the gallavotti cohen local fluctuation theorem and it is illustrated via a number of examples