information theoretic determination of minimax rates of convergence we present some general results determining minimax bounds on statistical risk for density estimation based on certain information theoretic considerations these bounds depend only on metric entropy conditions and are used to identify the minimax rates of convergence

uniform estimates for the finite time ruin probability in the dependent renewal risk model abstract this paper investigates the finite time ruin probability in the dependent renewal risk model where the claim sizes are independent and identically distributed random variables with strongly subexponential tails and the interarrival times are negatively dependent we establish an asymptotic estimate which holds uniformly for the time horizon varying in the positive half line

center manifolds and normal forms for a class of retarded functional differential equations with parameter associated with fold hopf singularity in this paper we present explicit formulas for computing the coefficients of a center manifolds for the fold hopf singularity in autonomous retarded functional differential equations as consequence normal forms associated with the flow on a center manifold up to an arbitrary order are derived the explicit formulas have been implemented using the computer algebra system maple we apply our results to a delayed system in order to show the applicability of the methodology

robust linear least squares regression we consider the problem of robustly predicting as well as the best linear combination of d given functions in least squares regression and variants of this problem including constraints on the parameters of the linear combination for the ridge estimator and the ordinary least squares estimator and their variants we provide new risk bounds of order d n without logarithmic factor unlike some standard results where n is the size of the training data we also provide a new estimator with better deviations in presence of heavy tailed noise it is based on truncating differences of losses in a min max framework and satisfies a d n risk bound both in expectation and in deviations the key common surprising factor of these results is the absence of exponential moment condition on the output distribution while achieving exponential deviations all risk bounds are obtained through a pac bayesian analysis on truncated differences of losses experimental results strongly back up our truncated min max estimator

on estimating a density using hellinger distance and some other strange facts on s interesse ici aux possibles vitesses d estimation d une densite a support compact dans m sous des hypotheses de regularite lorsque la perte est mesuree par le carre de la distance de hellinger on regardera aussi le cas connu des normes mathbb l q pour 1 q 2 et le risque est le risque minimax sur la famille on donne une methode generale permettant de traiter les problemes dans le cadre de la theorie de l approximation sous des conditions concernant l entropie metrique et l e capacite des familles a estimer les rapports entre regularite et entropie metrique etant bien connus nous pourrons aussi traiter les cas classiques et d autres qui le sont moins sous des conditions de bornes inferieures les vitesses sont celles observees pour la norme mathbb l q mais elles different dans le cas general on montre aussi que les restrictions sur la compacite du support ou la regularite sont indispensables et que leur absence mene a l impossibilite d obtenir une estimation raisonnable en ce sens que n importe quelle suite d estimateurs sera arbitrairement mauvaise en un point au moins un resultat analogue est vrai sous des conditions de regularite

uniform asymptotics for the finite time ruin probabilities of two kinds of nonstandard bidimensional risk models abstract in this paper we consider uniform asymptotics for the finite time ruin probabilities of two kinds of nonstandard bidimensional renewal risk models with constant interest forces and diffusion generated by brownian motions in one of the models two classes of claims have different arrival times while in the another model two classes of claims share the same arrival times in both models two classes of claim sizes are both upper tail asymptotically independent and their distributions belong to the intersection of the long tailed distribution class and the dominatedly varying tailed distribution class and the inter arrival times follow a widely lower orthant dependence structure in each model we obtain three kinds of uniform asymptotics for the finite time ruin probabilities respectively

rates of convergence for minimum contrast estimators we shall present here a general study of minimum contrast estimators in a nonparametric setting although our results are also valid in the classical parametric case for independent observations these estimators include many of the most popular estimators in various situations such as maximum likelihood estimators least squares and other estimators of the regression function estimators for mixture models or deconvolution the main theorem relates the rate of convergence of those estimators to the entropy structure of the space of parameters optimal rates depending on entropy conditions are already known at least for some of the models involved and they agree with what we get for minimum contrast estimators as long as the entropy counts are not too large but under some circumstances large entropies or changes in the entropy structure due to local perturbations the resulting the rates are only suboptimal counterexamples are constructed which show that the phenomenon is real for non parametric maximum likelihood or regression this proves that under purely metric assumptions our theorem is optimal and that minimum contrast estimators happen to be suboptimal

density estimation by wavelet thresholding density estimation is a commonly used test case for nonparametric estimation methods we explore the asymptotic properties of estimators based on thresholding of empirical wavelet coefficients minimax rates of convergence are studied over a large range of besov function classes b pq and for a range of global l p error measures 1 p p some form of nonlinearity is essential since the minimax linear estimators are suboptimal by polynomial powers of n a second approach using an approximation of a gaussian white noise model in a mallows metric is used to attain exactly optimal rates of convergence for quadratic error p 2

risk bounds for statistical learning we propose a general theorem providing upper bounds for the risk of an empirical risk minimizer erm we essentially focus on the binary classification framework we extend tsybakov s analysis of the risk of an erm under margin type conditions by using concentration inequalities for conveniently weighted empirical processes this allows us to deal with ways of measuring the size of a class of classifiers other than entropy with bracketing as in tsybakov s work in particular we derive new risk bounds for the erm when the classification rules belong to some vc class under margin conditions and discuss the optimality of these bounds in a minimax sense

the effects of harvesting and time delay on predator prey systems with holling type ii functional response in this paper the effects of harvesting and time delay on two different types of predator prey systems with delayed predator specific growth and holling type ii functional response are studied by applying the normal form theory of retarded functional differential equations developed by faria and magalhaes j differential equations 122 1995 pp 181 200 j differential equations 122 1995 pp 201 224 hopf bifurcations are demonstrated in models with harvesting of the prey at a constant rate by taking the delay as a bifurcation parameter and numerical examples supporting our theoretical prediction are also given furthermore bifurcation analysis indicates that delayed predator prey systems with predator harvesting exhibit bogdanov takens bifurcation the versal unfoldings of the models at the bogdanov takens singularity are obtained and numerical simulations and bifurcation diagrams are given to illustrate the obtained results

restrictions and unfolding of double hopf bifurcation in functional differential equations abstract the normal form of a vector field generated by scalar delay differential equations at nonresonant double hopf bifurcation points is investigated using the methods developed by faria and magalhaes j differential equations 122 1995 181 we show that 1 there exists linearly independent unfolding parameters of classes of delay differential equations for a double hopf point which generically map to linearly independent unfolding parameters of the normal form equations ordinary differential equations 2 there are generically no restrictions on the possible flows near a double hopf point for both general and z 2 symmetric first order scalar equations with two delays in the nonlinearity and 3 there always are restrictions on the possible flows near a double hopf point for first order scalar delay differential equations with one delay in the nonlinearity and in nth order scalar delay differential equations n 2 with one delay feedback

some concepts of negative dependence abstract the theory of positive dependence notions cannot yield useful results for some widely used distributions such as the multinomial dirichlet and the multivariate hypergeometric some conditions of negative dependence that are satisfied by these distributions and which have practical meaning are introduced preservation results for some of these concepts are derived useful inequalities for some widely used distributions are obtained results of mallows 1969 that apply to the multinomial distributions are extended to more distributions examples are listed author

on density estimation in the view of kolmogorov s ideas in approximation theory the paper deals with upper and lower bounds for the quality of probability density estimation connections are established between these problems and the theory of approximation of functions particularly it is demonstrated how some of kolmogorov s concepts work

a new method for estimation and model selection rho estimation the aim of this paper is to present a new estimation procedure that can be applied in various statistical frameworks including density and regression and which leads to both robust and optimal or nearly optimal estimators in density estimation they asymptotically coincide with the celebrated maximum likelihood estimators at least when the statistical model is regular enough and contains the true density to estimate for very general models of densities including non compact ones these estimators are robust with respect to the hellinger distance and converge at optimal rate up to a possible logarithmic factor in all cases we know in the regression setting our approach improves upon the classical least squares in many respects in simple linear regression for example it provides an estimation of the coefficients that are both robust to outliers and simultaneously rate optimal or nearly rate optimal for a large class of error distributions including gaussian laplace cauchy and uniform among others

minimax nonparametric classification ii model selection for adaptation for pt i see ibid vol 45 no 7 p 2271 84 1999 we study nonparametric estimation of a conditional probability for classification based on a collection of finite dimensional models for the sake of flexibility different types of models linear or nonlinear are allowed as long as each satisfies a dimensionality assumption we show that with a suitable model selection criterion the penalized maximum likelihood estimator has a risk bounded by an index of resolvability expressing a good tradeoff among approximation error estimation error and model complexity the bound does not require any assumption on the target conditional probability and can be used to demonstrate the adaptivity of estimators based on model selection examples are given with both splines and neural nets and problems of high dimensional estimation are considered the resulting adaptive estimator is shown to behave optimally or near optimally over sobolev classes with unknown orders of interaction and smoothness and classes of integrable fourier transform of gradient in terms of rates of convergence the performance is the same as if one knew which of them contains the true conditional probability in advance the corresponding classifier also converges optimally or nearly optimally simultaneously over these classes

concentration inequalities and asymptotic results for ratio type empirical processes let be a class of measurable functions on a measurable space s mathcal s with values in 0 1 and let pn n 1 i 1n xi be the empirical measure based on an i i d sample x1 xn from a probability distribution p on s mathcal s we study the behavior of suprema of the following type sup r n textless sigma p f leq delta n frac p n f pf phi sigma p f where pf var1 2pf and is a continuous strictly increasing function with 0 0 using talagrand s concentration inequality for empirical processes we establish concentration inequalities for such suprema and use them to derive several results about their asymptotic behavior expressing the conditions in terms of expectations of localized suprema of empirical processes we also prove new bounds for expected values of sup norms of empirical processes in terms of the largest pf and the l2 p norm of the envelope of the function class which are especially suited for estimating localized suprema with this technique we extend to function classes most of the known results on ratio type suprema of empirical processes including some of alexander s results for vc classes of sets we also consider applications of these results to several important problems in nonparametric statistics and in learning theory including general excess risk bounds in empirical risk minimization and their versions for l2 regression and classification and ratio type bounds for margin distributions in classification

hellinger consistency of certain nonparametric maximum likelihood estimators consider a class mathscr p p theta theta in theta of probability measures on a measurable space mathscr x mathscr a dominated by a sigma finite measure mu let f theta dp theta d mu theta in theta and let theta n be a maximum likelihood estimator based on n independent observations from p theta 0 theta 0 in theta we use results from empirical process theory to obtain convergence for the hellinger distance h f hat theta n f theta 0 under certain entropy conditions on the class of densities f theta theta in theta the examples we present are a model with interval censored observations smooth densities monotone densities and convolution models in most examples the convexity of the class of densities is of special importance

exponential series estimator of multivariate densities abstract we present an exponential series estimator ese of multivariate densities which has an appealing information theoretic interpretation for a d dimensional random variable x with density p 0 the ese takes the form p x exp i 1 0 m 1 i d 0 m d i i x where i are some real valued linearly independent functions defined on the support of p 0 we derive the convergence rate of the ese in terms of the kullback leibler information criterion the integrated squared error and some other metrics we also derive its almost sure uniform convergence rate we then establish the asymptotic normality of p we undertake two sets of monte carlo experiments the first experiment examines the ese performance using mixtures of multivariate normal densities the second estimates copula density functions the results demonstrate the efficacy of the ese an empirical application on the joint distributions of stock returns is presented

a distributed one step estimator distributed statistical inference has recently attracted enormous attention many existing work focuses on the averaging estimator e g zhang and duchi j mach learn res 14 3321 3363 2013 together with many others we propose a one step approach to enhance a simple averaging based distributed estimator by utilizing a single newton raphson updating we derive the corresponding asymptotic properties of the newly proposed estimator we find that the proposed one step estimator enjoys the same asymptotic properties as the idealized centralized estimator in particular the asymptotic normality was established for the proposed estimator while other competitors may not enjoy the same property the proposed one step approach merely requires one additional round of communication in relative to the averaging estimator so the extra communication burden is insignificant the proposed one step approach leads to a lower upper bound of the mean squared error than other alternatives in finite sample cases numerical examples show that the proposed estimator outperforms the simple averaging estimator with a large margin in terms of the sample mean squared error a potential application of the one step approach is that one can use multiple machines to speed up large scale statistical inference with little compromise in the quality of estimators the proposed method becomes more valuable when data can only be available at distributed machines with limited communication bandwidth

a simple lemma on greedy approximation in hilbert space and convergence rates for projection pursuit regression and neural network training a general convergence criterion for certain iterative sequences in hilbert space is presented for an important subclass of these sequences estimates of the rate of convergence are given under very mild assumptions these results establish an 0 1 f4n nonsampling convergence rate for projection pursuit regression and neural network training where n represents the number of ridge functions neurons or coefficients in a greedy basis expansion

estimating divergence functionals and the likelihood ratio by convex risk minimization we develop and analyze m estimation methods for divergence functionals and the likelihood ratios of two probability distributions our method is based on a nonasymptotic variational characterization of f divergences which allows the problem of estimating divergences to be tackled via convex empirical risk optimization the resulting estimators are simple to implement requiring only the solution of standard convex programs we present an analysis of consistency and convergence for these estimators given conditions only on the ratios of densities we show that our estimators can achieve optimal minimax rates for the likelihood ratio and the divergence functionals in certain regimes we derive an efficient optimization algorithm for computing our estimates and illustrate their convergence behavior and practical viability by simulations

high dimensional analysis of semidefinite relaxations for sparse principal components principal component analysis pca is a classical method for dimensionality reduction based on extracting the dominant eigenvectors of the sample covariance matrix however pca is well known to behave poorly in the large p small n setting in which the problem dimension p is comparable to or larger than the sample size n this paper studies pca in this high dimensional regime but under the additional assumption that the maximal eigenvector is sparse say with at most k nonzero components we consider a spiked covariance model in which a base matrix is perturbed by adding a k sparse maximal eigenvector and we analyze two computationally tractable methods for recovering the support set of this maximal eigenvector as follows a a simple diagonal thresholding method which transitions from success to failure as a function of the rescaled sample size dia n p k n k 2 log p k and b a more sophisticated semidefinite programming sdp relaxation which succeeds once the rescaled sample size sdp n p k n klog p k is larger than a critical threshold in addition we prove that no method including the best method which has exponential time complexity can succeed in recovering the support if the order parameter sdp n p k is below a threshold our results thus highlight an interesting trade off between computational and statistical efficiency in high dimensional inference

density estimation by stochastic complexity the results by p hall and e j hannan 1988 on optimization of histogram density estimators with equal bin widths by minimization of the stochastic complexity are extended and sharpened in two separate ways as the first contribution two generalized histogram estimators are constructed the first has unequal bin widths which together with the number of the bins are determined by minimization of the stochastic complexity using dynamic programming the other estimator consists of a mixture of equal bin width estimators each of which is defined by the associated stochastic complexity as the main contribution in the present work two theorems are proved which together extend the universal coding theorems to a large class of data generating densities the first gives an asymptotic upper bound for the code redundancy in the order of magnitude achieved with a special predictive type of histogram estimator which sharpens a related bound the second theorem states that this bound cannot be improved upon by any code whatsoever

approximation dans les espaces metriques et theorie de l estimation we investigate the relations between the speed of estimation and the metric structure of the parameter space especially in the case when its metric dimension is infinite given some distance d on generally hellinger distance in the case of n i i d variables we consider the minimax risk for n observations r n q mathop inf sup limits t n theta in theta text ie theta d q theta t n t n being any estimate of we shall look for functions r such that for positive constants c 1 q and c 2 q c 1 r q n r n q c2 r q n r n is the speed of estimation and we shall show under fairly general conditions including i i d variables and regular cases of markov chains and stationnary gaussian processes that r n is determined up to multiplicative constants by the metric structure of we shall also give a construction for some sort of universal estimates the risk of which is bounded by c 2 r q n in all cases where the preceding theory applies

uniform asymptotics for finite time ruin probability of a bidimensional risk model abstract consider a continuous time bidimensional risk model with constant force of interest in which the claim sizes from the same business are heavy tailed and upper tail asymptotically independent we investigate two cases one is that the two claim number processes are arbitrarily dependent and the other is that the two corresponding claim inter arrival times from different lines are positively quadrant dependent some uniformly asymptotic formulas for finite time ruin probability are established

on the consistency of bayesian function approximation using step functions we consider the problem of estimating a step function with an unknown number of jumps under noisy observations on a grid under mild assumptions the bayesian approach is shown to produce a consistent estimate even when the underlying true function is not piecewise constant a simple prior is constructed to illustrate our assumptions

universal coding information prediction and estimation a connection between universal codes and the problems of prediction and statistical estimation is established a known lower bound for the mean length of universal codes is sharpened and generalized and optimum universal codes constructed the bound is defined to give the information in strings relative to the considered class of processes the earlier derived minimum description length criterion for estimation of parameters including their number is given a fundamental information theoretic justification by showing that its estimators achieve the information in the strings it is also shown that one cannot do prediction in gaussian autoregressive moving average arma processes below a bound which is determined by the information in the data

estimation of integral functionals of a density let be a smooth function of k 2 variables we shall investigate in this paper the rates of convergence of estimators of t f f f x f x f k x x dx when f belongs to some class of densities of smoothness s we prove that when 2k 1 4 one can define an estimator t n of t f based on n i i d observations of density f on the real line which converges at the semiparametric rate 1 n on the other hand when s 2k 1 4 t f cannot be estimated at a rate faster than n with 4 s k 4s 1 we shall also provide some extensions to the multidimensional case those results extend previous works of levit of bickel and ritov and of donoho and nussbaum on estimation of quadratic functionals

minimax nonparametric classification i rates of convergence this paper studies minimax aspects of nonparametric classification we first study minimax estimation of the conditional probability of a class label given the feature variable this function say f is assumed to be in a general nonparametric class we show the minimax rate of convergence under square l sub 2 loss is determined by the massiveness of the class as measured by metric entropy the second part of the paper studies minimax classification the loss of interest is the difference between the probability of misclassification of a classifier and that of the bayes decision as is well known an upper bound on risk for estimating f gives an upper bound on the risk for classification but the rate is known to be suboptimal for the class of monotone functions this suggests that one does not have to estimate f well in order to classify well however we show that the two problems are in fact of the same difficulty in terms of rates of convergence under a sufficient condition which is satisfied by many function classes including besov sobolev lipschitz and bounded variation this is somewhat surprising in view of a result of devroye gorfi and lugosi see a probabilistic theory of pattern recognition new york springer verlag 1996

versal unfoldings for linear retarded functional differential equations we consider parametrized families of linear retarded functional differential equations rfdes projected onto finite dimensional invariant manifolds and address the question of versality of the resulting parametrized family of linear ordinary differential equations a sufficient criterion for versality is given in terms of readily computable quantities in the case where the unfolding is not versal we show how to construct a perturbation of the original linear rfde in terms of delay differential operators whose finite dimensional projection generates a versal unfolding we illustrate the theory with several examples and comment on the applicability of these results to bifurcation analyses of nonlinear rfdes

lower bounds for the minimax risk using f divergences and applications lower bounds involving f divergences between the underlying probability measures are proved for the minimax risk in estimation problems our proofs just use simple convexity facts special cases and straightforward corollaries of our bounds include well known inequalities for establishing minimax lower bounds such as fano s inequality pinsker s inequality and inequalities based on global entropy conditions two applications are provided a new minimax lower bound for the reconstruction of convex bodies from noisy support function measurements and a different proof of a recent minimax lower bound for the estimation of a covariance matrix

minimal penalties for gaussian model selection this paper is mainly devoted to a precise analysis of what kind of penalties should be used in order to perform model selection via the minimiza tion of a penalized least squares type criterion within some general gaussian framework including the classical ones as compared to our previous paper on this topic birge and massart in j eur math soc 3 203 268 2001 more elaborate forms of the penalties are given which are shown to be in some sense optimal we indeed provide more precise upper bounds for the risk of the penalized estimators and lower bounds for the penalty terms showing that the use of smaller penalties may lead to disastrous results these lower bounds may also be used to design a practical strategy that allows to estimate the penalty from the data when the amount of noise is unknown we provide an illustra tion of the method for the problem of estimating a piecewise constant signal in gaussian noise when neither the number nor the location of the change points are known

basic renewal theorems for random walks with widely dependent increments abstract in this paper we derive some basic renewal theorems for random walks with widely dependent increments which contain some common negatively dependent random variables r v s some positively dependent r v s and some others for this purpose we investigate uniform integrability for related counting processes and the strong law of large numbers for the widely dependent r v s

fast rates in statistical and online learning the speed with which a learning algorithm converges as it is presented with more data is a central problem in machine learning a fast rate of convergence means less data is needed for the same level of performance the pursuit of fast rates in online and statistical learning has led to the discovery of many conditions in learning theory under which fast learning is possible we show that most of these conditions are special cases of a single unifying condition that comes in two forms the central condition for proper learning algorithms that always output a hypothesis in the given model and stochastic mixability for online algorithms that may make predictions outside of the model we show that under surprisingly weak assumptions both conditions are in a certain sense equivalent the central condition has a re interpretation in terms of convexity of a set of pseudoprobabilities linking it to density estimation under misspecification for bounded losses we show how the central condition enables a direct proof of fast rates and we prove its equivalence to the bernstein condition itself a generalization of the tsybakov margin condition both of which have played a central role in obtaining fast rates in statistical learning yet while the bernstein condition is two sided the central condition is one sided making it more suitable to deal with unbounded losses in its stochastic mixability form our condition generalizes both a stochastic exp concavity condition identified by juditsky rigollet and tsybakov and vovk s notion of mixability our unifying conditions thus provide a substantial step towards a characterization of fast rates in statistical learning similar to how classical mixability characterizes constant regret in the sequential prediction with expert advice setting

random walks with non convolution equivalent increments and their applications abstract this paper mainly presents some global and local asymptotic estimates for the tail probabilities of the supremum and overshoot of a random walk in the intermediate case where the related distributions of the increments of the random walk may not belong to the convolution equivalent distribution class some of the obtained results can include the classical results for this the paper first introduces some new distribution classes using the transform of distributions and investigates their properties and relations with some other existing distribution classes based on the above results some equivalent conditions for the global and local asymptotics of the transform of the distribution of the supremum of the above random walk are given applying these results to risk theory and infinitely divisible laws the paper obtains some asymptotic estimates for the ruin probability and the local ruin probability of the renewal risk model with non convolution equivalent claims and the global and local asymptotics of an infinitely divisible law with a non convolution equivalent levy measure

fast learning rate of multiple kernel learning trade off between sparsity and smoothness we investigate the learning rate of multiple kernel leaning mkl with l1 and elastic net regularizations the elastic net regularization is a composition of an l1 regularizer for inducing the sparsity and an l2 regularizer for controlling the smoothness we focus on a sparse setting where the total number of kernels is large but the number of non zero components of the ground truth is relatively small and show sharper convergence rates than the learning rates ever shown for both l1 and elastic net regularizations our analysis shows there appears a trade off between the sparsity and the smoothness when it comes to selecting which of l1 and elastic net regularizations to use if the ground truth is smooth the elastic net regularization is preferred otherwise the l1 regularization is preferred

estimation of nonparametric conditional moment models with possibly nonsmooth generalized residuals this paper studies nonparametric estimation of conditional moment restrictions in which the generalized residual functions can be nonsmooth in the unknown functions of endogenous variables this is a nonparametric nonlinear instrumental variables iv problem we propose a class of penalized sieve minimum distance psmd estimators which are minimizers of a penalized empirical minimum distance criterion over a collection of sieve spaces that are dense in the infinite dimensional function parameter space some of the psmd procedures use slowly growing finite dimensional sieves with flexible penalties or without any penalty others use large dimensional sieves with lower semicompact and or convex penalties we establish their consistency and the convergence rates in banach space norms such as a sup norm or a root mean squared norm allowing for possibly noncompact infinite dimensional parameter spaces for both mildly and severely ill posed nonlinear inverse problems our convergence rates in hilbert space norms such as a root mean squared norm achieve the known minimax optimal rate for the nonparametric mean iv regression we illustrate the theory with a nonparametric additive quantile iv regression we present a simulation study and an empirical application of estimating nonparametric quantile iv engel curves

rates of convergence of minimum distance estimators and kolmogorov s entropy on construit des estimateurs robustes et on montre que le taux de convergence resultant depend naturellement d une fonction entropie

normal forms for retarded functional differential equations with parameters and applications to hopf bifurcation abstract the paper addresses for retarded functional differential equations fdes with parameters the computation of normal forms associated with the flow on a finite dimensional invariant manifold tangent to an invariant space for the infinitesimal generator of the linearized equation at a singularity the analysis is based on the theory previously developed for fdes without parameters by considering an extension of a phase space appropriate to the computation of these normal forms as an application the generic hopf bifurcation for scalar retarded fdes is considered

risk bounds for model selection via penalization performance bounds for criteria for model selection are devel oped using recent theory for sieves the model selection criteria are based on an empirical loss or contrast function with an added penalty term moti vated by empirical process theory and roughly proportional to the number of parameters needed to describe the model divided by the number of ob servations most of our examples involve density or regression estimation settings and we focus on the problem of estimating the unknown density or regression function we show that the quadratic risk of the minimum penal ized empirical contrast estimatoris bounded by an index of the accuracy of the sieve this accuracy index quanties the trade off among the candidate models between the approximation error and parameter dimension relative to sample size if we choose a list of models which exhibit good approximation prop erties with respect to different classes of smoothness the estimator can be simultaneously minimax rate optimal in each of those classes this is what is usually called adaptation the type of classes of smoothness in which one gets adaptation depends heavily on the list of models if too many models are involved in order to get accurate approximation of many wide classes of functions simultaneously it may happen that the estimator is only approx

probability inequalities for likelihood ratios and convergence rates of sieve mles let y 1 y n be independent identically distributed with density p 0 and let f be a space of densities we show that the supremum of the likelihood ratios i 1 n p y i p 0 y i where the supremum is over p f with p l 2 p 0 1 2 2 e is exponentially small with probability exponentially close to 1 the exponent is proportional to ne 2 the only condition required for this to hold is that e exceeds a value determined by the bracketing hellinger entropy of f a similar inequality also holds if we replace f by f n and p 0 by q n where q n is an approximation to p 0 in a suitable sense these results are applied to establish rates of convergence of sieve mles furthermore weak conditions are given under which the optimal rate e n defined by h e n f ne n 2 where h f is the hellinger entropy of f is nearly achievable by sieve estimators

universal donsker classes and metric entropy when mathfrak f is a universal donsker class then for independent indetically distributed i i d observation mathbf x 1 ldots mathbf x n with an unknown law p for any mathfrak f i in mathfrak f i 1 ldots m quad n 1 2 left mathfrak f 1 left mathbf x 1 right ldots mathfrak f i left mathbf x n right right 1 leq i leq m is asymptotically normal with mean vector n 1 2 left int mathfrak f i left mathbf x n right d mathbf p left x right right 1 leq i leq m and covariance matrix int mathfrak f i mathfrak f j d mathbf p int mathfrak f id mathbf p int mathfrak f jd mathbf p uniformly for mathfrak f i in mathfrak f then for certain statistics formed frome the mathfrak f i left mathbf x k right even where mathfrak f i may be chosen depending on the mathbf x k there will be asymptotic distribution as n rightarrow infty for example for mathbf x 2 statistics where f i are indicators of disjoint intervals depending suitably on mathbf x 1 ldots mathbf x n whose union is the real line mathbf x 2 quadratic forms have limiting distributions roy 1956 and watson 1958 which may however not be mathbf x 2 distributions and may depend on p chernoff and lehmann 1954 universal donsker classes of sets are up to mild measurability conditions just classes satisfying the vapnik cervonenkis comdinatorial conditions defined later in this section donsker the vapnik cervonenkis combinatorial conditions defined later in this section durst and dudley 1981 and dudley 1984 chapter 11 the use of such classes allows a variety of extensions of the roy watson results to general multidimensional sample spaces pollard 1979 and moore and subblebine 1981 vapnik and cervonenkis 1974 indicated application of their families of sets to classification pattern recognition problems more recently the classes have been applied to tree structured classifiacation breiman friedman olshen and stone 1984 chapter 12

the consistency of the nearest neighbor estimator of the density function based on wod samples abstract in this paper the consistency of the nearest neighbor estimator of the density function based on widely orthant dependent wod in short samples is investigated the convergence rate of strong consistency the complete consistency the uniformly complete consistency and uniformly strong consistency of the nearest neighbor estimator of the density function based on wod samples are established our results established in the paper generalize or improve the corresponding ones for independent samples and some negatively dependent samples

restricted strong convexity and weighted matrix completion optimal bounds with noise we consider the matrix completion problem under a form of row column weighted entrywise sampling including the case of uniformentrywise sampling as a special case we analyze the associated random observation operator and prove that with high probability it satisfies a form of restricted strong convexity with respect to weighted frobenius norm using this property we obtain as corollaries a number of error bounds on matrix completion in the weighted frobenius norm under noisy sampling and for both exact and near low rank matrices our results are based on measures of the spikiness and low rankness of matrices that are less restrictive than the incoherence conditions imposed in previous work our technique involves an m estimator that includes controls on both the rank and spikiness of the solution and we establish non asymptotic error bounds in weighted frobenius norm for recovering matrices lying with lq balls of bounded spikiness using information theoretic methods we show that no algorithm can achieve better estimates up to a logarithmic factor over these same sets showing that our conditions on matrices and associated rates are essentially optimal

normal forms for retarded functional differential equations and applications to bogdanov takens singularity abstract the paper addresses for retarded functional differential equations fdes the computation of normal forms associated with the flow on a finite dimensional invariant manifold tangent to invariant spaces for the infinitesimal generator of the linearized equation at a singularity a phase space appropriate to the computation of these normal forms is introduced and adequate nonresonance conditions for the computation of the normal forms are derived as an application the general situation of bogdanov takens singularity and its versal unfolding for scalar retarded fdes with nondegeneracy at second order is considered both in the general case and in the case of differential delay equations of the form t x t x t 1

minimax bounds for sparse pca with noisy high dimensional data we study the problem of estimating the leading eigenvectors of a high dimensional population covariance matrix based on independent gaussian observations we establish a lower bound on the minimax risk of estimators under the l2 loss in the joint limit as dimension and sample size increase to infinity under various models of sparsity for the population eigenvectors the lower bound on the risk points to the existence of different regimes of sparsity of the eigenvectors we also propose a new method for estimating the eigenvectors by a two stage coordinate selection scheme

uniform asymptotics of the finite time ruin probability for all times abstract in this paper we study the asymptotics of the finite time ruin probability for a generalized renewal risk model with independent strong subexponential claim sizes and widely lower orthant dependent inter occurrence times this widely lower orthant dependence structure can include some common negative dependent random variables r v s such as negatively lower orthant dependent r v s some positive dependent r v s and some other dependent r v s for this generalized renewal risk model we show that the asymptotics of the finite time ruin probability hold uniformly for all times in the sense that the magnitude of the initial reserve of some insurance portfolio does not influence the length of the operating time further the uniform asymptotics of the random time ruin probability in the above model have been obtained

approximation of density functions by sequences of exponential families 1 introduction consider the estimation of a probability density func tion p x defined on a bounded interval we approximate the logarithm of the density by a basis function expansion consisting of polynomials splines or trigonometric series the expansion yields a regular exponential family within which we estimate the density by the method of maximum likelihood this method of density estimation arises by application of the principle of maxi mum entropy or minimum relative entropy subject to empirical constraints we show that if the logarithm of the density has r square integrable deriva tives f idr log p12 ox then the sequence of density estimators p converges to p in the sense of relative entropy kullback leibler fp log p pn at rate opr 1 m2r m n as m x c and m2 n 0 in the spline and trigonometric cases and m3 n o 0 in the polynomial case where m is the dimension of the family and n is the sample size boundary conditions are assumed for the density in the trigonometric case this convergence rate specializes to op r n 2r 2r 1 by setting m nl 2r 1 when the log density is known to have degree of smoothness at least r analogous convergence results for the relative entropy are shown to hold in general for any class of log density functions and sequence of finite dimensional linear spaces having l2 and l approximation properties the approximation of log densities using polynomials has previously been considered by neyman 1937 to define alternatives for goodness of fit tests by good 1963 as an application of the method of maximum entropy or minimum relative entropy by crain 1974 1976a b 1977 who demonstrates existence and consistency of the maximum likelihood estimator and by mead and

approximation of least squares regression on nested subspaces on presente une methode generale pour approximer le biais et la variance dans une echelle de normes hilbertiennes

information theory and superefficiency the asymptotic risk of efficient estimators with kullback leibler loss in smoothly parametrized statistical models is k 2n where k is the parameter dimension and n is the sample size under fairly general conditions we given a simple information theoretic proof that the set of parameter values where any arbitrary estimator is superefficient is negligible the proof is based on a result of rissanen that codes have asymptotic redundancy not smaller than k 2 log n except in a set of measure 0

concentration around the mean for maxima of empirical processes in this paper we give optimal constants in talagrand s concentration inequalities for maxima of empirical processes associated to independent and eventually nonidentically distributed random variables our approach is based on the entropy method introduced by ledoux

information theoretic upper and lower bounds for statistical estimation in this paper we establish upper and lower bounds for some statistical estimation problems through concise information theoretic arguments our upper bound analysis is based on a simple yet general inequality which we call the information exponential inequality we show that this inequality naturally leads to a general randomized estimation method for which performance upper bounds can be obtained the lower bounds applicable for all statistical estimators are obtained by original applications of some well known information theoretic inequalities and approximately match the obtained upper bounds for various important problems moreover our framework can be regarded as a natural generalization of the standard minimax framework in that we allow the performance of the estimator to vary for different possible underlying distributions according to a predefined prior

optimal rates for zero order convex optimization the power of two function evaluations we consider derivative free algorithms for stochastic and nonstochastic convex optimization problems that use only function values rather than gradients focusing on nonasymptotic bounds on convergence rates we show that if pairs of function values are available algorithms for d dimensional optimization that use gradient estimates based on random perturbations suffer a factor of at most sqrt d in convergence rate over traditional stochastic gradient methods we establish such results for both smooth and nonsmooth cases sharpening previous analyses that suggested a worse dimension dependence and extend our results to the case of multiple m ge 2 evaluations we complement our algorithmic development with information theoretic lower bounds on the minimax convergence rate of such problems establishing the sharpness of our achievable results up to constant sometimes logarithmic factors

minimum complexity density estimation the authors introduce an index of resolvability that is proved to bound the rate of convergence of minimum complexity density estimators as well as the information theoretic redundancy of the corresponding total description length the results on the index of resolvability demonstrate the statistical effectiveness of the minimum description length principle as a method of inference the minimum complexity estimator converges to true density nearly as fast as an estimator based on prior knowledge of the true subclass of densities interpretations and basic properties of minimum complexity estimators are discussed some regression and classification problems that can be examined from the minimum description length framework are considered

mutual information metric entropy and cumulative relative entropy risk assume p theta theta epsilon theta is a set of probability distributions with a common dominating measure on a complete separable metric space y a state theta epsilon theta is chosen by nature a statistician obtains n independent observations y 1 dots y n from y distributed according to p theta for each time t between 1 and n based on the observations y 1 dots y t 1 the statistician produces an estimated distribution hat p t for p theta and suffers a loss l p theta hat p t the cumulative risk for the statistician is the average total loss up to time n of special interest in information theory data compression mathematical finance computational learning theory and statistical mechanics is the special case when the loss l p theta hat p t is the relative entropy between the true distribution p theta and the estimated distribution hat p t here the cumulative bayes risk from time 1 to n is the mutual information between the random parameter theta and the observations y 1 dots y n new bounds on this mutual information are given in terms of the laplace transform of the hellinger distance between pairs of distributions indexed by parameters in theta from these bounds on the cumulative minimax risk are given in terms of the metric entropy of theta with respect to the hellinger distance the assumptions required for these bounds are very general and do not depend on the choice of the dominating measure they apply to both finite and infinite dimensional theta they apply in some cases where y is infinite dimensional in some cases where y is not compact in some cases where the distributions are not smooth and in some parametric cases where asymptotic normality of the posterior distribution fails

local rademacher complexities and oracle inequalities in risk minimization let be a class of measurable functions f s 0 1 defined on a probability space s mathcal a p given a sample x1 xn of i i d random variables taking values in s with common distribution p let pn denote the empirical measure based on x1 xn we study an empirical risk minimization problem pnf min f given a solution fn of this problem the goal is to obtain very general upper bounds on its excess risk mathcal e p hat f n p hat f n inf f in mathcal f pf expressed in terms of relevant geometric parameters of the class using concentration inequalities and other empirical processes tools we obtain both distribution dependent and data dependent upper bounds on the excess risk that are of asymptotically correct order in many examples the bounds involve localized sup norms of empirical and rademacher processes indexed by functions from the class we use these bounds to develop model selection techniques in abstract risk minimization problems that can be applied to more specialized frameworks of regression and classification

sparsity and error analysis of empirical feature based regularization schemes we consider a learning algorithm generated by a regularization scheme with a concave regularizer for the purpose of achieving sparsity and good learning rates in a least squares regression setting the regularization is induced for linear combinations of empirical features constructed in the literatures of kernel principal component analysis and kernel projection machines based on kernels and samples in addition to the separability of the involved optimization problem caused by the empirical features we carry out sparsity and error analysis giving bounds in the norm of the reproducing kernel hilbert space based on a priori conditions which do not require assumptions on sparsity in terms of any basis or system in particular we show that as the concave exponent q of the concave regularizer increases to 1 the learning ability of the algorithm improves some numerical simulations for both artificial and real mhc peptide binding data involving the lq regularizer and the scad penalty are presented to demonstrate the sparsity and error analysis

fast learning rates for plug in classifiers it has been recently shown that under the margin or low noise assumption there exist classifiers attaining fast rates of convergence of the excess bayes risk that is rates faster than n 1 2 the work on this subject has suggested the following two conjectures i the best achievable fast rate is of the order n 1 and ii the plug in classifiers generally converge more slowly than the classifiers based on empirical risk minimization we show that both conjectures are not correct in particular we construct plug in classifiers that can achieve not only fast but also super fast rates that is rates faster than n 1 we establish minimax lower bounds showing that the obtained rates cannot be improved

a max norm constrained minimization approach to 1 bit matrix completion we consider in this paper the problem of noisy 1 bit matrix completion under a general non uniform sampling distribution using the max norm as a convex relaxation for the rank a max norm constrained maximum likelihood estimate is introduced and studied the rate of convergence for the estimate is obtained information theoretical methods are used to establish a minimax lower bound under the general sampling model the minimax upper and lower bounds together yield the optimal rate of convergence for the frobenius norm loss computational algorithms and numerical performance are also discussed

the minimum description length principle in coding and modeling we review the principles of minimum description length and stochastic complexity as used in data compression and statistical modeling stochastic complexity is formulated as the solution to optimum universal coding problems extending shannon s basic source coding theorem the normalized maximized likelihood mixture and predictive codings are each shown to achieve the stochastic complexity to within asymptotically vanishing terms we assess the performance of the minimum description length criterion both from the vantage point of quality of data compression and accuracy of statistical inference context tree modeling density estimation and model selection in gaussian linear regression serve as examples

parametric or nonparametric a parametricness index for model selection in model selection literature two classes of criteria perform well asymptotically in different situations bayesian information criterion bic as a representative is consistent in selection when the true model is finite dimensional parametric scenario akaike s information criterion aic performs well in an asymptotic efficiency when the true model is infinite dimensional nonparametric scenario but there is little work that addresses if it is possible and how to detect the situation that a specific model selection problem is in in this work we differentiate the two scenarios theoretically under some conditions we develop a measure parametricness index pi to assess whether a model selected by a potentially consistent procedure can be practically treated as the true model which also hints on aic or bic is better suited for the data for the goal of estimating the regression function a consequence is that by switching between aic and bic based on the pi the resulting regression estimator is simultaneously asymptotically efficient for both parametric and nonparametric scenarios in addition we systematically investigate the behaviors of pi in simulation and real data and show its usefulness

a general minimax result for relative entropy suppose nature picks a probability measure p sub spl theta on a complete separable metric space x at random from a measurable set p sub spl theta p spl theta spl theta spl isin spl theta then without knowing spl theta a statistician picks a measure q on s finally the statistician suffers a loss d p sub 0 spl par q the relative entropy between p sub spl theta and q we show that the minimax and maximin values of this game are always equal and there is always a minimax strategy in the closure of the set of all bayes strategies this generalizes previous results of gallager 1979 and davisson and leon garcia 1980

an asymptotic property of model selection criteria probability models are estimated by use of penalized log likelihood criteria related to akaike 1973 information criterion aic and minimum description length mdl the accuracies of the density estimators are shown to be related to the tradeoff between three terms the accuracy of approximation the model dimension and the descriptive complexity of the model classes the asymptotic risk is determined under conditions on the penalty term and is shown to be minimax optimal for some cases as an application we show that the optimal rate of convergence is simultaneously achieved for log densities in sobolev spaces w sub 2 sup s u without knowing the smoothness parameter s and norm parameter u in advance applications to neural network models and sparse density function estimation are also provided

covering numbers for convex functions in this paper we study the covering numbers of the space of convex and uniformly bounded functions in multidimension we find optimal upper and lower bounds for the epsilon covering number of cal c a b d b in the l p metric 1 leq p in terms of the relevant constants where d geq 1 a b 0 and cal c a b d b denotes the set of all convex functions on a b d that are uniformly bounded by b we summarize previously known results on covering numbers for convex functions and also provide alternate proofs of some known results our results have direct implications in the study of rates of convergence of empirical minimization procedures as well as optimal convergence rates in the numerous convexity constrained function estimation problems

convergence rates of posterior distributions we consider the asymptotic behavior of posterior distributions and bayes estimators for infinite dimensional statistical models we give general results on the rate of convergence of the posterior measure these are applied to several examples including priors on finite sieves log spline models dirichlet processes and interval censoring

robust statistical learning with lipschitz and convex loss functions we obtain estimation and excess risk bounds for empirical risk minimizers erm and minmax median of means mom estimators based on loss functions that are both lipschitz and convex results for the erm are derived under weak assumptions on the outputs and subgaussian assumptions on the design as in alquier et al estimation bounds and sharp oracle inequalities of regularized procedures with lipschitz loss functions arxiv 1702 01402 2017 the difference with alquier et al 2017 is that the global bernstein condition of this paper is relaxed here into a local assumption we also obtain estimation and excess risk bounds for minmax mom estimators under similar assumptions on the output and only moment assumptions on the design moreover the dataset may also contains outliers in both inputs and outputs variables without deteriorating the performance of the minmax mom estimators unlike alternatives based on mom s principle lecue and lerasle in ann stat 2017 lugosi and mendelson in jems 2016 the analysis of minmax mom estimators is not based on the small ball assumption sba of koltchinskii and mendelson int math res not imrn 23 12991 13008 2015 in particular the basic example of non parametric statistics where the learning class is the linear span of localized bases that does not satisfy sba saumard in bernoulli 24 3 2176 2203 2018 can now be handled finally minmax mom estimators are analysed in a setting where the local bernstein condition is also dropped out it is shown to achieve excess risk bounds with exponentially large probability under minimal assumptions insuring only the existence of all objects

center manifolds theorem for parameterized delay differential equations with applications to zero singularities abstract the goal of this paper is to develop a center manifold theory for delay differential equations with parameters as applications we use the center manifold theorem to establish fold and bogdanov takens bifurcations in particular we obtain the versal unfoldings of delayed predator prey systems with predator harvesting at the bogdanov takens singularity

minimax estimation in sparse canonical correlation analysis canonical correlation analysis is a widely used multivariate statistical technique for exploring the relation between two sets of variables this paper considers the problem of estimating the leading canonical correlation directions in high dimensional settings recently under the assumption that the leading canonical correlation directions are sparse various procedures have been proposed for many high dimensional applications involving massive data sets however there has been few theoretical justification available in the literature in this paper we establish rate optimal nonasymptotic minimax estimation with respect to an appropriate loss function for a wide range of model spaces two interesting phenomena are observed first the minimax rates are not affected by the presence of nuisance parameters namely the covariance matrices of the two sets of random variables though they need to be estimated in the canonical correlation analysis problem second we allow the presence of the residual canonical correlation directions however they do not influence the minimax rates under a mild condition on eigengap a generalized sin theta theorem and an empirical process bound for gaussian quadratic forms under rank constraint are used to establish the minimax upper bounds which may be of independent interest

geometrizing rates of convergence ii consider estimating a functional t f of an unknown distribution f in mathbf f from data x 1 cdots x n i i d f let omega varepsilon denote the modulus of continuity of the functional t over mathbf f computed with respect to hellinger distance for well behaved loss functions l t we show that inf t n sup mathbf f e fl t n t f is equivalent to l omega n 1 2 to within constants whenever t is linear and mathbf f is convex the same conclusion holds in three nonlinear cases estimating the rate of decay of a density estimating the mode and robust nonparametric regression we study the difficulty of testing between the composite infinite dimensional hypotheses h 0 t f leq t and h 1 t f geq t delta our results hold in the cases studied because the difficulty of the full infinite dimensional composite testing problem is comparable to the difficulty of the hardest simple two point testing subproblem

a lower bound on the error in nonparametric regression type problems let x1 y1 xn yn be a sample denote the conditional density of ylxi xi as f ylxi 0 xi and 0 an element of a metric space 9 d a lower bound is provided for the d error in estimating 0 the order of the bound depends on the local behavior of the kullback information of the conditional density as an application we consider the case where 1 is the space of q smooth functions on 0 i d metrized with the lr distance 1 r oo

properties of center manifolds the center manifold has a number of puzzling properties associated with the basic questions of existence uniqueness differentiability and analyticity which may cloud its profitable application in e g bifurcation theory this paper aims to deal with some of these subtle properties regarding existence and uniqueness it is shown that the cut off function appear ing in the usual existence proof is responsible for the selection of a single center manifold thereby hiding the inherent nonuniqueness conditions are given for different center manifolds at an equilibrium point to have a nonempty intersection this intersection will include at least the families of stationary and periodic solutions crossing through the equilibrium in the case of nonuniqueness the difference between any two center manifolds will be less than el exp e2x l with el and e2 constants which explains why the formal taylor expansions of different center manifolds are the same while the expansions do not converge the differentiability of a center manifold will in certain cases decrease when moving out of the origin and a simple example shows how the differentiability may be lost center manifolds are usually not analytic however an analytic manifold may exist which contains all periodic solutions of a certain type but which may otherwise not be invariant using this manifold a new and very simple proof of the lyapunov subcenter theorem is given

information theoretic asymptotics of bayes methods in the absence of knowledge of the true density function bayesian models take the joint density function for a sequence of n random variables to be an average of densities with respect to a prior the authors examine the relative entropy distance d sub n between the true density and the bayesian density and show that the asymptotic distance is d 2 log n c where d is the dimension of the parameter vector therefore the relative entropy rate d sub n n converges to zero at rate log n n the constant c which the authors explicitly identify depends only on the prior density function and the fisher information matrix evaluated at the true parameter value consequences are given for density estimation universal data compression composite hypothesis testing and stock market portfolio selection