information theoretic determination of minimax rates of convergence we present some general results determining minimax bounds on statistical risk for density estimation based on certain information theoretic considerations these bounds depend only on metric entropy conditions and are used to identify the minimax rates of convergence

eigenvalues of the laplacian in two dimensions the eigenvalue problem for the laplace operator in two dimensions is classical in mathematics and physics nevertheless computational methods for estimating the eigenvalues are still of much current interest particularly in applications to acoustic and electromagnetic waveguides although our primary interest is with the computational methods there are a number of theoretical results on the behavior of the eigenvalues and eigenfunctions that are useful for understanding the methods and in addition are of interest in themselves these results are discussed first and then the various computational methods that have been used to estimate the eigenvalues are reviewed with particular emphasis on methods that give error bounds some of the more powerful techniques available are illustrated by applying them to a model problem

on estimating a density using hellinger distance and some other strange facts on s interesse ici aux possibles vitesses d estimation d une densite a support compact dans m sous des hypotheses de regularite lorsque la perte est mesuree par le carre de la distance de hellinger on regardera aussi le cas connu des normes mathbb l q pour 1 q 2 et le risque est le risque minimax sur la famille on donne une methode generale permettant de traiter les problemes dans le cadre de la theorie de l approximation sous des conditions concernant l entropie metrique et l e capacite des familles a estimer les rapports entre regularite et entropie metrique etant bien connus nous pourrons aussi traiter les cas classiques et d autres qui le sont moins sous des conditions de bornes inferieures les vitesses sont celles observees pour la norme mathbb l q mais elles different dans le cas general on montre aussi que les restrictions sur la compacite du support ou la regularite sont indispensables et que leur absence mene a l impossibilite d obtenir une estimation raisonnable en ce sens que n importe quelle suite d estimateurs sera arbitrairement mauvaise en un point au moins un resultat analogue est vrai sous des conditions de regularite

on the generalization ability of on line learning algorithms in this paper it is shown how to extract a hypothesis with small risk from the ensemble of hypotheses generated by an arbitrary on line learning algorithm run on an independent and identically distributed i i d sample of data using a simple large deviation argument we prove tight data dependent bounds for the risk of this hypothesis in terms of an easily computable statistic m sub n associated with the on line performance of the ensemble via sharp pointwise bounds on m sub n we then obtain risk tail bounds for kernel perceptron algorithms in terms of the spectrum of the empirical kernel matrix these bounds reveal that the linear hypotheses found via our approach achieve optimal tradeoffs between hinge loss and margin size over the class of all linear functions an issue that was left open by previous results a distinctive feature of our approach is that the key tools for our analysis come from the model of prediction of individual sequences i e a model making no probabilistic assumptions on the source generating the data in fact these tools turn out to be so powerful that we only need very elementary statistical facts to obtain our final risk bounds

learning by mirror averaging given a collection of m different estimators or classifiers we study the problem of model selection type aggregation i e we construct a new estimator or classifier called aggregate which is nearly as good as the best among them with respect to a given risk criterion we define our aggregate by a simple recursive procedure which solves an auxiliary stochastic linear programming problem related to the original non linear one and constitutes a special case of the mirror averaging algorithm we show that the aggregate satisfies sharp oracle inequalities under some general assumptions the results allow one to construct in an easy way sharp adaptive nonparametric estimators for several problems including regression classification and density estimation

rates of convergence for minimum contrast estimators we shall present here a general study of minimum contrast estimators in a nonparametric setting although our results are also valid in the classical parametric case for independent observations these estimators include many of the most popular estimators in various situations such as maximum likelihood estimators least squares and other estimators of the regression function estimators for mixture models or deconvolution the main theorem relates the rate of convergence of those estimators to the entropy structure of the space of parameters optimal rates depending on entropy conditions are already known at least for some of the models involved and they agree with what we get for minimum contrast estimators as long as the entropy counts are not too large but under some circumstances large entropies or changes in the entropy structure due to local perturbations the resulting the rates are only suboptimal counterexamples are constructed which show that the phenomenon is real for non parametric maximum likelihood or regression this proves that under purely metric assumptions our theorem is optimal and that minimum contrast estimators happen to be suboptimal

density estimation by wavelet thresholding density estimation is a commonly used test case for nonparametric estimation methods we explore the asymptotic properties of estimators based on thresholding of empirical wavelet coefficients minimax rates of convergence are studied over a large range of besov function classes b pq and for a range of global l p error measures 1 p p some form of nonlinearity is essential since the minimax linear estimators are suboptimal by polynomial powers of n a second approach using an approximation of a gaussian white noise model in a mallows metric is used to attain exactly optimal rates of convergence for quadratic error p 2

risk bounds for statistical learning we propose a general theorem providing upper bounds for the risk of an empirical risk minimizer erm we essentially focus on the binary classification framework we extend tsybakov s analysis of the risk of an erm under margin type conditions by using concentration inequalities for conveniently weighted empirical processes this allows us to deal with ways of measuring the size of a class of classifiers other than entropy with bracketing as in tsybakov s work in particular we derive new risk bounds for the erm when the classification rules belong to some vc class under margin conditions and discuss the optimality of these bounds in a minimax sense

on density estimation in the view of kolmogorov s ideas in approximation theory the paper deals with upper and lower bounds for the quality of probability density estimation connections are established between these problems and the theory of approximation of functions particularly it is demonstrated how some of kolmogorov s concepts work

hellinger consistency of certain nonparametric maximum likelihood estimators consider a class mathscr p p theta theta in theta of probability measures on a measurable space mathscr x mathscr a dominated by a sigma finite measure mu let f theta dp theta d mu theta in theta and let theta n be a maximum likelihood estimator based on n independent observations from p theta 0 theta 0 in theta we use results from empirical process theory to obtain convergence for the hellinger distance h f hat theta n f theta 0 under certain entropy conditions on the class of densities f theta theta in theta the examples we present are a model with interval censored observations smooth densities monotone densities and convolution models in most examples the convexity of the class of densities is of special importance

efficient agnostic learning of neural networks with bounded fan in we show that the class of two layer neural networks with bounded fan in is efficiently learnable in a realistic extension to the probably approximately correct pac learning model in this model a joint probability distribution is assumed to exist on the observations and the learner is required to approximate the neural network which minimizes the expected quadratic error as special cases the model allows learning real valued functions with bounded noise learning probabilistic concepts and learning the best approximation to a target function that cannot be well approximated by the neural network the networks we consider have real valued inputs and outputs an unlimited number of threshold hidden units with bounded fan in and a bound on the sum of the absolute values of the output weights the number of computation steps of the learning algorithm is bounded by a polynomial in 1 spl epsiv 1 spl delta n and b where spl epsiv is the desired accuracy spl delta is the probability that the algorithm fails n is the input dimension and b is the bound on both the absolute value of the target which may be a random variable and the sum of the absolute values of the output weights in obtaining the result we also extended some results on iterative approximation of functions in the closure of the convex hull of a function class and on the sample complexity of agnostic learning with the quadratic loss function

optimal aggregation of classifiers in statistical learning classification can be considered as nonparametric estimation of sets where the risk is defined by means of a specific distance between sets associated with misclassification error it is shown that the rates of convergence of classifiers depend on two parameters the complexity of the class of candidate sets and the margin parameter the dependence is explicitly given indicating that optimal fast rates approaching o n 1 can be attained where n is the sample size and that the proposed classifiers have the property of robustness to the margin the main result of the paper concerns optimal aggregation of classifiers we suggest a classifier that automatically adapts both to the complexity and to the margin and attains the optimal fast rates up to a logarithmic factor

smooth discrimination analysis ties f and g can be based on the estimation of the set g x fx gx we consider applications where it is appropriate to assume that the region g has a smooth boundary or belongs to another nonparametric class of sets in particular this assumption makes sense if discrimination is used as a data analytic tool decision rules based on minimization of empirical risk over the whole class of sets and over sieves are considered their rates of convergence are obtained we show that these rules achieve optimal rates for estimation of g and optimal rates of convergence for bayes risks an interesting conclusion is that the optimal rates for bayes risks can be very fast in particular faster than the parametric root n rate these fast rates cannot be guaranteed for plug in rules

from ɛ entropy to kl entropy analysis of minimum information complexity density estimation we consider an extension of e entropy to a kl divergence based complexity measure for randomized density estimation methods based on this extension we develop a general information theoretical inequality that measures the statistical complexity of some deterministic and randomized density estimators consequences of the new inequality will be presented in particular we show that this technique can lead to improvements of some classical results concerning the convergence of minimum description length and bayesian posterior distributions moreover we are able to derive clean finite sample convergence bounds that are not obtainable using previous approaches

enhancing the generality of fuzzy relational models for control abstract a promising area of research in fuzzy control is the model based fuzzy controller at the heart of this approach is a fuzzy relational model of the process to be controlled since this model is identified directly from process input output data it is likely that holes will be present in the identified relational model these holes are real problems when the model is incorporated into a model based controller since the model will be unable to make any predictions whatsoever if the system drifts into an unknown region the present work deals with the completeness of the fuzzy relational model which forms the core of the controller this work proposes a scheme of post processing to fiil in the fuzzy relational model once it has been built and thereby improve its applicability for on line control a comparative study of the post processed model and conventional relational model is presented for box jenkins data identification system and a real time highly non linear application of ph control identification

a simple lemma on greedy approximation in hilbert space and convergence rates for projection pursuit regression and neural network training a general convergence criterion for certain iterative sequences in hilbert space is presented for an important subclass of these sequences estimates of the rate of convergence are given under very mild assumptions these results establish an 0 1 f4n nonsampling convergence rate for projection pursuit regression and neural network training where n represents the number of ridge functions neurons or coefficients in a greedy basis expansion

density estimation by stochastic complexity the results by p hall and e j hannan 1988 on optimization of histogram density estimators with equal bin widths by minimization of the stochastic complexity are extended and sharpened in two separate ways as the first contribution two generalized histogram estimators are constructed the first has unequal bin widths which together with the number of the bins are determined by minimization of the stochastic complexity using dynamic programming the other estimator consists of a mixture of equal bin width estimators each of which is defined by the associated stochastic complexity as the main contribution in the present work two theorems are proved which together extend the universal coding theorems to a large class of data generating densities the first gives an asymptotic upper bound for the code redundancy in the order of magnitude achieved with a special predictive type of histogram estimator which sharpens a related bound the second theorem states that this bound cannot be improved upon by any code whatsoever

mixability is bayes risk curvature relative to log loss mixability of a loss characterizes fast rates in the online learning setting of prediction with expert advice the determination of the mixability constant for binary losses is straight forward but opaque in the binary case we make this transparent and simpler by characterising mixability in terms of the second derivative of the bayes risk of proper losses we then extend this result to multiclass proper losses where there are few existing results we show that mixability is governed by the maximum eigenvalue of the hessian of the bayes risk relative to the hessian of the bayes risk for log loss we conclude by comparing our result to other work that bounds prediction performance in terms of the geometry of the bayes risk although all calculations are for proper losses we also show how to carry the results across to improper losses

approximation dans les espaces metriques et theorie de l estimation we investigate the relations between the speed of estimation and the metric structure of the parameter space especially in the case when its metric dimension is infinite given some distance d on generally hellinger distance in the case of n i i d variables we consider the minimax risk for n observations r n q mathop inf sup limits t n theta in theta text ie theta d q theta t n t n being any estimate of we shall look for functions r such that for positive constants c 1 q and c 2 q c 1 r q n r n q c2 r q n r n is the speed of estimation and we shall show under fairly general conditions including i i d variables and regular cases of markov chains and stationnary gaussian processes that r n is determined up to multiplicative constants by the metric structure of we shall also give a construction for some sort of universal estimates the risk of which is bounded by c 2 r q n in all cases where the preceding theory applies

the importance of convexity in learning with squared loss we show that if the closure of a function class f under the metric induced by some probability distribution is not convex then the sample complexity for agnostically learning f with squared loss using only hypotheses in f is spl omega ln 1 spl delta spl epsiv sup 2 where 1 spl delta is the probability of success and spl epsiv is the required accuracy in comparison if the class f is convex and has finite pseudodimension then the sample complexity is o 1 spl epsiv ln 1 spl epsiv ln 1 b if a nonconvex class f has finite pseudodimension then the sample complexity for agnostically learning the closure of the convex hull of f is o 1 spl epsiv 1 spl epsiv ln 1 spl epsiv ln 1 spl delta hence for agnostic learning learning the convex hull provides better approximation capabilities with little sample complexity penalty

estimation of integral functionals of a density let be a smooth function of k 2 variables we shall investigate in this paper the rates of convergence of estimators of t f f f x f x f k x x dx when f belongs to some class of densities of smoothness s we prove that when 2k 1 4 one can define an estimator t n of t f based on n i i d observations of density f on the real line which converges at the semiparametric rate 1 n on the other hand when s 2k 1 4 t f cannot be estimated at a rate faster than n with 4 s k 4s 1 we shall also provide some extensions to the multidimensional case those results extend previous works of levit of bickel and ritov and of donoho and nussbaum on estimation of quadratic functionals

universal coding information prediction and estimation a connection between universal codes and the problems of prediction and statistical estimation is established a known lower bound for the mean length of universal codes is sharpened and generalized and optimum universal codes constructed the bound is defined to give the information in strings relative to the considered class of processes the earlier derived minimum description length criterion for estimation of parameters including their number is given a fundamental information theoretic justification by showing that its estimators achieve the information in the strings it is also shown that one cannot do prediction in gaussian autoregressive moving average arma processes below a bound which is determined by the information in the data

misspecification in infinite dimensional bayesian statistics we consider the asymptotic behavior of posterior distributions if the model is misspecified given a prior distribution and a random sample from a distribution p 0 which may not be in the support of the prior we show that the posterior concentrates its mass near the points in the support of the prior that minimize the kullback leibler divergence with respect to p 0 an entropy condition and a prior mass condition determine the rate of convergence the method is applied to several examples with special interest for infinite dimensional models these include gaussian mixtures nonparametric regression and parametric models

minimax nonparametric classification i rates of convergence this paper studies minimax aspects of nonparametric classification we first study minimax estimation of the conditional probability of a class label given the feature variable this function say f is assumed to be in a general nonparametric class we show the minimax rate of convergence under square l sub 2 loss is determined by the massiveness of the class as measured by metric entropy the second part of the paper studies minimax classification the loss of interest is the difference between the probability of misclassification of a classifier and that of the bayes decision as is well known an upper bound on risk for estimating f gives an upper bound on the risk for classification but the rate is known to be suboptimal for the class of monotone functions this suggests that one does not have to estimate f well in order to classify well however we show that the two problems are in fact of the same difficulty in terms of rates of convergence under a sufficient condition which is satisfied by many function classes including besov sobolev lipschitz and bounded variation this is somewhat surprising in view of a result of devroye gorfi and lugosi see a probabilistic theory of pattern recognition new york springer verlag 1996

convexity classification and risk bounds many of the classification algorithms developed in the machine learning literature including the support vector machine and boosting can be viewed as minimum contrast methods that minimize a convex surrogate of the 0 1 loss function the convexity makes these algorithms computationally efficient the use of a surrogate however has statistical consequences that must be balanced against the computational virtues of convexity to study these issues we provide a general quantitative relationship between the risk as assessed using the 0 1 loss and the risk as assessed using any nonnegative surrogate loss function we show that this relationship gives nontrivial upper bounds on excess risk under the weakest possible condition on the loss function that it satisfies a pointwise form of fisher consistency for classification the relationship is based on a simple variational transformation of the loss function that is easy to compute in many applications we also present a refined version of this result in the

fast rates in statistical and online learning the speed with which a learning algorithm converges as it is presented with more data is a central problem in machine learning a fast rate of convergence means less data is needed for the same level of performance the pursuit of fast rates in online and statistical learning has led to the discovery of many conditions in learning theory under which fast learning is possible we show that most of these conditions are special cases of a single unifying condition that comes in two forms the central condition for proper learning algorithms that always output a hypothesis in the given model and stochastic mixability for online algorithms that may make predictions outside of the model we show that under surprisingly weak assumptions both conditions are in a certain sense equivalent the central condition has a re interpretation in terms of convexity of a set of pseudoprobabilities linking it to density estimation under misspecification for bounded losses we show how the central condition enables a direct proof of fast rates and we prove its equivalence to the bernstein condition itself a generalization of the tsybakov margin condition both of which have played a central role in obtaining fast rates in statistical learning yet while the bernstein condition is two sided the central condition is one sided making it more suitable to deal with unbounded losses in its stochastic mixability form our condition generalizes both a stochastic exp concavity condition identified by juditsky rigollet and tsybakov and vovk s notion of mixability our unifying conditions thus provide a substantial step towards a characterization of fast rates in statistical learning similar to how classical mixability characterizes constant regret in the sequential prediction with expert advice setting

on identification in fuzzy systems and its applications in control problems abstract the problem of identification in fuzzy systems described by the use of fuzzy equation is considered the identification method and its performance index is also presented the formal procedure of the identification algorithm is illustrated by means of a numerical example the possibility of using the proposed algorithm for the solution of a control problem is given as well

structure identification of fuzzy model abstract the problems of structure identification of a fuzzy model are formulated a criterion for the verification of a structure is discussed using the criterion an algorithm for identifying a structure is suggested further a successive identification algorithm of the parameters is suggested the proposed methods are applied to an example

localization of vc classes beyond local rademacher complexities abstract in statistical learning the excess risk of empirical risk minimization erm is controlled by comp n f n where n is a size of a learning sample comp n f is a complexity term associated with a given class f and 1 2 1 interpolates between slow and fast learning rates in this paper we introduce an alternative localization approach for binary classification that leads to a novel complexity measure fixed points of the local empirical entropy we show that this complexity measure gives a tight control over comp n f in the upper bounds under bounded noise our results are accompanied by a minimax lower bound that involves the same quantity in particular we practically answer the question of optimality of erm under bounded noise for general vc classes

risk bounds for model selection via penalization performance bounds for criteria for model selection are devel oped using recent theory for sieves the model selection criteria are based on an empirical loss or contrast function with an added penalty term moti vated by empirical process theory and roughly proportional to the number of parameters needed to describe the model divided by the number of ob servations most of our examples involve density or regression estimation settings and we focus on the problem of estimating the unknown density or regression function we show that the quadratic risk of the minimum penal ized empirical contrast estimatoris bounded by an index of the accuracy of the sieve this accuracy index quanties the trade off among the candidate models between the approximation error and parameter dimension relative to sample size if we choose a list of models which exhibit good approximation prop erties with respect to different classes of smoothness the estimator can be simultaneously minimax rate optimal in each of those classes this is what is usually called adaptation the type of classes of smoothness in which one gets adaptation depends heavily on the list of models if too many models are involved in order to get accurate approximation of many wide classes of functions simultaneously it may happen that the estimator is only approx

rates of convergence of minimum distance estimators and kolmogorov s entropy on construit des estimateurs robustes et on montre que le taux de convergence resultant depend naturellement d une fonction entropie

probability inequalities for likelihood ratios and convergence rates of sieve mles let y 1 y n be independent identically distributed with density p 0 and let f be a space of densities we show that the supremum of the likelihood ratios i 1 n p y i p 0 y i where the supremum is over p f with p l 2 p 0 1 2 2 e is exponentially small with probability exponentially close to 1 the exponent is proportional to ne 2 the only condition required for this to hold is that e exceeds a value determined by the bracketing hellinger entropy of f a similar inequality also holds if we replace f by f n and p 0 by q n where q n is an approximation to p 0 in a suitable sense these results are applied to establish rates of convergence of sieve mles furthermore weak conditions are given under which the optimal rate e n defined by h e n f ne n 2 where h f is the hellinger entropy of f is nearly achievable by sieve estimators

universal donsker classes and metric entropy when mathfrak f is a universal donsker class then for independent indetically distributed i i d observation mathbf x 1 ldots mathbf x n with an unknown law p for any mathfrak f i in mathfrak f i 1 ldots m quad n 1 2 left mathfrak f 1 left mathbf x 1 right ldots mathfrak f i left mathbf x n right right 1 leq i leq m is asymptotically normal with mean vector n 1 2 left int mathfrak f i left mathbf x n right d mathbf p left x right right 1 leq i leq m and covariance matrix int mathfrak f i mathfrak f j d mathbf p int mathfrak f id mathbf p int mathfrak f jd mathbf p uniformly for mathfrak f i in mathfrak f then for certain statistics formed frome the mathfrak f i left mathbf x k right even where mathfrak f i may be chosen depending on the mathbf x k there will be asymptotic distribution as n rightarrow infty for example for mathbf x 2 statistics where f i are indicators of disjoint intervals depending suitably on mathbf x 1 ldots mathbf x n whose union is the real line mathbf x 2 quadratic forms have limiting distributions roy 1956 and watson 1958 which may however not be mathbf x 2 distributions and may depend on p chernoff and lehmann 1954 universal donsker classes of sets are up to mild measurability conditions just classes satisfying the vapnik cervonenkis comdinatorial conditions defined later in this section donsker the vapnik cervonenkis combinatorial conditions defined later in this section durst and dudley 1981 and dudley 1984 chapter 11 the use of such classes allows a variety of extensions of the roy watson results to general multidimensional sample spaces pollard 1979 and moore and subblebine 1981 vapnik and cervonenkis 1974 indicated application of their families of sets to classification pattern recognition problems more recently the classes have been applied to tree structured classifiacation breiman friedman olshen and stone 1984 chapter 12

game theory maximum entropy minimum discrepancy and robust bayesian decision theory we describe and develop a close relationship between two problems that have customarily been regarded as distinct that of maximizing entropy and that of minimizing worst case expected loss using a formulation grounded in the equilibrium theory of zero sum games between decision maker and nature these two problems are shown to be dual to each other the solution to each providing that to the other although topsoe described this connection for the shannon entropy over 20 years ago it does not appear to be widely known even in that important special case we here generalize this theory to apply to arbitrary decision problems and loss functions we indicate how an appropriate generalized definition of entropy can be associated with such a problem and we show that subject to certain regularity conditions the above mentioned duality continues to apply in this extended context this simultaneously provides a possible rationale for maximizing entropy and a tool for finding robust bayes acts we also describe the essential identity between the problem of maximizing entropy and that of minimizing a related discrepancy or divergence between distributions this leads to an extension to arbitrary discrepancies of a well known minimax theorem for the case of kullback leibler divergence the redundancy capacity theorem of information theory for the important case of families of distributions having certain mean values specified we develop simple sufficient conditions and methods for identifying the desired solutions we use this theory to introduce a new concept of generalized exponential family linked to the specific decision problem under consideration and we demonstrate that this shares many of the properties of standard exponential families finally we show that the existence of an equilibrium in our game can be rephrased in terms of a pythagorean property of the related divergence thus generalizing previously announced results for kullback leibler and bregman divergences

the fundamental frequency of non homogeneous rectangular membranes the present study deals with the case in which the density of the membrane is a linear function of the x variable two independent solutions are obtained using the optimized galerkin kantorovich approach and the differential quadrature method

lower bounds for the empirical minimization algorithm in this correspondence we present a simple argument that proves that under mild geometric assumptions on the class f and the set of target functions t the empirical minimization algorithm cannot yield a uniform error rate that is faster than 1 radic k in the function learning setup this result holds for various loss functionals and the target functions from t that cause the slow uniform error rate are clearly exhibited

approximation of density functions by sequences of exponential families 1 introduction consider the estimation of a probability density func tion p x defined on a bounded interval we approximate the logarithm of the density by a basis function expansion consisting of polynomials splines or trigonometric series the expansion yields a regular exponential family within which we estimate the density by the method of maximum likelihood this method of density estimation arises by application of the principle of maxi mum entropy or minimum relative entropy subject to empirical constraints we show that if the logarithm of the density has r square integrable deriva tives f idr log p12 ox then the sequence of density estimators p converges to p in the sense of relative entropy kullback leibler fp log p pn at rate opr 1 m2r m n as m x c and m2 n 0 in the spline and trigonometric cases and m3 n o 0 in the polynomial case where m is the dimension of the family and n is the sample size boundary conditions are assumed for the density in the trigonometric case this convergence rate specializes to op r n 2r 2r 1 by setting m nl 2r 1 when the log density is known to have degree of smoothness at least r analogous convergence results for the relative entropy are shown to hold in general for any class of log density functions and sequence of finite dimensional linear spaces having l2 and l approximation properties the approximation of log densities using polynomials has previously been considered by neyman 1937 to define alternatives for goodness of fit tests by good 1963 as an application of the method of maximum entropy or minimum relative entropy by crain 1974 1976a b 1977 who demonstrates existence and consistency of the maximum likelihood estimator and by mead and

approximation of least squares regression on nested subspaces on presente une methode generale pour approximer le biais et la variance dans une echelle de normes hilbertiennes

information theory and superefficiency the asymptotic risk of efficient estimators with kullback leibler loss in smoothly parametrized statistical models is k 2n where k is the parameter dimension and n is the sample size under fairly general conditions we given a simple information theoretic proof that the set of parameter values where any arbitrary estimator is superefficient is negligible the proof is based on a result of rissanen that codes have asymptotic redundancy not smaller than k 2 log n except in a set of measure 0

information theoretic upper and lower bounds for statistical estimation in this paper we establish upper and lower bounds for some statistical estimation problems through concise information theoretic arguments our upper bound analysis is based on a simple yet general inequality which we call the information exponential inequality we show that this inequality naturally leads to a general randomized estimation method for which performance upper bounds can be obtained the lower bounds applicable for all statistical estimators are obtained by original applications of some well known information theoretic inequalities and approximately match the obtained upper bounds for various important problems moreover our framework can be regarded as a natural generalization of the standard minimax framework in that we allow the performance of the estimator to vary for different possible underlying distributions according to a predefined prior

posterior concentration and fast convergence rates for generalized bayesian learning abstract in this paper we study the learning rate of generalized bayes estimators in a general setting where the hypothesis class can be uncountable and have an irregular shape the loss function can have heavy tails and the optimal hypothesis may not be unique we prove that under the multi scale bernstein s condition the generalized posterior distribution concentrates around the set of optimal hypotheses and the generalized bayes estimator can achieve fast learning rate our results are applied to show that the standard bayesian linear regression is robust to heavy tailed distributions

minimum complexity density estimation the authors introduce an index of resolvability that is proved to bound the rate of convergence of minimum complexity density estimators as well as the information theoretic redundancy of the corresponding total description length the results on the index of resolvability demonstrate the statistical effectiveness of the minimum description length principle as a method of inference the minimum complexity estimator converges to true density nearly as fast as an estimator based on prior knowledge of the true subclass of densities interpretations and basic properties of minimum complexity estimators are discussed some regression and classification problems that can be examined from the minimum description length framework are considered

local rademacher complexities and oracle inequalities in risk minimization let be a class of measurable functions f s 0 1 defined on a probability space s mathcal a p given a sample x1 xn of i i d random variables taking values in s with common distribution p let pn denote the empirical measure based on x1 xn we study an empirical risk minimization problem pnf min f given a solution fn of this problem the goal is to obtain very general upper bounds on its excess risk mathcal e p hat f n p hat f n inf f in mathcal f pf expressed in terms of relevant geometric parameters of the class using concentration inequalities and other empirical processes tools we obtain both distribution dependent and data dependent upper bounds on the excess risk that are of asymptotically correct order in many examples the bounds involve localized sup norms of empirical and rademacher processes indexed by functions from the class we use these bounds to develop model selection techniques in abstract risk minimization problems that can be applied to more specialized frameworks of regression and classification

mutual information metric entropy and cumulative relative entropy risk assume p theta theta epsilon theta is a set of probability distributions with a common dominating measure on a complete separable metric space y a state theta epsilon theta is chosen by nature a statistician obtains n independent observations y 1 dots y n from y distributed according to p theta for each time t between 1 and n based on the observations y 1 dots y t 1 the statistician produces an estimated distribution hat p t for p theta and suffers a loss l p theta hat p t the cumulative risk for the statistician is the average total loss up to time n of special interest in information theory data compression mathematical finance computational learning theory and statistical mechanics is the special case when the loss l p theta hat p t is the relative entropy between the true distribution p theta and the estimated distribution hat p t here the cumulative bayes risk from time 1 to n is the mutual information between the random parameter theta and the observations y 1 dots y n new bounds on this mutual information are given in terms of the laplace transform of the hellinger distance between pairs of distributions indexed by parameters in theta from these bounds on the cumulative minimax risk are given in terms of the metric entropy of theta with respect to the hellinger distance the assumptions required for these bounds are very general and do not depend on the choice of the dominating measure they apply to both finite and infinite dimensional theta they apply in some cases where y is infinite dimensional in some cases where y is not compact in some cases where the distributions are not smooth and in some parametric cases where asymptotic normality of the posterior distribution fails

some exact solutions of the vibration of non homogeneous membranes the purpose of this note is to show that the linear density variation case studied by masad has a closed form exact solution also we shall present another exact solution for the vibration of a continuously non homogeneous annular membrane

an identification algorithm in fuzzy relational systems abstract the paper presents an effective identification method in fuzzy relational systems we propose an algorithm for constructing models on the basis of fuzzy and nonfuzzy data with the aid of fuzzy discretization and clustering techniques the usefulness of the method provided is demonstrated by means of two numerical examples also a possible way of generating a linguistic decision making algorithm is discussed

a general minimax result for relative entropy suppose nature picks a probability measure p sub spl theta on a complete separable metric space x at random from a measurable set p sub spl theta p spl theta spl theta spl isin spl theta then without knowing spl theta a statistician picks a measure q on s finally the statistician suffers a loss d p sub 0 spl par q the relative entropy between p sub spl theta and q we show that the minimax and maximin values of this game are always equal and there is always a minimax strategy in the closure of the set of all bayes strategies this generalizes previous results of gallager 1979 and davisson and leon garcia 1980

convergence rates of posterior distributions we consider the asymptotic behavior of posterior distributions and bayes estimators for infinite dimensional statistical models we give general results on the rate of convergence of the posterior measure these are applied to several examples including priors on finite sieves log spline models dirichlet processes and interval censoring

fast learning rates in statistical inference through aggregation we develop minimax optimal risk bounds for the general learning task consisting in predicting as well as the best function in a reference set mathcal g up to the smallest possible additive term called the convergence rate when the reference set is finite and when n denotes the size of the training data we provide minimax convergence rates of the form c frac log mathcal g n v with tight evaluation of the positive constant c and with exact 0 v 1 the latter value depending on the convexity of the loss function and on the level of noise in the output distribution the risk upper bounds are based on a sequential randomized algorithm which at each step concentrates on functions having both low risk and low variance with respect to the previous step prediction function our analysis puts forward the links between the probabilistic and worst case viewpoints and allows to obtain risk bounds unachievable with the standard statistical learning approach one of the key ideas of this work is to use probabilistic inequalities with respect to appropriate gibbs distributions on the prediction function space instead of using them with respect to the distribution generating the data the risk lower bounds are based on refinements of the assouad lemma taking particularly into account the properties of the loss function our key example to illustrate the upper and lower bounds is to consider the lq regression setting for which an exhaustive analysis of the convergence rates is given while q ranges in 1

geometrizing rates of convergence ii consider estimating a functional t f of an unknown distribution f in mathbf f from data x 1 cdots x n i i d f let omega varepsilon denote the modulus of continuity of the functional t over mathbf f computed with respect to hellinger distance for well behaved loss functions l t we show that inf t n sup mathbf f e fl t n t f is equivalent to l omega n 1 2 to within constants whenever t is linear and mathbf f is convex the same conclusion holds in three nonlinear cases estimating the rate of decay of a density estimating the mode and robust nonparametric regression we study the difficulty of testing between the composite infinite dimensional hypotheses h 0 t f leq t and h 1 t f geq t delta our results hold in the cases studied because the difficulty of the full infinite dimensional composite testing problem is comparable to the difficulty of the hardest simple two point testing subproblem

a lower bound on the error in nonparametric regression type problems let x1 y1 xn yn be a sample denote the conditional density of ylxi xi as f ylxi 0 xi and 0 an element of a metric space 9 d a lower bound is provided for the d error in estimating 0 the order of the bound depends on the local behavior of the kullback information of the conditional density as an application we consider the case where 1 is the space of q smooth functions on 0 i d metrized with the lr distance 1 r oo

information theoretic asymptotics of bayes methods in the absence of knowledge of the true density function bayesian models take the joint density function for a sequence of n random variables to be an average of densities with respect to a prior the authors examine the relative entropy distance d sub n between the true density and the bayesian density and show that the asymptotic distance is d 2 log n c where d is the dimension of the parameter vector therefore the relative entropy rate d sub n n converges to zero at rate log n n the constant c which the authors explicitly identify depends only on the prior density function and the fisher information matrix evaluated at the true parameter value consequences are given for density estimation universal data compression composite hypothesis testing and stock market portfolio selection

composite multiclass losses we consider loss functions for multiclass prediction problems we show when a multiclass loss can be expressed as a proper composite loss which is the composition of a proper loss and a link function we extend existing results for binary losses to multiclass losses we subsume results on classification calibration by relating it to properness we determine the stationarity condition bregman representation order sensitivity and quasi convexity of multiclass proper losses we then characterise the existence and uniqueness of the composite representation for multiclass losses we show how the composite representation is related to other core properties of a loss mixability admissibility and strong convexity of multiclass losses which we characterise in terms of the hessian of the bayes risk we show that the simple integral representation for binary proper losses can not be extended to multiclass losses but offer concrete guidance regarding how to design different loss functions the conclusion drawn from these results is that the proper composite representation is a natural and convenient tool for the design of multiclass loss functions