information theoretic determination of minimax rates of convergence we present some general results determining minimax bounds on statistical risk for density estimation based on certain information theoretic considerations these bounds depend only on metric entropy conditions and are used to identify the minimax rates of convergence

refined error bounds for several learning algorithms this article studies the achievable guarantees on the error rates of certain learning algorithms with particular focus on refining logarithmic factors many of the results are based on a general technique for obtaining bounds on the error rates of sample consistent classifiers with monotonic error regions in the realizable case we prove bounds of this type expressed in terms of either the vc dimension or the sample compression size this general technique also enables us to derive several new bounds on the error rates of general sample consistent learning algorithms as well as refined bounds on the label complexity of the cal active learning algorithm additionally we establish a simple necessary and sufficient condition for the existence of a distribution free bound on the error rates of all sample consistent learning rules converging at a rate inversely proportional to the sample size we also study learning in the presence of classification noise deriving a new excess error rate guarantee for general vc classes under tsybakov s noise condition and establishing a simple and general necessary and sufficient condition for the minimax excess risk under bounded noise to converge at a rate inversely proportional to the sample size

on estimating a density using hellinger distance and some other strange facts on s interesse ici aux possibles vitesses d estimation d une densite a support compact dans m sous des hypotheses de regularite lorsque la perte est mesuree par le carre de la distance de hellinger on regardera aussi le cas connu des normes mathbb l q pour 1 q 2 et le risque est le risque minimax sur la famille on donne une methode generale permettant de traiter les problemes dans le cadre de la theorie de l approximation sous des conditions concernant l entropie metrique et l e capacite des familles a estimer les rapports entre regularite et entropie metrique etant bien connus nous pourrons aussi traiter les cas classiques et d autres qui le sont moins sous des conditions de bornes inferieures les vitesses sont celles observees pour la norme mathbb l q mais elles different dans le cas general on montre aussi que les restrictions sur la compacite du support ou la regularite sont indispensables et que leur absence mene a l impossibilite d obtenir une estimation raisonnable en ce sens que n importe quelle suite d estimateurs sera arbitrairement mauvaise en un point au moins un resultat analogue est vrai sous des conditions de regularite

information theoretic lower bounds on the oracle complexity of stochastic convex optimization relative to the large literature on upper bounds on complexity of convex optimization lesser attention has been paid to the fundamental hardn4516420ess of these problems given the extensive use of convex optimization in machine learning and statistics gaining an understanding of these complexity theoretic issues is important in this paper we study the complexity of stochastic convex optimization in an oracle model of computation we introduce a new notion of discrepancy between functions and use it to reduce problems of stochastic convex optimization to statistical parameter estimation which can be lower bounded using information theoretic methods using this approach we improve upon known results and obtain tight minimax complexity estimates for various function classes

minimax analysis of active learning this work establishes distribution free upper and lower bounds on the minimax label complexity of active learning with general hypothesis classes under various noise models the results reveal a number of surprising facts in particular under the noise model of tsybakov 2004 the minimax label complexity of active learning with a vc class is always asymptotically smaller than that of passive learning and is typically signi cantly smaller than the best previously published upper bounds in the active learning literature in high noise regimes it turns out that all active learning problems of a given vc dimension have roughly the same minimax label complexity which contrasts with well known results for bounded noise in low noise regimes we find that the label complexity is well characterized by a simple combinatorial complexity measure we call the star number interestingly we find that almost all of the complexity measures previously explored in the active learning literature have worst case values exactly equal to the star number we also propose new active learning strategies that nearly achieve these minimax label complexities

asymptotic expansions and boundary values of eigenfunctions on riemannian symmetric spaces soit x g k un espace de riemann symetrique de type non compacte a bord b k m et soit d x l algebre de tous les operateurs differentiels g invariants sur x on obtient des developpements asymptotiques pour une grande classe de fonctions jointives f sur x

rates of convergence for minimum contrast estimators we shall present here a general study of minimum contrast estimators in a nonparametric setting although our results are also valid in the classical parametric case for independent observations these estimators include many of the most popular estimators in various situations such as maximum likelihood estimators least squares and other estimators of the regression function estimators for mixture models or deconvolution the main theorem relates the rate of convergence of those estimators to the entropy structure of the space of parameters optimal rates depending on entropy conditions are already known at least for some of the models involved and they agree with what we get for minimum contrast estimators as long as the entropy counts are not too large but under some circumstances large entropies or changes in the entropy structure due to local perturbations the resulting the rates are only suboptimal counterexamples are constructed which show that the phenomenon is real for non parametric maximum likelihood or regression this proves that under purely metric assumptions our theorem is optimal and that minimum contrast estimators happen to be suboptimal

density estimation by wavelet thresholding density estimation is a commonly used test case for nonparametric estimation methods we explore the asymptotic properties of estimators based on thresholding of empirical wavelet coefficients minimax rates of convergence are studied over a large range of besov function classes b pq and for a range of global l p error measures 1 p p some form of nonlinearity is essential since the minimax linear estimators are suboptimal by polynomial powers of n a second approach using an approximation of a gaussian white noise model in a mallows metric is used to attain exactly optimal rates of convergence for quadratic error p 2

risk bounds for statistical learning we propose a general theorem providing upper bounds for the risk of an empirical risk minimizer erm we essentially focus on the binary classification framework we extend tsybakov s analysis of the risk of an erm under margin type conditions by using concentration inequalities for conveniently weighted empirical processes this allows us to deal with ways of measuring the size of a class of classifiers other than entropy with bracketing as in tsybakov s work in particular we derive new risk bounds for the erm when the classification rules belong to some vc class under margin conditions and discuss the optimality of these bounds in a minimax sense

a new lower bound for multiple hypothesis testing the purpose of this correspondence is to give a new easily tractable and sharp lower bound for the maximal error in multiple hypothesis testing with an application to nonasymptotic lower bounds for the minimax risk of estimators

some inequalities for information divergence and related measures of discrimination inequalities which connect information divergence with other measures of discrimination or distance between probability distributions are used in information theory and its applications to mathematical statistics ergodic theory and other scientific fields we suggest new inequalities of this type often based on underlying identities as a consequence we obtain certain improvements of the well known pinsker inequality our study depends on two measures of discrimination called capacitory discrimination and triangular discrimination the discussion contains references to related research and comparison with other measures of discrimination e g ali silvey csiszar 1996 1966 divergences and in particular the hellinger distance

on density estimation in the view of kolmogorov s ideas in approximation theory the paper deals with upper and lower bounds for the quality of probability density estimation connections are established between these problems and the theory of approximation of functions particularly it is demonstrated how some of kolmogorov s concepts work

minimax nonparametric classification ii model selection for adaptation for pt i see ibid vol 45 no 7 p 2271 84 1999 we study nonparametric estimation of a conditional probability for classification based on a collection of finite dimensional models for the sake of flexibility different types of models linear or nonlinear are allowed as long as each satisfies a dimensionality assumption we show that with a suitable model selection criterion the penalized maximum likelihood estimator has a risk bounded by an index of resolvability expressing a good tradeoff among approximation error estimation error and model complexity the bound does not require any assumption on the target conditional probability and can be used to demonstrate the adaptivity of estimators based on model selection examples are given with both splines and neural nets and problems of high dimensional estimation are considered the resulting adaptive estimator is shown to behave optimally or near optimally over sobolev classes with unknown orders of interaction and smoothness and classes of integrable fourier transform of gradient in terms of rates of convergence the performance is the same as if one knew which of them contains the true conditional probability in advance the corresponding classifier also converges optimally or nearly optimally simultaneously over these classes

information theoretic bounds for compressed sensing in this paper we derive information theoretic performance bounds to sensing and reconstruction of sparse phenomena from noisy projections we consider two settings output noise models where the noise enters after the projection and input noise models where the noise enters before the projection we consider two types of distortion for reconstruction support errors and mean squared errors our goal is to relate the number of measurements m and snr to signal sparsity k distortion level d and signal dimension n we consider support errors in a worst case setting we employ different variations of fano s inequality to derive necessary conditions on the number of measurements and snr required for exact reconstruction to derive sufficient conditions we develop new insights on max likelihood analysis based on a novel superposition property in particular this property implies that small support errors are the dominant error events consequently our ml analysis does not suffer the conservatism of the union bound and leads to a tighter analysis of max likelihood these results provide order wise tight bounds for output noise models we show that asymptotically an snr of n together with k n k measurements is necessary and sufficient for exact support recovery furthermore if a small fraction of support errors can be tolerated a constant snr turns out to be sufficient in the linear sparsity regime in contrast for input noise models we show that support recovery fails if the number of measurements scales as o n n snr implying poor compression performance for such cases motivated by the fact that the worst case setup requires significantly high snr and substantial number of measurements for input and output noise models we consider a bayesian setup to derive necessary conditions we develop novel extensions to fano s inequality to handle continuous domains and arbitrary distortions we then develop a new max likelihood analysis over the set of rate distortion quantization points to characterize tradeoffs between mean squared distortion and the number of measurements using rate distortion theory we show that with constant snr the number of measurements scales linearly with the rate distortion function of the sparse phenomena

hellinger consistency of certain nonparametric maximum likelihood estimators consider a class mathscr p p theta theta in theta of probability measures on a measurable space mathscr x mathscr a dominated by a sigma finite measure mu let f theta dp theta d mu theta in theta and let theta n be a maximum likelihood estimator based on n independent observations from p theta 0 theta 0 in theta we use results from empirical process theory to obtain convergence for the hellinger distance h f hat theta n f theta 0 under certain entropy conditions on the class of densities f theta theta in theta the examples we present are a model with interval censored observations smooth densities monotone densities and convolution models in most examples the convexity of the class of densities is of special importance

exponential series estimator of multivariate densities abstract we present an exponential series estimator ese of multivariate densities which has an appealing information theoretic interpretation for a d dimensional random variable x with density p 0 the ese takes the form p x exp i 1 0 m 1 i d 0 m d i i x where i are some real valued linearly independent functions defined on the support of p 0 we derive the convergence rate of the ese in terms of the kullback leibler information criterion the integrated squared error and some other metrics we also derive its almost sure uniform convergence rate we then establish the asymptotic normality of p we undertake two sets of monte carlo experiments the first experiment examines the ese performance using mixtures of multivariate normal densities the second estimates copula density functions the results demonstrate the efficacy of the ese an empirical application on the joint distributions of stock returns is presented

a semilocal convergence analysis for directional newton methods a semilocal convergence analysis for directional newton methods in n variables is provided in this study using weaker hypotheses than in the elegant related work by y levin and a ben israel and introducing the center lipschitz condition we provide under the same computational cost as in levin and ben israel a semilocal convergence analysis with the following advantages weaker convergence conditions larger convergence domain finer error estimates on the distances involved and an at least as precise information on the location of the zero of the function a numerical example where our results apply to solve an equation but not the ones in levin and ben israel is also provided in this study

a distributed one step estimator distributed statistical inference has recently attracted enormous attention many existing work focuses on the averaging estimator e g zhang and duchi j mach learn res 14 3321 3363 2013 together with many others we propose a one step approach to enhance a simple averaging based distributed estimator by utilizing a single newton raphson updating we derive the corresponding asymptotic properties of the newly proposed estimator we find that the proposed one step estimator enjoys the same asymptotic properties as the idealized centralized estimator in particular the asymptotic normality was established for the proposed estimator while other competitors may not enjoy the same property the proposed one step approach merely requires one additional round of communication in relative to the averaging estimator so the extra communication burden is insignificant the proposed one step approach leads to a lower upper bound of the mean squared error than other alternatives in finite sample cases numerical examples show that the proposed estimator outperforms the simple averaging estimator with a large margin in terms of the sample mean squared error a potential application of the one step approach is that one can use multiple machines to speed up large scale statistical inference with little compromise in the quality of estimators the proposed method becomes more valuable when data can only be available at distributed machines with limited communication bandwidth

a simple lemma on greedy approximation in hilbert space and convergence rates for projection pursuit regression and neural network training a general convergence criterion for certain iterative sequences in hilbert space is presented for an important subclass of these sequences estimates of the rate of convergence are given under very mild assumptions these results establish an 0 1 f4n nonsampling convergence rate for projection pursuit regression and neural network training where n represents the number of ridge functions neurons or coefficients in a greedy basis expansion

from ɛ entropy to kl entropy analysis of minimum information complexity density estimation we consider an extension of e entropy to a kl divergence based complexity measure for randomized density estimation methods based on this extension we develop a general information theoretical inequality that measures the statistical complexity of some deterministic and randomized density estimators consequences of the new inequality will be presented in particular we show that this technique can lead to improvements of some classical results concerning the convergence of minimum description length and bayesian posterior distributions moreover we are able to derive clean finite sample convergence bounds that are not obtainable using previous approaches

estimating divergence functionals and the likelihood ratio by convex risk minimization we develop and analyze m estimation methods for divergence functionals and the likelihood ratios of two probability distributions our method is based on a nonasymptotic variational characterization of f divergences which allows the problem of estimating divergences to be tackled via convex empirical risk optimization the resulting estimators are simple to implement requiring only the solution of standard convex programs we present an analysis of consistency and convergence for these estimators given conditions only on the ratios of densities we show that our estimators can achieve optimal minimax rates for the likelihood ratio and the divergence functionals in certain regimes we derive an efficient optimization algorithm for computing our estimates and illustrate their convergence behavior and practical viability by simulations

high dimensional analysis of semidefinite relaxations for sparse principal components principal component analysis pca is a classical method for dimensionality reduction based on extracting the dominant eigenvectors of the sample covariance matrix however pca is well known to behave poorly in the large p small n setting in which the problem dimension p is comparable to or larger than the sample size n this paper studies pca in this high dimensional regime but under the additional assumption that the maximal eigenvector is sparse say with at most k nonzero components we consider a spiked covariance model in which a base matrix is perturbed by adding a k sparse maximal eigenvector and we analyze two computationally tractable methods for recovering the support set of this maximal eigenvector as follows a a simple diagonal thresholding method which transitions from success to failure as a function of the rescaled sample size dia n p k n k 2 log p k and b a more sophisticated semidefinite programming sdp relaxation which succeeds once the rescaled sample size sdp n p k n klog p k is larger than a critical threshold in addition we prove that no method including the best method which has exponential time complexity can succeed in recovering the support if the order parameter sdp n p k is below a threshold our results thus highlight an interesting trade off between computational and statistical efficiency in high dimensional inference

adaptive regression by mixing adaptation over different procedures is of practical importance different procedures perform well under different conditions in many practical situations it is rather hard to assess which conditions are approximately satisfied so as to identify the best procedure for the data at hand thus automatic adaptation over various scenarios is desirable a practically feasible method named adaptive regression by mixing arm is proposed to convexly combine general candidate regression procedures under mild conditions the resulting estimator is theoretically shown to perform optimally in rates of convergence without knowing which of the original procedures work the best simulations are conducted in several settings including comparing a parametric model with nonparametric alternatives comparing a neural network with a projection pursuit in multidimensional regression and combining bandwidths in kernel regression the results clearly support the theoretical property of arm the arm algorithm assigns we

approximation dans les espaces metriques et theorie de l estimation we investigate the relations between the speed of estimation and the metric structure of the parameter space especially in the case when its metric dimension is infinite given some distance d on generally hellinger distance in the case of n i i d variables we consider the minimax risk for n observations r n q mathop inf sup limits t n theta in theta text ie theta d q theta t n t n being any estimate of we shall look for functions r such that for positive constants c 1 q and c 2 q c 1 r q n r n q c2 r q n r n is the speed of estimation and we shall show under fairly general conditions including i i d variables and regular cases of markov chains and stationnary gaussian processes that r n is determined up to multiplicative constants by the metric structure of we shall also give a construction for some sort of universal estimates the risk of which is bounded by c 2 r q n in all cases where the preceding theory applies

convergence rates for bayesian density estimation of infinite dimensional exponential families we study the rate of convergence of posterior distributions in density estimation problems for log densities in periodic sobolev classes characterized by a smoothness parameter p the posterior expected density provides a nonparametric estimation procedure attaining the optimal minimax rate of convergence under hellinger loss if the posterior distribution achieves the optimal rate over certain uniformity classes a prior on the density class of interest is induced by a prior on the coefficients of the trigonometric series expansion of the log density we show that when p is known the posterior distribution of a gaussian prior achieves the optimal rate provided the prior variances die off sufficiently rapidly for a mixture of normal distributions the mixing weights on the dimension of the exponential family are assumed to be bounded below by an exponentially decreasing sequence to avoid the use of infinite bases we develop priors that cut off the series at a sample size dependent truncation point when the degree of smoothness is unknown a finite mixture of normal priors indexed by the smoothness parameter which is also assigned a prior produces the best rate a rate adaptive estimator is derived

universal coding information prediction and estimation a connection between universal codes and the problems of prediction and statistical estimation is established a known lower bound for the mean length of universal codes is sharpened and generalized and optimum universal codes constructed the bound is defined to give the information in strings relative to the considered class of processes the earlier derived minimum description length criterion for estimation of parameters including their number is given a fundamental information theoretic justification by showing that its estimators achieve the information in the strings it is also shown that one cannot do prediction in gaussian autoregressive moving average arma processes below a bound which is determined by the information in the data

estimation of integral functionals of a density let be a smooth function of k 2 variables we shall investigate in this paper the rates of convergence of estimators of t f f f x f x f k x x dx when f belongs to some class of densities of smoothness s we prove that when 2k 1 4 one can define an estimator t n of t f based on n i i d observations of density f on the real line which converges at the semiparametric rate 1 n on the other hand when s 2k 1 4 t f cannot be estimated at a rate faster than n with 4 s k 4s 1 we shall also provide some extensions to the multidimensional case those results extend previous works of levit of bickel and ritov and of donoho and nussbaum on estimation of quadratic functionals

density estimation by stochastic complexity the results by p hall and e j hannan 1988 on optimization of histogram density estimators with equal bin widths by minimization of the stochastic complexity are extended and sharpened in two separate ways as the first contribution two generalized histogram estimators are constructed the first has unequal bin widths which together with the number of the bins are determined by minimization of the stochastic complexity using dynamic programming the other estimator consists of a mixture of equal bin width estimators each of which is defined by the associated stochastic complexity as the main contribution in the present work two theorems are proved which together extend the universal coding theorems to a large class of data generating densities the first gives an asymptotic upper bound for the code redundancy in the order of magnitude achieved with a special predictive type of histogram estimator which sharpens a related bound the second theorem states that this bound cannot be improved upon by any code whatsoever

on the consistency of bayesian function approximation using step functions we consider the problem of estimating a step function with an unknown number of jumps under noisy observations on a grid under mild assumptions the bayesian approach is shown to produce a consistent estimate even when the underlying true function is not piecewise constant a simple prior is constructed to illustrate our assumptions

minimax nonparametric classification i rates of convergence this paper studies minimax aspects of nonparametric classification we first study minimax estimation of the conditional probability of a class label given the feature variable this function say f is assumed to be in a general nonparametric class we show the minimax rate of convergence under square l sub 2 loss is determined by the massiveness of the class as measured by metric entropy the second part of the paper studies minimax classification the loss of interest is the difference between the probability of misclassification of a classifier and that of the bayes decision as is well known an upper bound on risk for estimating f gives an upper bound on the risk for classification but the rate is known to be suboptimal for the class of monotone functions this suggests that one does not have to estimate f well in order to classify well however we show that the two problems are in fact of the same difficulty in terms of rates of convergence under a sufficient condition which is satisfied by many function classes including besov sobolev lipschitz and bounded variation this is somewhat surprising in view of a result of devroye gorfi and lugosi see a probabilistic theory of pattern recognition new york springer verlag 1996

sparse pca optimal rates and adaptive estimation principal component analysis pca is one of the most commonly used statistical procedures with a wide range of applications this paper considers both minimax and adaptive estimation of the principal subspace in the high dimensional setting under mild technical conditions we first establish the optimal rates of convergence for estimating the principal subspace which are sharp with respect to all the parameters thus providing a complete characterization of the difficulty of the estimation problem in term of the convergence rate the lower bound is obtained by calculating the local metric entropy and an application of fano s lemma the rate optimal estimator is constructed using aggregation which however might not be computationally feasible we then introduce an adaptive procedure for estimating the principal subspace which is fully data driven and can be computed efficiently it is shown that the estimator attains the optimal rates of convergence simultaneously over a large collection of the parameter spaces a key idea in our construction is a reduction scheme which reduces the sparse pca problem to a high dimensional multivariate regression problem this method is potentially also useful for other related problems

lower bounds for the minimax risk using f divergences and applications lower bounds involving f divergences between the underlying probability measures are proved for the minimax risk in estimation problems our proofs just use simple convexity facts special cases and straightforward corollaries of our bounds include well known inequalities for establishing minimax lower bounds such as fano s inequality pinsker s inequality and inequalities based on global entropy conditions two applications are provided a new minimax lower bound for the reconstruction of convex bodies from noisy support function measurements and a different proof of a recent minimax lower bound for the estimation of a covariance matrix

information theoretic limits of selecting binary graphical models in high dimensions the problem of graphical model selection is to estimate the graph structure of a markov random field given samples from it we analyze the information theoretic limitations of the problem of graph selection for binary markov random fields under high dimensional scaling in which the graph size and the number of edges k and or the maximal node degree d are allowed to increase to infinity as a function of the sample size n for pair wise binary markov random fields we derive both necessary and sufficient conditions for correct graph selection over the class gp k of graphs on vertices with at most k edges and over the class gp d of graphs on p vertices with maximum degree at most d for the class gp k we establish the existence of constants c and c such that if n c k2 log p similarly for the class gp d we exhibit constants c and c such that for n c d3 log p

fast rates in statistical and online learning the speed with which a learning algorithm converges as it is presented with more data is a central problem in machine learning a fast rate of convergence means less data is needed for the same level of performance the pursuit of fast rates in online and statistical learning has led to the discovery of many conditions in learning theory under which fast learning is possible we show that most of these conditions are special cases of a single unifying condition that comes in two forms the central condition for proper learning algorithms that always output a hypothesis in the given model and stochastic mixability for online algorithms that may make predictions outside of the model we show that under surprisingly weak assumptions both conditions are in a certain sense equivalent the central condition has a re interpretation in terms of convexity of a set of pseudoprobabilities linking it to density estimation under misspecification for bounded losses we show how the central condition enables a direct proof of fast rates and we prove its equivalence to the bernstein condition itself a generalization of the tsybakov margin condition both of which have played a central role in obtaining fast rates in statistical learning yet while the bernstein condition is two sided the central condition is one sided making it more suitable to deal with unbounded losses in its stochastic mixability form our condition generalizes both a stochastic exp concavity condition identified by juditsky rigollet and tsybakov and vovk s notion of mixability our unifying conditions thus provide a substantial step towards a characterization of fast rates in statistical learning similar to how classical mixability characterizes constant regret in the sequential prediction with expert advice setting

asymptotic theory of information theoretic experimental design we discuss an idea for collecting data in a relatively efficient manner our point of view is bayesian and information theoretic on any given trial we want to adaptively choose the input in such a way that the mutual information between the unknown state of the system and the stochastic output is maximal given any prior information including data collected on any previous trials we prove a theorem that quantifies the effectiveness of this strategy and give a few illustrative examples comparing the performance of this adaptive technique to that of the more usual nonadaptive experimental design in particular we calculate the asymptotic efficiency of the information maximization strategy and demonstrate that this method is in a well defined sense never less efficient and is generically more efficient than the nonadaptive strategy for example we are able to explicitly calculate the asymptotic relative efficiency of the staircase method widely employed in psychophysics research and to demonstrate the dependence of this efficiency on the form of the psychometric function underlying the output responses

fast learning rate of multiple kernel learning trade off between sparsity and smoothness we investigate the learning rate of multiple kernel leaning mkl with l1 and elastic net regularizations the elastic net regularization is a composition of an l1 regularizer for inducing the sparsity and an l2 regularizer for controlling the smoothness we focus on a sparse setting where the total number of kernels is large but the number of non zero components of the ground truth is relatively small and show sharper convergence rates than the learning rates ever shown for both l1 and elastic net regularizations our analysis shows there appears a trade off between the sparsity and the smoothness when it comes to selecting which of l1 and elastic net regularizations to use if the ground truth is smooth the elastic net regularization is preferred otherwise the l1 regularization is preferred

estimation of nonparametric conditional moment models with possibly nonsmooth generalized residuals this paper studies nonparametric estimation of conditional moment restrictions in which the generalized residual functions can be nonsmooth in the unknown functions of endogenous variables this is a nonparametric nonlinear instrumental variables iv problem we propose a class of penalized sieve minimum distance psmd estimators which are minimizers of a penalized empirical minimum distance criterion over a collection of sieve spaces that are dense in the infinite dimensional function parameter space some of the psmd procedures use slowly growing finite dimensional sieves with flexible penalties or without any penalty others use large dimensional sieves with lower semicompact and or convex penalties we establish their consistency and the convergence rates in banach space norms such as a sup norm or a root mean squared norm allowing for possibly noncompact infinite dimensional parameter spaces for both mildly and severely ill posed nonlinear inverse problems our convergence rates in hilbert space norms such as a root mean squared norm achieve the known minimax optimal rate for the nonparametric mean iv regression we illustrate the theory with a nonparametric additive quantile iv regression we present a simulation study and an empirical application of estimating nonparametric quantile iv engel curves

bayesian and conditional frequentist testing of a parametric model versus nonparametric alternatives testing the fit of data to a parametric model can be done by embedding the parametric model in a nonparametric alternative and computing the bayes factor of the parametric model to the nonparametric alternative doing so by specifying the nonparametric alternative via a polya tree process is particularly attractive from both theoretical and methodological perspectives among the benefits is a degree of computational simplicity that even allows for robustness analyses to be implemented default nonsubjective versions of this analysis are developed herein in the sense that recommended choices are provided for the many features of the polya tree process that need to be specified considerable discussion of these features is also provided to assist those who might be interested in subjective choices a variety of examples involving location scale models are studied finally it is shown that the resulting procedure can be viewed as a conditional frequentist test resulting in data dependent reported err

rates of convergence of minimum distance estimators and kolmogorov s entropy on construit des estimateurs robustes et on montre que le taux de convergence resultant depend naturellement d une fonction entropie

beta beta bounds finite blocklength analog of the golden formula it is well known that the mutual information between two random variables can be expressed as the difference of two relative entropies that depend on an auxiliary distribution a relation sometimes referred to as the golden formula this paper is concerned with a finite blocklength extension of this relation this extension consists of two elements 1 a finite blocklength channel coding converse bound by polyanskiy and verdu which involves the ratio of two neyman pearson beta functions beta beta converse bound and 2 a novel beta beta channel coding achievability bound expressed again as the ratio of two neyman pearson beta functions to demonstrate the usefulness of this finite blocklength extension of the golden formula the beta beta achievability and converse bounds are used to obtain a finite blocklength extension of verdu s wideband slope approximation the proof parallels the derivation of the latter with the beta beta bounds used in place of the golden formula the beta beta achievability bound is also shown to be useful in cases where the capacity achieving output distribution is not a product distribution due to e g a cost constraint or structural constraints on the codebook such as orthogonality or constant composition as an example the bound is used to characterize the channel dispersion of the additive exponential noise channel and to obtain a finite blocklength achievability bound the tightest to date for multiple input multiple output rayleigh fading channels with perfect channel state information at the receiver

risk bounds for model selection via penalization performance bounds for criteria for model selection are devel oped using recent theory for sieves the model selection criteria are based on an empirical loss or contrast function with an added penalty term moti vated by empirical process theory and roughly proportional to the number of parameters needed to describe the model divided by the number of ob servations most of our examples involve density or regression estimation settings and we focus on the problem of estimating the unknown density or regression function we show that the quadratic risk of the minimum penal ized empirical contrast estimatoris bounded by an index of the accuracy of the sieve this accuracy index quanties the trade off among the candidate models between the approximation error and parameter dimension relative to sample size if we choose a list of models which exhibit good approximation prop erties with respect to different classes of smoothness the estimator can be simultaneously minimax rate optimal in each of those classes this is what is usually called adaptation the type of classes of smoothness in which one gets adaptation depends heavily on the list of models if too many models are involved in order to get accurate approximation of many wide classes of functions simultaneously it may happen that the estimator is only approx

a complement to le cam s theorem this paper examines asymptotic equivalence in the sense of le cam between density estimation experiments and the accompanying poisson experiments the significance of asymptotic equivalence is that all asymptotically optimal statistical procedures can be carried over from one experiment to the other the equivalence given here is established under a weak assumption on the parameter space f in particular a sharp besov smoothness condition is given on f which is sufficient for poissonization namely if f is in a besov ball b p g m with ap 1 2 examples show poissonization is not possible whenever ap 0

mixing strategies for density estimation general results on adaptive density estimation are obtained with respect to any countable collection of estimation strategies under kullback leibler and squared l 2 losses it is shown that without knowing which strategy works best for the underlying density a single strategy can be constructed by mixing the proposed ones to be adaptive in terms of statistical risks a consequence is that under some mild conditions an asymptotically minimax rate adaptive estimator exists for a given countable collection of density classes that is a single estimator can be constructed to be simultaneously minimax rate optimal for all the function classes being considered a demonstration is given for high dimensional density estimation on 0 1 d where the constructed estimator adapts to smoothness and interaction order over some piecewise besov classes and is consistent for all the densities with finite entropy

regularized policy iteration with nonparametric function spaces we study two regularization based approximate policy iteration algorithms namely reg lspi and reg brm to solve reinforcement learning and planning problems in discounted markov decision processes with large state and finite action spaces the core of these algorithms are the regularized extensions of the least squares temporal difference lstd learning and bellman residual minimization brm which are used in the algorithms policy evaluation steps regularization provides a convenient way to control the complexity of the function space to which the estimated value function belongs and as a result enables us to work with rich nonparametric function spaces we derive efficient implementations of our methods when the function space is a reproducing kernel hilbert space we analyze the statistical properties of reg lspi and provide an upper bound on the policy evaluation error and the performance loss of the policy returned by this method our bound shows the dependence of the loss on the number of samples the capacity of the function space and some intrinsic properties of the underlying markov decision process the dependence of the policy evaluation bound on the number of samples is minimax optimal this is the first work that provides such a strong guarantee for a nonparametric approximate policy iteration algorithm

probability inequalities for likelihood ratios and convergence rates of sieve mles let y 1 y n be independent identically distributed with density p 0 and let f be a space of densities we show that the supremum of the likelihood ratios i 1 n p y i p 0 y i where the supremum is over p f with p l 2 p 0 1 2 2 e is exponentially small with probability exponentially close to 1 the exponent is proportional to ne 2 the only condition required for this to hold is that e exceeds a value determined by the bracketing hellinger entropy of f a similar inequality also holds if we replace f by f n and p 0 by q n where q n is an approximation to p 0 in a suitable sense these results are applied to establish rates of convergence of sieve mles furthermore weak conditions are given under which the optimal rate e n defined by h e n f ne n 2 where h f is the hellinger entropy of f is nearly achievable by sieve estimators

universal donsker classes and metric entropy when mathfrak f is a universal donsker class then for independent indetically distributed i i d observation mathbf x 1 ldots mathbf x n with an unknown law p for any mathfrak f i in mathfrak f i 1 ldots m quad n 1 2 left mathfrak f 1 left mathbf x 1 right ldots mathfrak f i left mathbf x n right right 1 leq i leq m is asymptotically normal with mean vector n 1 2 left int mathfrak f i left mathbf x n right d mathbf p left x right right 1 leq i leq m and covariance matrix int mathfrak f i mathfrak f j d mathbf p int mathfrak f id mathbf p int mathfrak f jd mathbf p uniformly for mathfrak f i in mathfrak f then for certain statistics formed frome the mathfrak f i left mathbf x k right even where mathfrak f i may be chosen depending on the mathbf x k there will be asymptotic distribution as n rightarrow infty for example for mathbf x 2 statistics where f i are indicators of disjoint intervals depending suitably on mathbf x 1 ldots mathbf x n whose union is the real line mathbf x 2 quadratic forms have limiting distributions roy 1956 and watson 1958 which may however not be mathbf x 2 distributions and may depend on p chernoff and lehmann 1954 universal donsker classes of sets are up to mild measurability conditions just classes satisfying the vapnik cervonenkis comdinatorial conditions defined later in this section donsker the vapnik cervonenkis combinatorial conditions defined later in this section durst and dudley 1981 and dudley 1984 chapter 11 the use of such classes allows a variety of extensions of the roy watson results to general multidimensional sample spaces pollard 1979 and moore and subblebine 1981 vapnik and cervonenkis 1974 indicated application of their families of sets to classification pattern recognition problems more recently the classes have been applied to tree structured classifiacation breiman friedman olshen and stone 1984 chapter 12

restricted strong convexity and weighted matrix completion optimal bounds with noise we consider the matrix completion problem under a form of row column weighted entrywise sampling including the case of uniformentrywise sampling as a special case we analyze the associated random observation operator and prove that with high probability it satisfies a form of restricted strong convexity with respect to weighted frobenius norm using this property we obtain as corollaries a number of error bounds on matrix completion in the weighted frobenius norm under noisy sampling and for both exact and near low rank matrices our results are based on measures of the spikiness and low rankness of matrices that are less restrictive than the incoherence conditions imposed in previous work our technique involves an m estimator that includes controls on both the rank and spikiness of the solution and we establish non asymptotic error bounds in weighted frobenius norm for recovering matrices lying with lq balls of bounded spikiness using information theoretic methods we show that no algorithm can achieve better estimates up to a logarithmic factor over these same sets showing that our conditions on matrices and associated rates are essentially optimal

minimax bounds for sparse pca with noisy high dimensional data we study the problem of estimating the leading eigenvectors of a high dimensional population covariance matrix based on independent gaussian observations we establish a lower bound on the minimax risk of estimators under the l2 loss in the joint limit as dimension and sample size increase to infinity under various models of sparsity for the population eigenvectors the lower bound on the risk points to the existence of different regimes of sparsity of the eigenvectors we also propose a new method for estimating the eigenvectors by a two stage coordinate selection scheme

catching up faster by switching sooner a predictive approach to adaptive estimation with an application to the aic bic dilemma summary prediction and estimation based on bayesian model selection and model averaging and derived methods such as the bayesian information criterion bic do not always converge at the fastest possible rate we identify the catch up phenomenon as a novel explanation for the slow convergence of bayesian methods which inspires a modification of the bayesian predictive distribution called the switch distribution when used as an adaptive estimator the switch distribution does achieve optimal cumulative risk convergence rates in non parametric density estimation and gaussian regression problems we show that the minimax cumulative risk is obtained under very weak conditions and without knowledge of the underlying degree of smoothness unlike other adaptive model selection procedures such as the akaike information criterion aic and leave one out cross validation bic and bayes factor model selection are typically statistically consistent we show that this property is retained by the switch distribution which thus solves the aic bic dilemma for cumulative risk the switch distribution has an efficient implementation we compare its performance with aic bic and bayesian model selection and averaging on a regression problem with simulated data

approximation of density functions by sequences of exponential families 1 introduction consider the estimation of a probability density func tion p x defined on a bounded interval we approximate the logarithm of the density by a basis function expansion consisting of polynomials splines or trigonometric series the expansion yields a regular exponential family within which we estimate the density by the method of maximum likelihood this method of density estimation arises by application of the principle of maxi mum entropy or minimum relative entropy subject to empirical constraints we show that if the logarithm of the density has r square integrable deriva tives f idr log p12 ox then the sequence of density estimators p converges to p in the sense of relative entropy kullback leibler fp log p pn at rate opr 1 m2r m n as m x c and m2 n 0 in the spline and trigonometric cases and m3 n o 0 in the polynomial case where m is the dimension of the family and n is the sample size boundary conditions are assumed for the density in the trigonometric case this convergence rate specializes to op r n 2r 2r 1 by setting m nl 2r 1 when the log density is known to have degree of smoothness at least r analogous convergence results for the relative entropy are shown to hold in general for any class of log density functions and sequence of finite dimensional linear spaces having l2 and l approximation properties the approximation of log densities using polynomials has previously been considered by neyman 1937 to define alternatives for goodness of fit tests by good 1963 as an application of the method of maximum entropy or minimum relative entropy by crain 1974 1976a b 1977 who demonstrates existence and consistency of the maximum likelihood estimator and by mead and

covering numbers for bounded variation functions abstract in this paper we provide upper and lower estimates for the minimal number of functions needed to represent a bounded variation function with an accuracy of epsilon with respect to l 1 distance

information theory and superefficiency the asymptotic risk of efficient estimators with kullback leibler loss in smoothly parametrized statistical models is k 2n where k is the parameter dimension and n is the sample size under fairly general conditions we given a simple information theoretic proof that the set of parameter values where any arbitrary estimator is superefficient is negligible the proof is based on a result of rissanen that codes have asymptotic redundancy not smaller than k 2 log n except in a set of measure 0

approximation of least squares regression on nested subspaces on presente une methode generale pour approximer le biais et la variance dans une echelle de normes hilbertiennes

information theoretic upper and lower bounds for statistical estimation in this paper we establish upper and lower bounds for some statistical estimation problems through concise information theoretic arguments our upper bound analysis is based on a simple yet general inequality which we call the information exponential inequality we show that this inequality naturally leads to a general randomized estimation method for which performance upper bounds can be obtained the lower bounds applicable for all statistical estimators are obtained by original applications of some well known information theoretic inequalities and approximately match the obtained upper bounds for various important problems moreover our framework can be regarded as a natural generalization of the standard minimax framework in that we allow the performance of the estimator to vary for different possible underlying distributions according to a predefined prior

optimal rates for zero order convex optimization the power of two function evaluations we consider derivative free algorithms for stochastic and nonstochastic convex optimization problems that use only function values rather than gradients focusing on nonasymptotic bounds on convergence rates we show that if pairs of function values are available algorithms for d dimensional optimization that use gradient estimates based on random perturbations suffer a factor of at most sqrt d in convergence rate over traditional stochastic gradient methods we establish such results for both smooth and nonsmooth cases sharpening previous analyses that suggested a worse dimension dependence and extend our results to the case of multiple m ge 2 evaluations we complement our algorithmic development with information theoretic lower bounds on the minimax convergence rate of such problems establishing the sharpness of our achievable results up to constant sometimes logarithmic factors

minimum complexity density estimation the authors introduce an index of resolvability that is proved to bound the rate of convergence of minimum complexity density estimators as well as the information theoretic redundancy of the corresponding total description length the results on the index of resolvability demonstrate the statistical effectiveness of the minimum description length principle as a method of inference the minimum complexity estimator converges to true density nearly as fast as an estimator based on prior knowledge of the true subclass of densities interpretations and basic properties of minimum complexity estimators are discussed some regression and classification problems that can be examined from the minimum description length framework are considered

mutual information metric entropy and cumulative relative entropy risk assume p theta theta epsilon theta is a set of probability distributions with a common dominating measure on a complete separable metric space y a state theta epsilon theta is chosen by nature a statistician obtains n independent observations y 1 dots y n from y distributed according to p theta for each time t between 1 and n based on the observations y 1 dots y t 1 the statistician produces an estimated distribution hat p t for p theta and suffers a loss l p theta hat p t the cumulative risk for the statistician is the average total loss up to time n of special interest in information theory data compression mathematical finance computational learning theory and statistical mechanics is the special case when the loss l p theta hat p t is the relative entropy between the true distribution p theta and the estimated distribution hat p t here the cumulative bayes risk from time 1 to n is the mutual information between the random parameter theta and the observations y 1 dots y n new bounds on this mutual information are given in terms of the laplace transform of the hellinger distance between pairs of distributions indexed by parameters in theta from these bounds on the cumulative minimax risk are given in terms of the metric entropy of theta with respect to the hellinger distance the assumptions required for these bounds are very general and do not depend on the choice of the dominating measure they apply to both finite and infinite dimensional theta they apply in some cases where y is infinite dimensional in some cases where y is not compact in some cases where the distributions are not smooth and in some parametric cases where asymptotic normality of the posterior distribution fails

sparsity and error analysis of empirical feature based regularization schemes we consider a learning algorithm generated by a regularization scheme with a concave regularizer for the purpose of achieving sparsity and good learning rates in a least squares regression setting the regularization is induced for linear combinations of empirical features constructed in the literatures of kernel principal component analysis and kernel projection machines based on kernels and samples in addition to the separability of the involved optimization problem caused by the empirical features we carry out sparsity and error analysis giving bounds in the norm of the reproducing kernel hilbert space based on a priori conditions which do not require assumptions on sparsity in terms of any basis or system in particular we show that as the concave exponent q of the concave regularizer increases to 1 the learning ability of the algorithm improves some numerical simulations for both artificial and real mhc peptide binding data involving the lq regularizer and the scad penalty are presented to demonstrate the sparsity and error analysis

fast learning rates for plug in classifiers it has been recently shown that under the margin or low noise assumption there exist classifiers attaining fast rates of convergence of the excess bayes risk that is rates faster than n 1 2 the work on this subject has suggested the following two conjectures i the best achievable fast rate is of the order n 1 and ii the plug in classifiers generally converge more slowly than the classifiers based on empirical risk minimization we show that both conjectures are not correct in particular we construct plug in classifiers that can achieve not only fast but also super fast rates that is rates faster than n 1 we establish minimax lower bounds showing that the obtained rates cannot be improved

a max norm constrained minimization approach to 1 bit matrix completion we consider in this paper the problem of noisy 1 bit matrix completion under a general non uniform sampling distribution using the max norm as a convex relaxation for the rank a max norm constrained maximum likelihood estimate is introduced and studied the rate of convergence for the estimate is obtained information theoretical methods are used to establish a minimax lower bound under the general sampling model the minimax upper and lower bounds together yield the optimal rate of convergence for the frobenius norm loss computational algorithms and numerical performance are also discussed

the minimum description length principle in coding and modeling we review the principles of minimum description length and stochastic complexity as used in data compression and statistical modeling stochastic complexity is formulated as the solution to optimum universal coding problems extending shannon s basic source coding theorem the normalized maximized likelihood mixture and predictive codings are each shown to achieve the stochastic complexity to within asymptotically vanishing terms we assess the performance of the minimum description length criterion both from the vantage point of quality of data compression and accuracy of statistical inference context tree modeling density estimation and model selection in gaussian linear regression serve as examples

parametric or nonparametric a parametricness index for model selection in model selection literature two classes of criteria perform well asymptotically in different situations bayesian information criterion bic as a representative is consistent in selection when the true model is finite dimensional parametric scenario akaike s information criterion aic performs well in an asymptotic efficiency when the true model is infinite dimensional nonparametric scenario but there is little work that addresses if it is possible and how to detect the situation that a specific model selection problem is in in this work we differentiate the two scenarios theoretically under some conditions we develop a measure parametricness index pi to assess whether a model selected by a potentially consistent procedure can be practically treated as the true model which also hints on aic or bic is better suited for the data for the goal of estimating the regression function a consequence is that by switching between aic and bic based on the pi the resulting regression estimator is simultaneously asymptotically efficient for both parametric and nonparametric scenarios in addition we systematically investigate the behaviors of pi in simulation and real data and show its usefulness

consistency of restricted maximum likelihood estimators of principal components in this paper we consider two closely related problems estimation of eigenvalues and eigenfunctions of the covariance kernel of functional data based on possibly irregular measurements and the problem of estimating the eigenvalues and eigenvectors of the covariance matrix for high dimensional gaussian vectors in 23 a restricted maximum likelihood reml approach has been developed to deal with the flrst problem in this paper we establish consistency and derive rate of convergence of the reml estimator for the functional data case under appropriate smoothness conditions moreover we prove that when the number of measurements per sample curve is bounded under squared error loss the rate of convergence of the reml estimators of eigenfunctions is near optimal in the case of gaussian vectors asymptotic consistency and an e cient score representation of the estimators are obtained under the assumption that the efiective dimension grows at a rate slower than the sample size these results are derived through an explicit utilization of the intrinsic geometry of the parameter space which is non euclidean moreover the results derived in this paper suggest an asymptotic equivalence between the inference on functional data with dense measurements and that of the high dimensional gaussian vectors

a general minimax result for relative entropy suppose nature picks a probability measure p sub spl theta on a complete separable metric space x at random from a measurable set p sub spl theta p spl theta spl theta spl isin spl theta then without knowing spl theta a statistician picks a measure q on s finally the statistician suffers a loss d p sub 0 spl par q the relative entropy between p sub spl theta and q we show that the minimax and maximin values of this game are always equal and there is always a minimax strategy in the closure of the set of all bayes strategies this generalizes previous results of gallager 1979 and davisson and leon garcia 1980

an asymptotic property of model selection criteria probability models are estimated by use of penalized log likelihood criteria related to akaike 1973 information criterion aic and minimum description length mdl the accuracies of the density estimators are shown to be related to the tradeoff between three terms the accuracy of approximation the model dimension and the descriptive complexity of the model classes the asymptotic risk is determined under conditions on the penalty term and is shown to be minimax optimal for some cases as an application we show that the optimal rate of convergence is simultaneously achieved for log densities in sobolev spaces w sub 2 sup s u without knowing the smoothness parameter s and norm parameter u in advance applications to neural network models and sparse density function estimation are also provided

consistent model selection and data driven smooth tests for longitudinal data in the estimating equations approach summary model selection for marginal regression analysis of longitudinal data is challenging owing to the presence of correlation and the difficulty of specifying the full likelihood particularly for correlated categorical data the paper introduces a novel bayesian information criterion type model selection procedure based on the quadratic inference function which does not require the full likelihood or quasi likelihood with probability approaching 1 the criterion selects the most parsimonious correct model although a working correlation matrix is assumed there is no need to estimate the nuisance parameters in the working correlation matrix moreover the model selection procedure is robust against the misspecification of the working correlation matrix the criterion proposed can also be used to construct a data driven neyman smooth test for checking the goodness of fit of a postulated model this test is especially useful and often yields much higher power in situations where the classical directional test behaves poorly the finite sample performance of the model selection and model checking procedures is demonstrated through monte carlo studies and analysis of a clinical trial data set

covering numbers for convex functions in this paper we study the covering numbers of the space of convex and uniformly bounded functions in multidimension we find optimal upper and lower bounds for the epsilon covering number of cal c a b d b in the l p metric 1 leq p in terms of the relevant constants where d geq 1 a b 0 and cal c a b d b denotes the set of all convex functions on a b d that are uniformly bounded by b we summarize previously known results on covering numbers for convex functions and also provide alternate proofs of some known results our results have direct implications in the study of rates of convergence of empirical minimization procedures as well as optimal convergence rates in the numerous convexity constrained function estimation problems

adaptive model selection using empirical complexities given n independent replicates of a jointly distributed pair x y rd r we wish to select from a fixed sequence of model classes f1 f2 a deterministic prediction rule f rd r whose risk is small we investigate the possibility of empirically assessing the complexity of each model class that is the actual difficulty of the estimation problem within each class the estimated complexities are in turn used to define an adaptive model selection procedure which is based on complexity penalized empirical risk the available data are divided into two parts the first is used to form an empirical cover of each model class and the second is used to select a candidate rule from each cover based on empirical risk the covering radii are determined empirically to optimize a tight upper bound on the estimation error an estimate is chosen from the list of candidates in order to minimize the sum of class complexity and empirical risk a distinguishing feature of the approach is that the complexity of each model class is assessed empirically based on the size of its empirical cover finite sample performance bounds are established for the estimates and these bounds are applied to several non parametric estimation problems the estimates are shown to achieve a favorable tradeoff between approximation and estimation error and to perform as well as if the distribution dependent complexities of the model classes were known beforehand in addition it is shown that the estimate can be consistent and even possess near optimal rates of convergence when each model class has an infinite vc or pseudo dimension for regression estimation with squared loss we modify our estimate to achieve a faster rate of convergence appears in annals of statistics vol 27 1830 1864 ams 1991 subject classifications primary 62g07 62g20 secondary 62h30

minimax estimation in sparse canonical correlation analysis canonical correlation analysis is a widely used multivariate statistical technique for exploring the relation between two sets of variables this paper considers the problem of estimating the leading canonical correlation directions in high dimensional settings recently under the assumption that the leading canonical correlation directions are sparse various procedures have been proposed for many high dimensional applications involving massive data sets however there has been few theoretical justification available in the literature in this paper we establish rate optimal nonasymptotic minimax estimation with respect to an appropriate loss function for a wide range of model spaces two interesting phenomena are observed first the minimax rates are not affected by the presence of nuisance parameters namely the covariance matrices of the two sets of random variables though they need to be estimated in the canonical correlation analysis problem second we allow the presence of the residual canonical correlation directions however they do not influence the minimax rates under a mild condition on eigengap a generalized sin theta theorem and an empirical process bound for gaussian quadratic forms under rank constraint are used to establish the minimax upper bounds which may be of independent interest

geometrizing rates of convergence ii consider estimating a functional t f of an unknown distribution f in mathbf f from data x 1 cdots x n i i d f let omega varepsilon denote the modulus of continuity of the functional t over mathbf f computed with respect to hellinger distance for well behaved loss functions l t we show that inf t n sup mathbf f e fl t n t f is equivalent to l omega n 1 2 to within constants whenever t is linear and mathbf f is convex the same conclusion holds in three nonlinear cases estimating the rate of decay of a density estimating the mode and robust nonparametric regression we study the difficulty of testing between the composite infinite dimensional hypotheses h 0 t f leq t and h 1 t f geq t delta our results hold in the cases studied because the difficulty of the full infinite dimensional composite testing problem is comparable to the difficulty of the hardest simple two point testing subproblem

a lower bound on the error in nonparametric regression type problems let x1 y1 xn yn be a sample denote the conditional density of ylxi xi as f ylxi 0 xi and 0 an element of a metric space 9 d a lower bound is provided for the d error in estimating 0 the order of the bound depends on the local behavior of the kullback information of the conditional density as an application we consider the case where 1 is the space of q smooth functions on 0 i d metrized with the lr distance 1 r oo

consistency of bernstein polynomial posteriors a bernstein prior is a probability measure on the space of all the distribution functions on 0 1 under very general assumptions it selects absolutely continuous distribution functions whose densities are mixtures of known beta densities the bemstein prior is of interest in bayesian nonparametric inference with continuous data we study the consistency of the posterior from a bernstein prior we first show that under mild assumptions the posterior is weakly consistent for any distribution function p o on 0 1 with continuous and bounded lebesgue density with slightly stronger assumptions on the prior the posterior is also hellinger consistent this implies that the predictive density from a bernstein prior which is a bayesian density estimate converges in the hellinger sense to the true density assuming that it is continuous and bounded we also study a sieve maximum likelihood version of the density estimator and show that it is also hellinger consistent under weak assumptions when the order of the bernstein polynomial i e the number of components in the beta distribution mixture is truncated we show that under mild restrictions the posterior concentrates on the set of pseudotrue densities finally we study the behaviour of the predictive density numerically and we also study a hybrid bayes maximum likelihood density estimator

information theoretic asymptotics of bayes methods in the absence of knowledge of the true density function bayesian models take the joint density function for a sequence of n random variables to be an average of densities with respect to a prior the authors examine the relative entropy distance d sub n between the true density and the bayesian density and show that the asymptotic distance is d 2 log n c where d is the dimension of the parameter vector therefore the relative entropy rate d sub n n converges to zero at rate log n n the constant c which the authors explicitly identify depends only on the prior density function and the fisher information matrix evaluated at the true parameter value consequences are given for density estimation universal data compression composite hypothesis testing and stock market portfolio selection