information theoretic determination of minimax rates of convergence we present some general results determining minimax bounds on statistical risk for density estimation based on certain information theoretic considerations these bounds depend only on metric entropy conditions and are used to identify the minimax rates of convergence

evaluation of cox s model and logistic regression for matched case control data with time dependent covariates a simulation study case control studies are typically analysed using the conventional logistic model which does not directly account for changes in the covariate values over time yet many exposures may vary over time the most natural alternative to handle such exposures would be to use the cox model with time dependent covariates however its application to case control data opens the question of how to manipulate the risk sets through a simulation study we investigate how the accuracy of the estimates of cox s model depends on the operational definition of risk sets and or on some aspects of the time varying exposure we also assess the estimates obtained from conventional logistic regression the lifetime experience of a hypothetical population is first generated and a matched case control study is then simulated from this population we control the frequency the age at initiation and the total duration of exposure as well as the strengths of their effects all models considered include a fixed in time covariate and one or two time dependent covariate s the indicator of current exposure and or the exposure duration simulation results show that none of the models always performs well the discrepancies between the odds ratios yielded by logistic regression and the true hazard ratio depend on both the type of the covariate and the strength of its effect in addition it seems that logistic regression has difficulty separating the effects of inter correlated time dependent covariates by contrast each of the two versions of cox s model systematically induces either a serious under estimation or a moderate over estimation bias the magnitude of the latter bias is proportional to the true effect suggesting that an improved manipulation of the risk sets may eliminate or at least reduce the bias

a general framework for updating belief distributions we propose a framework for general bayesian inference we argue that a valid update of a prior belief distribution to a posterior can be made for parameters which are connected to observations through a loss function rather than the traditional likelihood function which is recovered as a special case modern application areas make it increasingly challenging for bayesians to attempt to model the true data generating mechanism for instance when the object of interest is low dimensional such as a mean or median it is cumbersome to have to achieve this via a complete model for the whole data distribution more importantly there are settings where the parameter of interest does not directly index a family of density functions and thus the bayesian approach to learning about such parameters is currently regarded as problematic our framework uses loss functions to connect information in the data to functionals of interest the updating of beliefs then follows from a decision theoretic approach involving cumulative loss functions importantly the procedure coincides with bayesian updating when a true likelihood is known yet provides coherent subjective inference in much more general settings connections to other inference frameworks are highlighted

generalization error bounds for bayesian mixture algorithms bayesian approaches to learning and estimation have played a significant role in the statistics literature over many years while they are often provably optimal in a frequentist setting and lead to excellent performance in practical applications there have not been many precise characterizations of their performance for finite sample sizes under general conditions in this paper we consider the class of bayesian mixture algorithms where an estimator is formed by constructing a data dependent mixture over some hypothesis space similarly to what is observed in practice our results demonstrate that mixture approaches are particularly robust and allow for the construction of highly complex estimators while avoiding undesirable overfitting effects our results while being data dependent in nature are insensitive to the underlying model assumptions and apply whether or not these hold at a technical level the approach applies to unbounded functions constrained only by certain moment conditions finally the bounds derived can be directly applied to non bayesian mixture approaches such as boosting and bagging

rates of convergence for minimum contrast estimators we shall present here a general study of minimum contrast estimators in a nonparametric setting although our results are also valid in the classical parametric case for independent observations these estimators include many of the most popular estimators in various situations such as maximum likelihood estimators least squares and other estimators of the regression function estimators for mixture models or deconvolution the main theorem relates the rate of convergence of those estimators to the entropy structure of the space of parameters optimal rates depending on entropy conditions are already known at least for some of the models involved and they agree with what we get for minimum contrast estimators as long as the entropy counts are not too large but under some circumstances large entropies or changes in the entropy structure due to local perturbations the resulting the rates are only suboptimal counterexamples are constructed which show that the phenomenon is real for non parametric maximum likelihood or regression this proves that under purely metric assumptions our theorem is optimal and that minimum contrast estimators happen to be suboptimal

constructing embedded lattice rules for multivariate integration lattice rules are a family of equal weight cubature formulae for approximating high dimensional integrals by now it is well established that good generating vectors for lattice rules having n points can be constructed component by component for integrands belonging to certain weighted function spaces and that they can achieve the optimal rate of convergence although the lattice rules constructed this way are extensible in dimension they are not extensible in n thus when n is changed the generating vector needs to be constructed anew in this paper we introduce a new algorithm for constructing good generating vectors for embedded lattice rules which can be used for a range of n while still being extensible in dimension by using an adaptation of the fast component by component construction algorithm which makes use of fast fourier transforms we are able to obtain good generating vectors for thousands of dimensions and millions of points under both product weight and order dependent weight settings at the cost of calo d n log n 2 operations with a sufficiently large number of points and good overall quality these embedded lattice rules can be used for practical purposes in the same way as a low discrepancy sequence we show for a range of weight settings in an unanchored sobolev space that our embedded lattice rules achieve the same optimal rate of convergence calo n 1 delta delta 0 as those constructed for a fixed number of points and that the implied constant gains only a factor of 1 30 to 1 55

on strong tractability of weighted multivariate integration we prove that for every dimension s and every number n of points there exists a point set pn s whose weighted unanchored l discrepancy is bounded from above by c b n1 2 b independently of s provided that the sequence k has k 1 a k for some even arbitrarily large a here b is a positive number that could be chosen arbitrarily close to zero and c b depends on b but not on s or n this result yields strong tractability of the corresponding integration problems including approximation of weighted integrals d f x x dx over unbounded domains such as d r s it also supplements the results that provide an upper bound of the form c s n when k 1

pac bayesian generalisation error bounds for gaussian process classification approximate bayesian gaussian process gp classification techniques are powerful non parametric learning methods similar in appearance and performance to support vector machines based on simple probabilistic models they render interpretable results and can be embedded in bayesian frameworks for model selection feature selection etc in this paper by applying the pac bayesian theorem of mcallester 1999a we prove distribution free generalisation error bounds for a wide range of approximate bayesian gp classification techniques we also provide a new and much simplified proof for this powerful theorem making use of the concept of convex duality which is a backbone of many machine learning techniques we instantiate and test our bounds for two particular gpc techniques including a recent sparse method which circumvents the unfavourable scaling of standard gp algorithms as is shown in experiments on a real world task the bounds can be very tight for moderate training sample sizes to the best of our knowledge these results provide the tightest known distribution free error bounds for approximate bayesian gpc methods giving a strong learning theoretical justification for the use of these techniques

from ɛ entropy to kl entropy analysis of minimum information complexity density estimation we consider an extension of e entropy to a kl divergence based complexity measure for randomized density estimation methods based on this extension we develop a general information theoretical inequality that measures the statistical complexity of some deterministic and randomized density estimators consequences of the new inequality will be presented in particular we show that this technique can lead to improvements of some classical results concerning the convergence of minimum description length and bayesian posterior distributions moreover we are able to derive clean finite sample convergence bounds that are not obtainable using previous approaches

fast rates in statistical and online learning the speed with which a learning algorithm converges as it is presented with more data is a central problem in machine learning a fast rate of convergence means less data is needed for the same level of performance the pursuit of fast rates in online and statistical learning has led to the discovery of many conditions in learning theory under which fast learning is possible we show that most of these conditions are special cases of a single unifying condition that comes in two forms the central condition for proper learning algorithms that always output a hypothesis in the given model and stochastic mixability for online algorithms that may make predictions outside of the model we show that under surprisingly weak assumptions both conditions are in a certain sense equivalent the central condition has a re interpretation in terms of convexity of a set of pseudoprobabilities linking it to density estimation under misspecification for bounded losses we show how the central condition enables a direct proof of fast rates and we prove its equivalence to the bernstein condition itself a generalization of the tsybakov margin condition both of which have played a central role in obtaining fast rates in statistical learning yet while the bernstein condition is two sided the central condition is one sided making it more suitable to deal with unbounded losses in its stochastic mixability form our condition generalizes both a stochastic exp concavity condition identified by juditsky rigollet and tsybakov and vovk s notion of mixability our unifying conditions thus provide a substantial step towards a characterization of fast rates in statistical learning similar to how classical mixability characterizes constant regret in the sequential prediction with expert advice setting

assigning a value to a power likelihood in a general bayesian model summarybayesian robustness under model misspecification is a current area of active research among recent ideas is that of raising the likelihood function to a power in this paper we discuss the choice of appropriate power and provide examples

fast algorithms for component by component construction of rank 1 lattice rules in shift invariant reproducing kernel hilbert spaces we reformulate the original component by component algorithm for rank 1 lattices in a matrix vector notation so as to highlight its structural properties for function spaces similar to a weighted korobov space we derive a technique which has construction cost o snlog n in contrast with the original algorithm which has construction cost o sn 2 herein s is the number of dimensions and n the number of points taken prime in contrast to other approaches to speed up construction our fast algorithm computes exactly the same quantity as the original algorithm the presented algorithm can also be used to construct randomly shifted lattice rules in weighted sobolev spaces

on computing the lattice rule criterion r lattice rules are integration rules for approximating integrals of periodic functions over the s dimensional unit cube one criterion for measuring the goodness of lattice rules is the quantity r this quantity is defined as a sum which contains about n sup s 1 terms where n is the number of quadrature points although various bounds involving r are known a procedure for calculating r itself does not appear to have been given previously here we show how an asymptotic series can be used to obtain an accurate approximation to r whereas an efficient direct calculation of r requires o nn sub 1 operations where n sub 1 is the largest invariant of the rule the use of this asymptotic expansion allows the operation count to be reduced to o n a complete error analysis for the asymptotic expansion is given the results of some calculations of r are also given 19 refs 2 tabs

robust bayesian inference via coarsening the standard approach to bayesian inference is based on the assumption that the distribution of the data belongs to the chosen model class however even a small violation of this assumption can ha

good lattice rules based on the general weighted star discrepancy we study the problem of constructing rank 1 lattice rules which have good bounds on the weighted star discrepancy here the non negative weights are general weights rather than the product weights considered in most earlier works in order to show the existence of such good lattice rules we use an averaging argument and a similar argument is used later to prove that these lattice rules may be obtained using a component by component cbc construction of the generating vector under appropriate conditions on the weights these lattice rules satisfy strong tractability bounds on the weighted star discrepancy particular classes of weights known as order dependent and finite order weights are then considered and we show that the cost of the construction can be very much reduced for these two classes of weights

shifted lattice rules based on a general weighted discrepancy for integrals over euclidean space we approximate weighted integrals over euclidean space by using shifted rank 1 lattice rules with good bounds on the generalised weighted star discrepancy this version of the discrepancy corresponds to the classic l weighted star discrepancy via a mapping to the unit cube the weights here are general weights rather than the product weights considered in earlier works on integrals over r d known methods based on an averaging argument are used to show the existence of these lattice rules while the component by component technique is used to construct the generating vector of these shifted lattice rules we prove that the bound on the weighted star discrepancy considered here is of order o n 1 d for any d 0 and with the constant involved independent of the dimension this convergence rate is better than the o n 1 2 achieved so far for both monte carlo and quasi monte carlo methods

information theoretic upper and lower bounds for statistical estimation in this paper we establish upper and lower bounds for some statistical estimation problems through concise information theoretic arguments our upper bound analysis is based on a simple yet general inequality which we call the information exponential inequality we show that this inequality naturally leads to a general randomized estimation method for which performance upper bounds can be obtained the lower bounds applicable for all statistical estimators are obtained by original applications of some well known information theoretic inequalities and approximately match the obtained upper bounds for various important problems moreover our framework can be regarded as a natural generalization of the standard minimax framework in that we allow the performance of the estimator to vary for different possible underlying distributions according to a predefined prior

posterior concentration and fast convergence rates for generalized bayesian learning abstract in this paper we study the learning rate of generalized bayes estimators in a general setting where the hypothesis class can be uncountable and have an irregular shape the loss function can have heavy tails and the optimal hypothesis may not be unique we prove that under the multi scale bernstein s condition the generalized posterior distribution concentrates around the set of optimal hypotheses and the generalized bayes estimator can achieve fast learning rate our results are applied to show that the standard bayesian linear regression is robust to heavy tailed distributions

minimum complexity density estimation the authors introduce an index of resolvability that is proved to bound the rate of convergence of minimum complexity density estimators as well as the information theoretic redundancy of the corresponding total description length the results on the index of resolvability demonstrate the statistical effectiveness of the minimum description length principle as a method of inference the minimum complexity estimator converges to true density nearly as fast as an estimator based on prior knowledge of the true subclass of densities interpretations and basic properties of minimum complexity estimators are discussed some regression and classification problems that can be examined from the minimum description length framework are considered

lattice methods for multiple integration theory error analysis and examples this paper is concerned with numerical integration over the unit cube in s dimensions of functions that have reasonably smooth periodic extensions in each dimension the concept of a lattice rule is introduced and error bounds are developed in terms of the dual lattice examples of lattice rules discussed in the paper are the rectangle rule the number theoretic good lattice methods of korobov and others the body centred cubic rule and a generalization of the latter to a family w nr the w nr rules appear to have interesting extrapolation properties both for fixed r and for r n

on the properties of variational approximations of gibbs posteriors the pac bayesian approach is a powerful set of techniques to derive nonasymptotic risk bounds for random estimators the corresponding optimal distribution of estimators usually called the gibbs posterior is unfortunately often intractable one may sample from it using markov chain monte carlo but this is usually too slow for big datasets we consider instead variational approximations of the gibbs posterior which are fast to compute we undertake a general study of the properties of such approximations our main finding is that such a variational approximation has often the same rate of convergence as the original pac bayesian procedure it approximates in addition we show that when the risk function is convex a variational approximation can be obtained in polynomial time using a convex solver we give finite sample oracle inequalities for the corresponding estimator we specialize our results to several learning tasks classification ranking matrix completion discuss how to implement a variational approximation in each case and illustrate the good properties of said approximation on real datasets

generalizing the fano inequality the fano inequality gives a lower bound on the mutual information between two random variables that take values on an m element set provided at least one of the random variables is equiprobable the authors show several simple lower bounds on mutual information which do not assume such a restriction in particular this ran be accomplished by replacing log m with the infinite order renyi entropy in the fano inequality applications to hypothesis testing are exhibited along with bounds on mutual information in terms of the a priori and a posteriori error probabilities

on the step by step construction of quasi monte carlo integration rules that achieve strong tractability error bounds in weighted sobolev spaces we develop and justify an algorithm for the construction of quasi monte carlo qmc rules for integration in weighted sobolev spaces the rules so constructed are shifted rank 1 lattice rules the parameters characterising the shifted lattice rule are found component by component the d 1 th component of the generator vector and the shift are obtained by successive 1 dimensional searches with the previous d components kept unchanged the rules constructed in this way are shown to achieve a strong tractability error bound in weighted sobolev spaces a search for n point rules with n prime and all dimensions 1 to d requires a total cost of o n3d2 operations this may be reduced to o n3d operations at the expense of o n2 storage numerical values of parameters and worst ease errors are given for dimensions up to 40 and n up to a few thousand the worst case errors for these rules are found to be much smaller than the theoretical bounds

good lattice rules in weighted korobov spaces with general weights we study the problem of multivariate integration and the construction of good lattice rules in weighted korobov spaces with general weights these spaces are not necessarily tensor products of spaces of univariate functions sufficient conditions for tractability and strong tractability of multivariate integration in such weighted function spaces are found these conditions are also necessary if the weights are such that the reproducing kernel of the weighted korobov space is pointwise non negative the existence of a lattice rule which achieves the nearly optimal convergence order is proven a component by component cbc algorithm that constructs good lattice rules is presented the resulting lattice rules achieve tractability or strong tractability error bounds and achieve nearly optimal convergence order for suitably decaying weights we also study special weights such as finite order and order dependent weights for these special weights the cost of the cbc algorithm is polynomial numerical computations show that the lattice rules constructed by the cbc algorithm give much smaller worst case errors than the mean worst case errors over all quasi monte carlo rules or over all lattice rules and generally smaller worst case errors than the best korobov lattice rules in dimensions up to hundreds numerical results are provided to illustrate the efficiency of cbc lattice rules and korobov lattice rules with suitably chosen weights in particular for high dimensional finance problems