probability estimation in the rare events regime we address the problem of estimating the probability of an observed string that is drawn i i d from an unknown distribution motivated by models of natural language we consider the regime in which the length of the observed string and the size of the underlying alphabet are comparably large in this regime the maximum likelihood distribution tends to overestimate the probability of the observed letters so the good turing probability estimator is typically used instead we show that when used to estimate the sequence probability the good turing estimator is not consistent in this regime we then introduce a novel sequence probability estimator that is consistent this estimator also yields consistent estimators for other quantities of interest and a consistent universal classifier

maximum likelihood estimation of functionals of discrete distributions we consider the problem of estimating functionals of discrete distributions and focus on a tight up to universal multiplicative constants for each specific functional nonasymptotic analysis of the worst case squared error risk of widely used estimators we apply concentration inequalities to analyze the random fluctuation of these estimators around their expectations and the theory of approximation using positive linear operators to analyze the deviation of their expectations from the true functional namely their bias we explicitly characterize the worst case squared error risk incurred by the maximum likelihood estimator mle in estimating the shannon entropy h p sum i 1 s p i ln p i and the power sum f alpha p sum i 1 s p i alpha alpha 0 up to universal multiplicative constants for each fixed functional for any alphabet size s leq infty and sample size n for which the risk may vanish as a corollary for shannon entropy estimation we show that it is necessary and sufficient to have n gg s observations for the mle to be consistent in addition we establish that it is necessary and sufficient to consider n gg s 1 alpha samples for the mle to consistently estimate f alpha p 0 the minimax rate optimal estimators for both problems require s ln s and s 1 alpha ln s samples which implies that the mle has a strictly sub optimal sample complexity when 1 we show that the worst case squared error rate of convergence for the mle is n 2 alpha 1 for infinite alphabet size while the minimax squared error rate is n ln n 2 alpha 1 when alpha geq 3 2 the mle achieves the minimax optimal rate n 1 regardless of the alphabet size as an application of the general theory we analyze the dirichlet prior smoothing techniques for shannon entropy estimation in this context one approach is to plug in the dirichlet prior smoothed distribution into the entropy functional while the other one is to calculate the bayes estimator for entropy under the dirichlet prior for squared error which is the conditional expectation we show that in general such estimators do not improve over the maximum likelihood estimator no matter how we tune the parameters in the dirichlet prior this approach cannot achieve the minimax rates in entropy estimation the performance of the minimax rate optimal estimator with n samples is essentially at least as good as that of dirichlet smoothed entropy estimators with n ln n samples

investigation on the high order approximation of the entropy bias abstract the estimation of entropy from experimental data has a considerable bias when the discretization of the variable domain is comparable to the sample size in this case the source of the bias is the difference between the a priori distribution and the observed distribution from sampled data in this paper we estimate the entropy bias considering an infinite sum of central moments of the binomial distribution using two probability mass functions we analyze the bias in the light of the ratio between the number of the partition of the domain and the sample size the main motivation of this study is improving statistical hypothesis testing in which probabilities are conceived beforehand we examine the adequacy of high order approximation according to the ratio between the sample size and the number of domain partitions finally we expand the analysis to the entropy derived mutual information and present an application for network reconstruction

quantum query complexity of entropy estimation estimation of shannon and renyi entropies of unknown discrete distributions is a fundamental problem in statistical property testing in this paper we give the first quantum algorithms for estimating alpha renyi entropies shannon entropy being 1 renyi entropy in particular we demonstrate a quadratic quantum speedup for shannon entropy estimation and a generic quantum speedup for alpha renyi entropy estimation for all alpha geq 0 values including tight bounds for the shannon entropy the hartley entropy alpha 0 and the collision entropy alpha 2 we also provide quantum upper bounds for estimating min entropy alpha infty as well as the kullback leibler divergence we complement our results with quantum lower bounds on alpha renyi entropy estimation for all alpha geq 0 values our approach is inspired by the pioneering work of bravyi harrow and hassidim bhh however with many new technical ingredients 1 we improve the error dependence of the bhh framework by a fine tuned error analysis together with montanaro s approach to estimating the expected output of quantum subroutines for alpha 0 1 2 we develop a procedure similar to cooling schedules in simulated annealing for general alpha geq 0 and 3 in the cases of integer alpha geq 2 and alpha infty we reduce the entropy estimation problem to the alpha distinctness and lceil log n rceil distinctness problems respectively

direct estimation of density functionals using a polynomial basis a number of fundamental quantities in statistical signal processing and information theory can be expressed as integral functions of two probability density functions such quantities are called density functionals as they map density functions onto the real line for example information divergence functions measure the dissimilarity between two probability density functions and are useful in a number of applications typically estimating these quantities requires complete knowledge of the underlying distribution followed by multidimensional integration existing methods make parametric assumptions about the data distribution or use nonparametric density estimation followed by high dimensional integration in this paper we propose a new alternative we introduce the concept of data driven basis functions functions of distributions whose value we can estimate given only samples from the underlying distributions without requiring distribution fitting or direct integration we derive a new data driven complete basis that is similar to the deterministic bernstein polynomial basis and develop two methods for performing basis expansions of functionals of two distributions we also show that the new basis set allows us to approximate functions of distributions as closely as desired finally we evaluate the methodology by developing data driven estimators for the kullback leibler divergences and the hellinger distance and by constructing empirical estimates of tight bounds on the bayes error rate

diagnostic testing and evaluation of maximum likelihood models abstract the paper develops a unified theory of likelihood specification testing based on m estimators of auxiliary parameters the theory is sufficiently general to encompass a wide class of specification tests including moment based tests pearson type goodness of fit tests the information matrix test and the cox test the paper also presents a framework based on frechet differentiation for determining the effects of misspecification on the almost sure limits of parameter estimates and specification test statistics

demystifying fixed k nearest neighbor information estimators estimating mutual information from independent identically distributed samples drawn from an unknown joint density function is a basic statistical problem of broad interest with multitudinous applications the most popular estimator is the one proposed by kraskov stogbauer and grassberger ksg in 2004 and is nonparametric and based on the distances of each sample to its k mathrm th nearest neighboring sample where k is a fixed small integer despite of its widespread use part of scientific software packages theoretical properties of this estimator have been largely unexplored in this paper we demonstrate that the estimator is consistent and also identify an upper bound on the rate of convergence of the ell 2 error as a function of a number of samples we argue that the performance benefits of the ksg estimator stems from a curious correlation boosting effect and build on this intuition to modify the ksg estimator in novel ways to construct a superior estimator as a by product of our investigations we obtain nearly tight rates of convergence of the ell 2 error of the well known fixed k nearest neighbor estimator of differential entropy by kozachenko and leonenko

information criteria for selecting possibly misspecified parametric models abstract we consider penalized likelihood criteria for selecting models of dependent processes the models may be strictly nested overlapping or nonnested linear or nonlinear and correctly specified or misspecified we provide sufficient conditions on the penalty to guarantee the selection with probability one or with probability approaching one of the model attaining the lower average kullback leibler information criterion klic or when both have the same klic the more parsimonious model as special cases our results describe the akaike schwarz and hannan quinn information criteria as examples we consider selection of armax garch and star models

estimating entropy on m bins given fewer than m samples consider a sequence p sub n of discrete probability measures supported on m sub n points and assume that we observe n independent and identically distributed i i d samples from each p sub n we demonstrate the existence of an estimator of the entropy h p sub n which is consistent even if the ratio n m sub n is bounded and as a corollary even if this ratio tends to zero albeit at a sufficiently slow rate

towards estimating extremal serial dependence via the bootstrapped extremogram abstract davis and mikosch 2009a introduced the extremogram as a flexible quantitative tool for measuring various types of extremal dependence in a stationary time series there we showed some standard statistical properties of the sample extremogram a major difficulty was the construction of credible confidence bands for the extremogram in this paper we employ the stationary bootstrap to overcome this problem the use of the stationary bootstrap for the extremogram and the resulting interpretations are illustrated with several financial time series

minimax rates of entropy estimation on large alphabets via best polynomial approximation consider the problem of estimating the shannon entropy of a distribution over k elements from n independent samples we show that the minimax mean square error is within the universal multiplicative constant factors of big k n log k big 2 log 2 k n if n exceeds a constant factor of k log k otherwise there exists no consistent estimator this refines the recent result of valiant and valiant that the minimal sample size for consistent entropy estimation scales according to theta k log k the apparatus of the best polynomial approximation plays a key role in both the construction of optimal estimators and by a duality argument the minimax lower bound

quantifying stimulus discriminability a comparison of information theory and ideal observer analysis performance in sensory discrimination tasks is commonly quantified using either information theory or ideal observer analysis these two quantitative frameworks are often assumed to be equivalent for example higher mutual information is said to correspond to improved performance of an ideal observer in a stimulus estimation task to the contrary drawing on and extending previous results we show that five information theoretic quantities entropy response conditional entropy specific information equivocation and mutual information violate this assumption more positively we show how these information measures can be used to calculate upper and lower bounds on ideal observer performance and vice versa the results show that the mathematical resources of ideal observer analysis are preferable to information theory for evaluating performance in a stimulus discrimination task we also discuss the applicability of information theory to questions that ideal observer analysis cannot address

inference on the tail process with application to financial time series modeling abstract to draw inference on serial extremal dependence within heavy tailed markov chains drees et al 2015 proposed nonparametric estimators of the spectral tail process the methodology can be extended to the more general setting of a stationary regularly varying time series the large sample distribution of the estimators is derived via empirical process theory for cluster functionals the finite sample performance of these estimators is evaluated via monte carlo simulations moreover two different bootstrap schemes are employed which yield confidence intervals for the pre asymptotic spectral tail process the stationary bootstrap and the multiplier block bootstrap the estimators are applied to stock price data to study the persistence of positive and negative shocks

a simple parametric model selection test abstractwe propose a simple model selection test for choosing among two parametric likelihoods which can be applied in the most general setting without any assumptions on the relation between the candidate models and the true distribution that is both one or neither is allowed to be correctly specified or misspecified they may be nested nonnested strictly nonnested or overlapping unlike in previous testing approaches no pretesting is needed since in each case the same test statistic together with a standard normal critical value can be used the new procedure controls asymptotic size uniformly over a large class of data generating processes we demonstrate its finite sample properties in a monte carlo experiment and its practical relevance in an empirical application comparing keynesian versus new classical macroeconomic models supplementary materials for this article are available online

bayesian bin distribution inference and mutual information we present an exact bayesian treatment of a simple yet sufficiently general probability distribution model we consider piecewise constant distributions p x with uniform second order prior over location of discontinuity points and assigned chances the predictive distribution and the model complexity can be determined completely from the data in a computational time that is linear in the number of degrees of freedom and quadratic in the number of possible values of x furthermore exact values of the expectations of entropies and their variances can be computed with polynomial effort the expectation of the mutual information becomes thus available too and a strict upper bound on its variance the resulting algorithm is particularly useful in experimental research areas where the number of available samples is severely limited e g neurophysiology estimates on a simulated data set provide more accurate results than using a previously proposed method

undersmoothed kernel entropy estimators we develop a ldquoplug inrdquo kernel estimator for the differential entropy that is consistent even if the kernel width tends to zero as quickly as 1 n where n is the number of independent and identically distributed i i d samples thus accurate density estimates are not required for accurate kernel entropy estimates in fact it is a good idea when estimating entropy to sacrifice some accuracy in the quality of the corresponding density estimate

indices for testing neural codes one of the most critical challenges in systems neuroscience is determining the neural code a principled framework for addressing this can be found in information theory with this approach one can determine whether a proposed code can account for the stimulus response relationship specifically one can compare the transmitted information between the stimulus and the hypothesized neural code with the transmitted information between the stimulus and the behavioral response if the former is smaller than the latter i e if the code cannot account for the behavior the code can be ruled out the information theoretic index most widely used in this context is shannon s mutual information the shannon test however is not ideal for this purpose while the codes it will rule out are truly nonviable there will be some nonviable codes that it will fail to rule out here we describe a wide range of alternative indices that can be used for ruling codes out the range includes a continuum from shannon information to measures of the performance of a bayesian decoder we analyze the relationship of these indices to each other and their complementary strengths and weaknesses for addressing this problem

a non parametric entropy based approach to detect changes in climate extremes summary the paper focuses primarily on temperature extremes measured at 24 european stations with at least 90 years of data here the term extremes refers to rare excesses of daily maxima and minima as mean temperatures in this region have been warming over the last century it is automatic that this positive shift can be detected also in extremes after removing this warming trend we focus on the question of determining whether other changes are still detectable in such extreme events as we do not want to hypothesize any parametric form of such possible changes we propose a new non parametric estimator based on the kullback leibler divergence tailored for extreme events the properties of our estimator are studied theoretically and tested with a simulation study our approach is also applied to seasonal extremes of daily maxima and minima for our 24 selected stations

the cross quantilogram measuring quantile dependence and testing directional predictability between time series this paper proposes the cross quantilogram to measure the quantile dependence between two time series we apply it to test the hypothesis that one time series has no directional predictability to another time series we establish the asymptotic distribution of the cross quantilogram and the corresponding test statistic the limiting distributions depend on nuisance parameters to construct consistent confidence intervals we employ the stationary bootstrap procedure we show the consistency of this bootstrap also we consider the self normalized approach which is shown to be asymptotically pivotal under the null hypothesis of no predictability we provide simulation studies and two empirical applications first we use the cross quantilogram to detect predictability from stock variance to excess stock return compared to existing tools used in the literature of stock return predictability our method provides a more complete relationship between a predictor and stock return second we investigate the systemic risk of individual financial institutions such as jp morgan chase goldman sachs and aig this article has supplementary materials online

the encompassing principle and its application to testing nonnested hypotheses the encompassing principle is used to develop a testing framework which unifies the literature on non nested testing allowing analysis of the relationship between alternative tests and in particular enabling asymptotic and finite sample equivalences and identities to be established easily when they exist as well as embracing nested hypothesis testing the concept of the implicit null hypothesis of a test is introduced to show that the effective acceptance region for some tests extends beyond the acceptance region corresponding to the null of interest and so such tests can be inconsistent against fixed alternatives closely related to the nominal null and alternative the analysis is illustrated by an application to two non nested linear regression models and it is shown that the conventional f test as well as all the one degree of freedom non nested tests has an encompassing interpretation and that the f test is a complete parametric encompassing test

bridging mixture model estimation and information bounds using i mmse we derive bounds on mutual information for arbitrary estimation problems in additive noise modeled using gaussian mixtures previous work exploiting the i minimum mean squared error mmse formula to formulate a bridge between bounds on the mmse for gaussian mixture model estimation problems and bounds on the mutual information are generalized to allow arbitrary noise modeling a novel upper bound on estimation information is also developed for the general estimation case in addition limits are analyzed to develop bounds on arbitrary entropy asymptotic behavior of all bounds and bound errors with some results bridged back to the mmse domain

likelihood ratio specification tests misspecification tests for parametric models f y that examine data for failure of moment conditions implied by the maintained parametric distribution are interpreted as score tests of h 0 0 in the context of a parametric family of distributions r y this family contains the maintained distribution as a special case 0 and has the property that only in that special case do the chosen moment conditions hold a likelihood ratio test of h 0 0 therefore constitutes an alternative test of the validity of the moment conditions this test admits a bartlett correction unlike conventional moment tests for which adjustments based on second order asymptotic theory may behave badly the dependence of the bartlett correction and of the o n 1 2 local power of the test on the way in which r y is constructed is studied in many cases the correction can be made to vanish leading to a specification test whose distribution is chi square to order o p n 2

testing probability distributions using conditional samples we study a new framework for property testing of probability distributions by considering distribution testing algorithms that have access to a conditional sampling oracle this is an oracle that takes as input a subset s subseteq n of the domain n of the unknown probability distribution cal d and returns a draw from the conditional probability distribution cal d restricted to s this new model allows considerable flexibility in the design of distribution testing algorithms in particular testing algorithms in this model can be adaptive we study a wide range of natural distribution testing problems in this new framework and some of its variants giving both upper and lower bounds on query complexity these problems include testing whether cal d is the uniform distribution cal u testing whether cal d cal d ast for an explicitly provided cal d ast testing whether two unknown distributions cal d 1 and cal d 2 are equivalent and estimating the va

tests of specification for parametric and semiparametric models abstract this paper provides a general framework for constructing specification tests for parametric and semiparametric models the paper develops new specification tests using the general framework in particular specification tests for semiparametric partially linear regression and sample selection models are introduced the results apply in time series and cross sectional contexts the method of proof exploits results concerning the stochastic equicontinuity or weak convergence of normalized sums of stochastic processes

estimation of entropy and mutual information we present some new results on the nonparametric estimation of entropy and mutual information first we use an exact local expansion of the entropy function to prove almost sure consistency and central limit theorems for three of the most commonly used discretized information estimators the setup is related to grenander s method of sieves and places no assumptions on the underlying probability measure generating the data second we prove a converse to these consistency theorems demonstrating that a misapplication of the most common estimation techniques leads to an arbitrarily poor estimate of the true information even given unlimited data this inconsistency theorem leads to an analytical approximation of the bias valid in surprisingly small sample regimes and more accurate than the usual 1 n formula of miller and madow over a large region of parameter space the two most practical implications of these results are negative 1 information estimates in a certain data regime are likely contaminated by bias even if bias corrected estimators are used and 2 confidence intervals calculated by standard techniques drastically underestimate the error of the most common estimation methods finally we note a very useful connection between the bias of entropy estimators and a certain polynomial approximation problem by casting bias calculation problems in this approximation theory framework we obtain the best possible generalization of known asymptotic bias results more interesting this framework leads to an estimator with some nice properties the estimator comes equipped with rigorous bounds on the maximum error over all possible underlying probability distributions and this maximum error turns out to be surprisingly small we demonstrate the application of this new estimator on both real and simulated data

estimating renyi entropy of discrete distributions it was shown recently that estimating the shannon entropy h p of a discrete k symbol distribution p requires theta k log k samples a number that grows near linearly in the support size in many applications h p can be replaced by the more general renyi entropy of order alpha and h alpha p we determine the number of samples needed to estimate h alpha p for all alpha showing that alpha requires a super linear roughly k 1 alpha samples noninteger alpha 1 requires a near linear k samples but perhaps surprisingly integer alpha 1 requires only theta k 1 1 alpha samples furthermore developing on a recently established connection between polynomial approximation and estimation of additive functions of the form sum x f p x we reduce the sample complexity for noninteger values of alpha by a factor of log k compared with the empirical estimator the estimators achieving these bounds are simple and run in time linear in the number of samples our lower bounds provide explicit constructions of distributions with different renyi entropies that are hard to distinguish

the stationary bootstrap abstract this article introduces a resampling procedure called the stationary bootstrap as a means of calculating standard errors of estimators and constructing confidence regions for parameters based on weakly dependent stationary observations previously a technique based on resampling blocks of consecutive observations was introduced to construct confidence intervals for a parameter of the m dimensional joint distribution of m consecutive observations where m is fixed this procedure has been generalized by constructing a blocks of blocks resampling scheme that yields asymptotically valid procedures even for a multivariate parameter of the whole i e infinite dimensional joint distribution of the stationary sequence of observations these methods share the construction of resampling blocks of observations to form a pseudo time series so that the statistic of interest may be recalculated based on the resampled data set but in the context of applying this method to stationary data it is natural

geometrizing rates of convergence ii consider estimating a functional t f of an unknown distribution f in mathbf f from data x 1 cdots x n i i d f let omega varepsilon denote the modulus of continuity of the functional t over mathbf f computed with respect to hellinger distance for well behaved loss functions l t we show that inf t n sup mathbf f e fl t n t f is equivalent to l omega n 1 2 to within constants whenever t is linear and mathbf f is convex the same conclusion holds in three nonlinear cases estimating the rate of decay of a density estimating the mode and robust nonparametric regression we study the difficulty of testing between the composite infinite dimensional hypotheses h 0 t f leq t and h 1 t f geq t delta our results hold in the cases studied because the difficulty of the full infinite dimensional composite testing problem is comparable to the difficulty of the hardest simple two point testing subproblem

statistical inference for max stable processes in space and time max stable processes have proved to be useful for the statistical modelling of spatial extremes several families of max stable random fields have been proposed in the literature one such representation is based on a limit of normalized and rescaled pointwise maxima of stationary gaussian processes that was first introduced by kabluchko and co workers this paper deals with statistical inference for max stable space time processes that are defined in an analogous fashion we describe pairwise likelihood estimation where the pairwise density of the process is used to estimate the model parameters for regular grid observations we prove strong consistency and asymptotic normality of the parameter estimates as the joint number of spatial locations and time points tends to furthermore we discuss extensions to irregularly spaced locations a simulation study shows that the method proposed works well for these models

a coincidence based test for uniformity given very sparsely sampled discrete data how many independent samples n do we need from a distribution p to decide that p is epsiv distant from uniform in an l1 sense sigmai 1 m p i 1 m epsiv here m is the number of bins on which the distribution is supported and is assumed known a priori somewhat surprisingly we only need n epsiv2 gt m 1 2 to make this decision reliably this condition is both sufficient and necessary the test for uniformity introduced here is based on the number of observed ldquocoincidencesrdquo samples that fall into the same bin the mean and variance of which may be computed explicitly for the uniform distribution and bounded nonparametrically for any distribution that is known to be epsiv distant from uniform some connections to the classical birthday problem are noted

a structural break test for extremal dependence in β mixing random vectors summarywe derive a structural break test for extremal dependence in beta mixing possibly high dimensional random vectors with either asymptotically dependent or asymptotically independent components existing tests require serially independent observations with asymptotically dependent components to avoid estimating a long run variance we use self normalization which obviates the need to estimate the coefficient of tail dependence when components are asymptotically independent simulations show favourable empirical size and power of the test which we apply to s p 500 and dax log returns we find evidence for one break in the coefficient of tail dependence for the upper and lower joint tail at the beginning of the 2007 08 financial crisis leading to more extremal co movement

minimax estimation of functionals of discrete distributions we propose a general methodology for the construction and analysis of essentially minimax estimators for a wide class of functionals of finite dimensional parameters and elaborate on the case of discrete distributions where the support size s is unknown and may be comparable with or even much larger than the number of observations n we treat the respective regions where the functional is nonsmooth and smooth separately in the nonsmooth regime we apply an unbiased estimator for the best polynomial approximation of the functional whereas in the smooth regime we apply a bias corrected version of the maximum likelihood estimator mle we illustrate the merit of this approach by thoroughly analyzing the performance of the resulting schemes for estimating two important information measures 1 the entropy h p sum i 1 s p i ln p i and 2 f alpha p sum i 1 s p i alpha alpha 0 we obtain the minimax l 2 rates for estimating these functionals in particular we demonstrate that our estimator achieves the optimal sample complexity n asymp s ln s for entropy estimation we also demonstrate that the sample complexity for estimating f alpha p 0 is n asymp s 1 alpha ln s which can be achieved by our estimator but not the mle for 1 we show the minimax l 2 rate for estimating f alpha p is n ln n 2 alpha 1 for infinite support size while the maximum l 2 rate for the mle is n 2 alpha 1 for all the above cases the behavior of the minimax rate optimal estimators with n samples is essentially that of the mle plug in rule with n ln n samples which we term effective sample size enlargement we highlight the practical advantages of our schemes for the estimation of entropy and mutual information we compare our performance with various existing approaches and demonstrate that our approach reduces running time and boosts the accuracy moreover we show that the minimax rate optimal mutual information estimator yielded by our framework leads to significant performance boosts over the chow liu algorithm in learning graphical models the wide use of information measure estimation suggests that the insights and estimators obtained in this paper could be broadly applicable