probability estimation in the rare events regime we address the problem of estimating the probability of an observed string that is drawn i i d from an unknown distribution motivated by models of natural language we consider the regime in which the length of the observed string and the size of the underlying alphabet are comparably large in this regime the maximum likelihood distribution tends to overestimate the probability of the observed letters so the good turing probability estimator is typically used instead we show that when used to estimate the sequence probability the good turing estimator is not consistent in this regime we then introduce a novel sequence probability estimator that is consistent this estimator also yields consistent estimators for other quantities of interest and a consistent universal classifier

maximum likelihood estimation of functionals of discrete distributions we consider the problem of estimating functionals of discrete distributions and focus on a tight up to universal multiplicative constants for each specific functional nonasymptotic analysis of the worst case squared error risk of widely used estimators we apply concentration inequalities to analyze the random fluctuation of these estimators around their expectations and the theory of approximation using positive linear operators to analyze the deviation of their expectations from the true functional namely their bias we explicitly characterize the worst case squared error risk incurred by the maximum likelihood estimator mle in estimating the shannon entropy h p sum i 1 s p i ln p i and the power sum f alpha p sum i 1 s p i alpha alpha 0 up to universal multiplicative constants for each fixed functional for any alphabet size s leq infty and sample size n for which the risk may vanish as a corollary for shannon entropy estimation we show that it is necessary and sufficient to have n gg s observations for the mle to be consistent in addition we establish that it is necessary and sufficient to consider n gg s 1 alpha samples for the mle to consistently estimate f alpha p 0 the minimax rate optimal estimators for both problems require s ln s and s 1 alpha ln s samples which implies that the mle has a strictly sub optimal sample complexity when 1 we show that the worst case squared error rate of convergence for the mle is n 2 alpha 1 for infinite alphabet size while the minimax squared error rate is n ln n 2 alpha 1 when alpha geq 3 2 the mle achieves the minimax optimal rate n 1 regardless of the alphabet size as an application of the general theory we analyze the dirichlet prior smoothing techniques for shannon entropy estimation in this context one approach is to plug in the dirichlet prior smoothed distribution into the entropy functional while the other one is to calculate the bayes estimator for entropy under the dirichlet prior for squared error which is the conditional expectation we show that in general such estimators do not improve over the maximum likelihood estimator no matter how we tune the parameters in the dirichlet prior this approach cannot achieve the minimax rates in entropy estimation the performance of the minimax rate optimal estimator with n samples is essentially at least as good as that of dirichlet smoothed entropy estimators with n ln n samples

delay independence of mutual information rate of two symbolic sequences introduced by shannon as a rate of actual transmission mutual information rate mir is an extension of mutual information to a pair of dynamical processes we show a delay independence theorem according to which mir is not sensitive to a time shift between the two processes numerical studies of several benchmark situations confirm that this theoretical asymptotic property remains valid for realistic finite sequences estimations based on block entropies and a causal state machine algorithm perform better than an estimation based on a lempel ziv compression algorithm provided that block length and maximum history length respectively can be chosen larger than the delay mir is thus a relevant index for measuring nonlinear correlations between two experimental or simulated sequences when the transmission delay in input output devices or dephasing in coupled systems is variable or unknown

estimating functions of probability distributions from a finite set of samples this paper addresses the problem of estimating a function of a probability distribution from a finite set of samples of that distribution a bayesian analysis of this problem is presented the optimal properties of the bayes estimators are discussed and as an example of the formalism closed form expressions for the bayes estimators for the moments of the shannon entropy function are derived then numerical results are presented that compare the bayes estimator to the frequency counts estimator for the shannon entropy we also present the closed form estimators all derived elsewhere for the mutual information mathrm ensuremath chi 2 covariance and some other statistics c 1995 the american physical society

new integrable generalizations of calogero moser quantum problem a one parameter deformation of calogero moser quantum problem is introduced it is shown that corresponding schrodinger operator is integrable for any value of the parameter and algebraically integrable in case of integer value

quantum query complexity of entropy estimation estimation of shannon and renyi entropies of unknown discrete distributions is a fundamental problem in statistical property testing in this paper we give the first quantum algorithms for estimating alpha renyi entropies shannon entropy being 1 renyi entropy in particular we demonstrate a quadratic quantum speedup for shannon entropy estimation and a generic quantum speedup for alpha renyi entropy estimation for all alpha geq 0 values including tight bounds for the shannon entropy the hartley entropy alpha 0 and the collision entropy alpha 2 we also provide quantum upper bounds for estimating min entropy alpha infty as well as the kullback leibler divergence we complement our results with quantum lower bounds on alpha renyi entropy estimation for all alpha geq 0 values our approach is inspired by the pioneering work of bravyi harrow and hassidim bhh however with many new technical ingredients 1 we improve the error dependence of the bhh framework by a fine tuned error analysis together with montanaro s approach to estimating the expected output of quantum subroutines for alpha 0 1 2 we develop a procedure similar to cooling schedules in simulated annealing for general alpha geq 0 and 3 in the cases of integer alpha geq 2 and alpha infty we reduce the entropy estimation problem to the alpha distinctness and lceil log n rceil distinctness problems respectively

consistent distribution free k sample and independence tests for univariate random variables a popular approach for testing if two univariate random variables are statistically independent consists of partitioning the sample space into bins and evaluating a test statistic on the binned data the partition size matters and the optimal partition size is data dependent while for detecting simple relationships coarse partitions may be best for detecting complex relationships a great gain in power can be achieved by considering finer partitions we suggest novel consistent distribution free tests that are based on summation or maximization aggregation of scores over all partitions of a fixed size we show that our test statistics based on summation can serve as good estimators of the mutual information moreover we suggest regularized tests that aggregate over all partition sizes and prove those are consistent too we provide polynomial time algorithms which are critical for computing the suggested test statistics efficiently we show that the power of the regularized tests is excellent compared to existing tests and almost as powerful as the tests based on the optimal yet unknown in practice partition size in simulations as well as on a real data example

the jackknife estimate of variance tukey s jackknife estimate of variance for a statistic s x 1 x 2 cdots x n which is a symmetric function of i i d random variables x i is investigated using an anova like decomposition of s it is shown that the jackknife variance estimate tends always to be biased upwards a theorem to this effect being proved for the natural jackknife estimate of operatorname var s x 1 x 2 cdots x n 1 based on x 1 x 2 cdots x n

f manifolds with flat structure and dubrovin s duality this work continues the study of f manifolds m first defined in hema int math res notices 6 1999 277 286 preprint math qa 9810132 and investigated in he frobenius manifolds and moduli spaces for singularities cambridge university press cambridge 2002 the notion of a compatible flat structure is introduced and it is shown that many constructions known for frobenius manifolds do not in fact require invariant metrics and can be developed for all such triples m in particular we extend and generalize recent dubrovin s duality du2 on almost duality for frobenius manifolds preprint math dg 0307374

estimating information rates with confidence intervals in neural spike trains information theory provides a natural set of statistics to quantify the amount of knowledge a neuron conveys about a stimulus a related work kennel shlens abarbanel chichilnisky 2005 demonstrated how to reliably estimate with a bayesian confidence interval the entropy rate from a discrete observed time series we extend this method to measure the rate of novel information that a neural spike train encodes about a stimulus the average and specific mutual information rates our estimator makes few assumptions about the underlying neural dynamics shows excellent performance in experimentally relevant regimes and uniquely provides confidence intervals bounding the range of information rates compatible with the observed spike train we validate this estimator with simulations of spike trains and highlight how stimulus parameters affect its convergence in bias and variance finally we apply these ideas to a recording from a guinea pig retinal ganglion cell and compare results to a simple linear decoder

poisson bracket on 1 forms and evolutionary partial differential equations we introduce a bracket on 1 forms defined on i e the infinite jet extension of the space of loops and prove that it satisfies the standard properties of a poisson bracket using this bracket we show that certain hierarchies appearing in the framework of f manifolds with a compatible flat connection m are hamiltonian in a generalized sense moreover we show that if a metric g compatible with is also invariant with respect to then this generalized hamiltonian setup reduces to the standard one

estimation of the information by an adaptive partitioning of the observation space we demonstrate that it is possible to approximate the mutual information arbitrarily closely in probability by calculating the relative frequencies on appropriate partitions and achieving conditional independence on the rectangles of which the partitions are made empirical results including a comparison with maximum likelihood estimators are presented

least conservative support and tolerance tubes this paper studies a distribution free estimator of the conditional support and tolerance intervals of a distributions underlying a set of paired independent and identically distributed i i d observations the key ingredients are a an appropriate notion of risk which measures what probability mass is not captured by the estimate b a uniform concentration inequality for the empirical risk based on a compression argument and c the derivation of a lower bound to the mutual information dictating how to maximize the informativeness of the estimator for this result we extend fano s inequality to the bivariate case

asymptotic bias in information estimates and the exponential bell polynomials we present a new derivation of the asymptotic correction for bias in the estimate of information from a finite sample the new derivation reveals a relationship between information estimates and a sequence of polynomials with combinatorial significance the exponential bell polynomials and helps to provide an understanding of the form and behavior of the asymptotic correction for bias

optimal estimation of wasserstein distance on a tree with an application to microbiome studies the weighted unifrac distance a plug in estimator of the wasserstein distance of read counts on a tree has been widely used to measure the microbial community difference in microbiome studies ou

edgeworth and saddle point approximations for random rectangular matrices abstract we present methods for obtaining asymptotic expansions for the probability density functions of sample means of independent and identically distributed random rectangular matrices the expansions are of the edgeworth saddle point and generalized edgeworth types an example of exact saddle point approximations and applications of the methods are illustrated

hermite and laguerre polynomials with complex matrix arguments this paper defines and discusses the complex hermite and laguerre polynomials associated with the complex matrix variate normal and wishart distributions respectively various properties of these polynomials are investigated including generating functions rodrigues formulae differential and integral versions and series expressions these polynomials are also discussed from the viewpoint of the multivariate complex meixner distributions we present applications in asymptotic distribution theory on the complex stiefel manifold the theory of complex zonal polynomials is of great use in the derivations

purely non local hamiltonian formalism kohno connections and systems in this paper we extend purely non local hamiltonian formalism to a class of riemannian f manifolds without assumptions on the semisimplicity of the product or on the flatness of the connection in the flat case we show that the recurrence relations for the principal hierarchy can be re interpreted using a local and purely non local hamiltonian operators and in this case they split into two lenard magri chains one involving the even terms the other involving the odd terms furthermore we give an elementary proof that the kohno property and the system condition are equivalent under suitable assumptions and we show how to associate a purely non local hamiltonian structure to any system including degenerate ones

a bayesian tutorial for data assimilation data assimilation is the process by which observational data are fused with scientific information the bayesian paradigm provides a coherent probabilistic approach for combining information and thus is an appropriate framework for data assimilation viewing data assimilation as a problem in bayesian statistics is not new however the field of bayesian statistics is rapidly evolving and new approaches for model construction and sampling have been utilized recently in a wide variety of disciplines to combine information this article includes a brief introduction to bayesian methods paying particular attention to data assimilation we review linkages to optimal interpolation kriging kalman filtering smoothing and variational analysis discussion is provided concerning monte carlo methods for implementing bayesian analysis including importance sampling particle filtering ensemble kalman filtering and markov chain monte carlo sampling finally hierarchical bayesian modeling is reviewed we indicate how this approach can be used to incorporate significant physically based prior information into statistical models thereby accounting for uncertainty the approach is illustrated in a simplified advection diffusion model

consistency of data driven histogram methods for density estimation and classification we present general sufficient conditions for the almost surel1 consistency of histogram density estimates based on data dependent partitions analogous conditions guarantee the almost sure risk consistency of histogram classification schemes based on data dependent partitions multivariate data is considered throughout in each case the desired consistency requires shrinking cells subexponential growth of a combinatorial complexity measure and sub linear growth of the number of cells it is not required that the cells of every partition be rectangles with sides paralles to the coordinate axis or that each cell contain a minimum number of points no assumptions are made concerning the common distribution of the training vectors we apply the results to establish the consistency of several known partitioning estimates including the kn spacing density estimate classifiers based on statistically equivalent blocks and classifiers based on multivariate clustering schemes

kernel methods for measuring independence we introduce two new functionals the constrained covariance and the kernel mutual information to measure the degree of independence of random variables these quantities are both based on the covariance between functions of the random variables in reproducing kernel hilbert spaces rkhss we prove that when the rkhss are universal both functionals are zero if and only if the random variables are pairwise independent we also show that the kernel mutual information is an upper bound near independence on the parzen window estimate of the mutual information analogous results apply for two correlation based dependence functionals introduced earlier we show the kernel canonical correlation and the kernel generalised variance to be independence measures for universal kernels and prove the latter to be an upper bound on the mutual information near independence the performance of the kernel dependence functionals in measuring independence is verified in the context of independent component analysis

demystifying fixed k nearest neighbor information estimators estimating mutual information from independent identically distributed samples drawn from an unknown joint density function is a basic statistical problem of broad interest with multitudinous applications the most popular estimator is the one proposed by kraskov stogbauer and grassberger ksg in 2004 and is nonparametric and based on the distances of each sample to its k mathrm th nearest neighboring sample where k is a fixed small integer despite of its widespread use part of scientific software packages theoretical properties of this estimator have been largely unexplored in this paper we demonstrate that the estimator is consistent and also identify an upper bound on the rate of convergence of the ell 2 error as a function of a number of samples we argue that the performance benefits of the ksg estimator stems from a curious correlation boosting effect and build on this intuition to modify the ksg estimator in novel ways to construct a superior estimator as a by product of our investigations we obtain nearly tight rates of convergence of the ell 2 error of the well known fixed k nearest neighbor estimator of differential entropy by kozachenko and leonenko

pac bayesian analysis of co clustering and beyond we derive pac bayesian generalization bounds for supervised and unsupervised learning models based on clustering such as co clustering matrix tri factorization graphical models graph clustering and pairwise clustering we begin with the analysis of co clustering which is a widely used approach to the analysis of data matrices we distinguish among two tasks in matrix data analysis discriminative prediction of the missing entries in data matrices and estimation of the joint probability distribution of row and column variables in co occurrence matrices we derive pac bayesian generalization bounds for the expected out of sample performance of co clustering based solutions for these two tasks the analysis yields regularization terms that were absent in the previous formulations of co clustering the bounds suggest that the expected performance of co clustering is governed by a trade off between its empirical performance and the mutual information preserved by the cluster variables on row and column ids we derive an iterative projection algorithm for finding a local optimum of this trade off for discriminative prediction tasks this algorithm achieved state of the art performance in the movielens collaborative filtering task our co clustering model can also be seen as matrix tri factorization and the results provide generalization bounds regularization terms and new algorithms for this form of matrix factorization the analysis of co clustering is extended to tree shaped graphical models which can be used to analyze high dimensional tensors according to the bounds the generalization abilities of tree shaped graphical models depend on a trade off between their empirical data fit and the mutual information that is propagated up the tree levels we also formulate weighted graph clustering as a prediction problem given a subset of edge weights we analyze the ability of graph clustering to predict the remaining edge weights the analysis of co clustering easily extends to this problem and suggests that graph clustering should optimize the trade off between empirical data fit and the mutual information that clusters preserve on graph nodes

density estimation on the spaces of symmetric and rectangular matrices abstract this paper develops the theory of density estimation on the space s m of all m m symmetric matrices and on the space r m p of all m p rectangular matrices we consider the two methods kernel density estimation and density estimation by orthogonal series and illustrate them with examples for the normal kernel density functions and for the hermite orthonormal bases with symmetric and rectangular matrix arguments on s m and r m p respectively the problem of estimating unknown joint density functions of multiple random matrices is also discussed

bayesian entropy estimation for countable discrete distributions we consider the problem of estimating shannon s entropy h from discrete data in cases where the number of possible symbols is unknown or even countably infinite the pitman yor process a generalization of dirichlet process provides a tractable prior distribution over the space of countably infinite discrete distributions and has found major applications in bayesian non parametric statistics and machine learning here we show that it provides a natural family of priors for bayesian entropy estimation due to the fact that moments of the induced posterior distribution over h can be computed analytically we derive formulas for the posterior mean bayes least squares estimate and variance under dirichlet and pitman yor process priors moreover we show that a fixed dirichlet or pitman yor process prior implies a narrow prior distribution over h meaning the prior strongly determines the entropy estimate in the under sampled regime we derive a family of continuous measures for mixing pitman yor processes to produce an approximately flat prior over h we show that the resulting pitman yor mixture pym entropy estimator is consistent for a large class of distributions finally we explore the theoretical properties of the resulting estimator and show that it performs well both in simulation and in application to real data

generalized noncentral hermite and laguerre polynomials in multiple matrices abstract we are concerned with the generalized noncentral hermite and laguerre polynomials in multiple matrices we define the generalized hermite polynomials h mn k r o y q a r in q m n rectangular matrix arguments y1 yq y q and r m m constant matrices a1 ar a r q r associated with the joint distribution of q independent m n rectangular matrix variate standard normal distributions and present various properties of theirs the discussion of the generalized hermite polynomials h m k r o x q a r in q m m symmetric matrix arguments x1 xq x q associated with the joint distribution of q independent m m symmetric matrix variate normal distributions introduced by chikuse is further facilitated the generalized noncentral hermite polynomials are defined as h mn k r o y n q a r and h m k r o x m q a r for q m n rectangular and m m symmetric matrices n q and m q respectively extending results of chikuse and davis to the noncentral case we define and discuss the generalized noncentral laguerre polynomials l u q k r o x q a r q in q m m noncentrality matrices 1 q q associated with the joint distribution of q independent noncentral wishart distributions many unsolved problems can now be resolved by introducing these noncentral polynomials in multiple matrices specifically both the differential and integral versions of the generalized multivariate rodrigues formulae are obtained for all of these polynomials they are of great use in multivariate distribution theory a simple example of applications is presented

universal estimation of erasure entropy erasure entropy rate differs from shannon s entropy rate in that the conditioning occurs with respect to both the past and the future as opposed to only the past or the future in this paper consistent universal algorithms for estimating erasure entropy rate are proposed based on the basic and extended context tree weighting ctw algorithms simulation results for those algorithms applied to markov sources tree sources and english texts are compared to those obtained by fixed order plug in estimators with different orders

estimating entropy on m bins given fewer than m samples consider a sequence p sub n of discrete probability measures supported on m sub n points and assume that we observe n independent and identically distributed i i d samples from each p sub n we demonstrate the existence of an estimator of the entropy h p sub n which is consistent even if the ratio n m sub n is bounded and as a corollary even if this ratio tends to zero albeit at a sufficiently slow rate

estimation of the entropy based on its polynomial representation estimating entropy from empirical samples of finite size is of central importance for information theory as well as the analysis of complex statistical systems yet this delicate task is marred by intrinsic statistical bias here we decompose the entropy function into a polynomial approximation function and a remainder function the approximation function is based on a taylor expansion of the logarithm given n observations we give an unbiased linear estimate of the first n power series terms based on counting sets of k coincidences for the remainder function we use nonlinear bayesian estimation with a nearly flat prior distribution on the entropy that was developed by nemenman shafee and bialek our simulations show that the combined entropy estimator has reduced bias in comparison to other available estimators

learning and generalization with the information bottleneck the information bottleneck is an information theoretic framework that finds concise representations for an input random variable that are as relevant as possible for an output random variable this framework has been used successfully in various supervised and unsupervised applications however its learning theoretic properties and justification remained unclear as it differs from standard learning models in several crucial aspects primarily its explicit reliance on the joint input output distribution in practice an empirical plug in estimate of the underlying distribution has been used so far without any finite sample performance guarantees in this paper we present several formal results that address these difficulties we prove several finite sample bounds which show that the information bottleneck can provide concise representations with good generalization based on smaller sample sizes than needed to estimate the underlying distribution the bounds are non uniform and adaptive to the complexity of the specific model chosen based on these results we also present a preliminary analysis on the possibility of analyzing the information bottleneck method as a learning algorithm in the familiar performance complexity tradeoff framework in addition we formally describe the connection between the information bottleneck and minimal sufficient statistics

spherical functions on the grassmann manifold and generalized jacobi polynomials part 1 abstract generalized jacobi polynomials constitute a complete system of orthogonal symmetric polynomials with respect to a multivariate beta measure this paper gives an explicit formula for these polynomials in terms of the author s invariant polynomials in two matrix arguments and derives the orthogonality property explicit formulas are also given for certain constants which appear in the theory of zonal polynomials in cases where the partitional parameters have not more than two nonzero parts zonal spherical functions on the grassmann manifold may be expressed in terms of generalized jacobi polynomials in part 2 of this paper it will be shown that these zonal spherical functions have an integral representation which generalizes the well known integrals for legendre and gegenbauer polynomials of even order in particular cases this integral representation may be used to construct the zonal and associated spherical functions in terms of univariate special functions

nonparametric triple collocation triple collocation has found widespread application in the hydrological sciences because it provides information about the errors in our measurements without requiring that we have any direct access to the true value of the variable being measured triple collocation derives variance covariance relationships between three or more independent measurement sources and an indirectly observed truth variable in the case where the measurement operators are additive we generalize that theory to arbitrary observation operators by deriving nonparametric analogues to the total error and total correlation statistics as integrations of divergences from conditional to marginal probability ratios the nonparametric solution to the full measurement problem is under determined and we therefore retrieve conservative bounds on the theoretical total nonparametric error and correlation statistics we examine the application of both linear and nonlinear triple collocation to synthetic examples and to a real data test case related to evaluating space borne soil moisture retrievals using sparse monitoring networks and dynamical process models

logarithmic frobenius structures and coxeter discriminants abstract we consider a class of solutions of the wdvv equation related to the special systems of covectors called systems and show that the corresponding logarithmic frobenius structures can be naturally restricted to any intersection of the corresponding hyperplanes for the coxeter arrangements the corresponding structures are shown to be almost dual in dubrovin s sense to the frobenius structures on the strata in the discriminants discussed by strachan for the classical coxeter root systems this leads to the families of systems from the earlier work by chalykh and veselov for the exceptional coxeter root systems we give the complete list of the corresponding systems we present also some new families of systems which cannot be obtained in such a way from the coxeter root systems

new families of skewed higher order kernel estimators to solve the bss ica problem for multimodal sources mixtures this letter suggests two new types of asymmetrical higher order kernels hok that are generated using the orthogonal polynomials laguerre positive or right skew and bessel negative or left skew these skewed hok are implemented in the blind source separation independent component analysis bss ica algorithm the tests for these proposed hok are accomplished using three scenarios to simulate a real environment using actual sound sources an environment of mixtures of multimodal fast changing probability density function pdf sources that represent a challenge to the symmetrical hok and an environment of an adverse case near gaussian the separation is performed by minimizing the mutual information mi among the mixed sources the performance of the skewed kernels is compared to the performance of the standard kernels such as epanechnikov bisquare trisquare and gaussian and the performance of the symmetrical hok generated using the polynomials chebyshev1 chebyshev2 gegenbauer jacobi and legendre to the tenth order the gaussian hok are generated using the hermite polynomial and the wand and schucany procedure the comparison among the 96 kernels is based on the average intersymbol interference ratio aisir and the time needed to complete the separation in terms of aisir the skewed kernels performance is better than that of the standard kernels and rivals most of the symmetrical kernels performance the importance of these new skewed hok is manifested in the environment of the multimodal pdf mixtures in such an environment the skewed hok come in first place compared with the symmetrical hok these new families can substitute for symmetrical hoks in such applications

nonparametric estimation of kullback leibler divergence in this letter we introduce an estimator of kullback leibler divergence based on two independent samples we show that on any finite alphabet this estimator has an exponentially decaying bias and that it is consistent and asymptotically normal to explain the importance of this estimator we provide a thorough analysis of the more standard plug in estimator we show that it is consistent and asymptotically normal but with an infinite bias moreover if we modify the plug in estimator to remove the rare events that cause the bias to become infinite the bias still decays at a rate no faster than further we extend our results to estimating the symmetrized kullback leibler divergence we conclude by providing simulation results which show that the asymptotic properties of these estimators hold even for relatively small sample sizes

binless strategies for estimation of information from neural data we present an approach to estimate information carried by experimentally observed neural spike trains elicited by known stimuli this approach makes use of an embedding of the observed spike trains into a set of vector spaces and entropy estimates based on the nearest neighbor euclidean distances within these vector spaces l f kozachenko and n n leonenko probl peredachi inf 23 9 1987 using numerical examples we show that this approach can be dramatically more efficient than standard bin based approaches such as the direct method s p strong r koberle r r de ruyter van steveninck and w bialek phys rev lett 80 197 1998 for amounts of data typically available from laboratory experiments

minimax rates of entropy estimation on large alphabets via best polynomial approximation consider the problem of estimating the shannon entropy of a distribution over k elements from n independent samples we show that the minimax mean square error is within the universal multiplicative constant factors of big k n log k big 2 log 2 k n if n exceeds a constant factor of k log k otherwise there exists no consistent estimator this refines the recent result of valiant and valiant that the minimal sample size for consistent entropy estimation scales according to theta k log k the apparatus of the best polynomial approximation plays a key role in both the construction of optimal estimators and by a duality argument the minimax lower bound

measuring dependence powerfully and equitably given a high dimensional data set we often wish to find the strongest relationships within it a common strategy is to evaluate a measure of dependence on every variable pair and retain the highest scoring pairs for follow up this strategy works well if the statistic used a has good power to detect non trivial relationships and b is equitable meaning that for some measure of noise it assigns similar scores to equally noisy relationships regardless of relationship type e g linear exponential periodic in this paper we define and theoretically characterize two new statistics that together yield an efficient approach for obtaining both power and equitability to do this we first introduce a new population measure of dependence and show three equivalent ways that it can be viewed including as a canonical smoothing of mutual information we then introduce an efficiently computable consistent estimator of our population measure of dependence and we empirically establish its equitability on a large class of noisy functional relationships this new statistic has better bias variance properties and better runtime complexity than a previous heuristic approach next we derive a second related statistic whose computation is a trivial side product of our algorithm and whose goal is powerful independence testing rather than equitability we prove that this statistic yields a consistent independence test and show in simulations that the test has good power against independence taken together our results suggest that these two statistics are a valuable pair of tools for exploratory data analysis

estimating the entropy rate of spike trains via lempel ziv complexity normalized lempel ziv complexity which measures the generation rate of new patterns along a digital sequence is closely related to such important source properties as entropy and compression ratio but in contrast to these it is a property of individual sequences in this article we propose to exploit this concept to estimate or at least to bound from below the entropy of neural discharges spike trains the main advantages of this method include fast convergence of the estimator as supported by numerical simulation and the fact that there is no need to know the probability law of the process generating the signal furthermore we present numerical and experimental comparisons of the new method against the standard method based on word frequencies providing evidence that this new approach is an alternative entropy estimator for binned spike trains

analyzing neural responses to natural signals maximally informative dimensions we propose a method that allows for a rigorous statistical analysis of neural responses to natural stimuli that are nongaussian and exhibit strong correlations we have in mind a model in which neurons are selective for a small number of stimulus dimensions out of a high dimensional stimulus space but within this subspace the responses can be arbitrarily nonlinear existing analysis methods are based on correlation functions between stimuli and responses but these methods are guaranteed to work only in the case of gaussian stimulus ensembles as an alternative to correlation functions we maximize the mutual information between the neural responses and projections of the stimulus onto low dimensional subspaces the procedure can be done iteratively by increasing the dimensionality of this subspace those dimensions that allow the recovery of all of the information between spikes and the full unprojected stimuli describe the relevant subspace if the dimensionality of the relevant subspace indeed is small it becomes feasible to map the neuron s input output function even under fully natural stimulus conditions these ideas are illustrated in simulations on model visual and auditory neurons responding to natural scenes and sounds respectively

an efron stein inequality for nonsymmetric statistics on etablit l analogue d un resultat d efron et stein 1981 que l on demontre a l aide d une technique d espace de hilbert introduite par vitale 1984

matrix models for beta ensembles this paper constructs tridiagonal random matrix models for general 0 hermite gaussian and laguerre wishart ensembles these generalize the well known gaussian and wishart models for 1 2 4 furthermore in the cases of the laguerre ensembles we eliminate the exponent quantization present in the previously known models we further discuss applications for the new matrix models and present some open problems

asymptotic normality of an entropy estimator with exponentially decaying bias this paper establishes the asymptotic normality of an entropy estimator with an exponentially decaying bias on any finite alphabet furthermore it is shown that the nonparametric estimator is asymptotically efficient

a large deviation analysis of the maximum likelihood learning of markov tree structures the problem of maximum likelihood ml estimation of discrete tree structured distributions is considered chow and liu established that ml estimation reduces to the construction of a maximum weight spanning tree using the empirical mutual information quantities as the edge weights using the theory of large deviations we analyze the exponent associated with the error probability of the event that the ml estimate of the markov tree structure differs from the true tree structure given a set of independently drawn samples by exploiting the fact that the output of ml estimation is a tree we establish that the error exponent is equal to the exponential rate of decay of a single dominant crossover event we prove that in this dominant crossover event a non neighbor node pair replaces a true edge of the distribution that is along the path of edges in the true tree graph connecting the nodes in the non neighbor pair using ideas from euclidean information theory we then analyze the scenario of ml estimation in the very noisy learning regime and show that the error exponent can be approximated as a ratio which is interpreted as the signal to noise ratio snr for learning tree distributions we show via numerical experiments that in this regime our snr approximation is accurate

efficient markov chain monte carlo methods for decoding neural spike trains stimulus reconstruction or decoding methods provide an important tool for understanding how sensory and motor information is represented in neural activity we discuss bayesian decoding methods based on an encoding generalized linear model glm that accurately describes how stimuli are transformed into the spike trains of a group of neurons the form of the glm likelihood ensures that the posterior distribution over the stimuli that caused an observed set of spike trains is log concave so long as the prior is this allows the maximum a posteriori map stimulus estimate to be obtained using efficient optimization algorithms unfortunately the map estimate can have a relatively large average error when the posterior is highly nongaussian here we compare several markov chain monte carlo mcmc algorithms that allow for the calculation of general bayesian estimators involving posterior expectations conditional on model parameters an efficient version of the hybrid monte carlo hmc algorithm was significantly superior to other mcmc methods for gaussian priors when the prior distribution has sharp edges and corners on the other hand the hit and run algorithm performed better than other mcmc methods using these algorithms we show that for this latter class of priors the posterior mean estimate can have a considerably lower average error than map whereas for gaussian priors the two estimators have roughly equal efficiency we also address the application of mcmc methods for extracting nonmarginal properties of the posterior distribution for example by using mcmc to calculate the mutual information between the stimulus and response we verify the validity of a computationally efficient laplace approximation to this quantity for gaussian priors in a wide range of model parameters this makes direct model based computation of the mutual information tractable even in the case of large observed neural populations where methods based on binning the spike train fail finally we consider the effect of uncertainty in the glm parameters on the posterior estimators

second order noiseless source coding theorems shannon s celebrated source coding theorem can be viewed as a one sided law of large numbers we formulate second order noiseless source coding theorems for the deviation of the codeword lengths from the entropy for a class of sources that includes markov chains we prove a one sided central limit theorem and a law of the iterated logarithm

tight data robust bounds to mutual information combining shuffling and model selection techniques the estimation of the information carried by spike times is crucial for a quantitative understanding of brain function but it is difficult because of an upward bias due to limited experimental sampling we present new progress based on two basic insights on reducing the bias problem first we show that by means of a careful application of data shuffling techniques it is possible to cancel almost entirely the bias of the noise entropy the most biased part of information this procedure provides a new information estimator that is much less biased than the standard direct one and has similar variance second we use a nonparametric test to determine whether all the information encoded by the spike train can be decoded assuming a low dimensional response model if this is the case the complexity of response space can be fully captured by a small number of easily sampled parameters combining these two different procedures we obtain a new class of precise estimators of information quantities which can provide data robust upper and lower bounds to the mutual information these bounds are tight even when the number of trials per stimulus available is one order of magnitude smaller than the number of possible responses the effectiveness and the usefulness of the methods are tested through applications to simulated data and recordings from somatosensory cortex this application shows that even in the presence of strong correlations our methods constrain precisely the amount of information encoded by real spike trains recorded in vivo

universal entropy estimation via block sorting in this correspondence we present a new universal entropy estimator for stationary ergodic sources prove almost sure convergence and establish an upper bound on the convergence rate for finite alphabet finite memory sources the algorithm is motivated by data compression using the burrows wheeler block sorting transform bwt by exploiting the property that the bwt output sequence is close to a piecewise stationary memoryless source we can segment the output sequence and estimate probabilities in each segment experimental results show that our algorithm outperforms lempel ziv lz string matching based algorithms

performance assessment of mimo bicm demodulators based on mutual information we provide a comprehensive performance comparison of soft output and hard output demodulators in the context of non iterative multiple input multiple output bit interleaved coded modulation mimo bicm coded bit error rate ber widely used in literature for demodulator comparison has the drawback of depending strongly on the error correcting code being used this motivates us to propose the mutual information of the equivalent modulation channel comprising modulator wireless channel and demodulator as a code independent performance measure we present extensive numerical results for spatially independent identically distributed i i d ergodic and quasi static fading channels under perfect and imperfect channel state information these results reveal that the performance ranking of mimo demodulators is rate dependent and provide new insights regarding mimo bicm system design i e the choice of antenna configuration symbol constellation and demodulator for a given target rate

estimating the unseen improved estimators for entropy and other properties we show that a class of statistical properties of distributions which includes such practically relevant properties as entropy the number of distinct elements and distance metrics between pairs of distributions can be estimated given a sublinear sized sample specifically given a sample consisting of independent draws from any distribution over at most k distinct elements these properties can be estimated accurately using a sample of size o k log k for these estimation tasks this performance is optimal to constant factors complementing these theoretical results we also demonstrate that our estimators perform exceptionally well in practice for a variety of estimation tasks on a variety of natural distributions for a wide range of parameters the key step in our approach is to first use the sample to characterize the unseen portion of the distribution effectively reconstructing this portion of the distribution as accurately as if one had a logarithmic factor larger sample this goes beyond such tools as the good turing frequency estimation scheme which estimates the total probability mass of the unobserved portion of the distribution we seek to estimate the shape of the unobserved portion of the distribution this work can be seen as introducing a robust general and theoretically principled framework that for many practical applications essentially amplifies the sample size by a logarithmic factor we expect that it may be fruitfully used as a component within larger machine learning and statistical analysis systems

equitability mutual information and the maximal information coefficient how should one quantify the strength of association between two random variables without bias for relationships of a specific form despite its conceptual simplicity this notion of statistical equitability has yet to receive a definitive mathematical formalization here we argue that equitability is properly formalized by a self consistency condition closely related to data processing inequality mutual information a fundamental quantity in information theory is shown to satisfy this equitability criterion these findings are at odds with the recent work of reshef et al reshef dn et al 2011 science 334 6062 1518 1524 which proposed an alternative definition of equitability and introduced a new statistic the maximal information coefficient mic said to satisfy equitability in contradistinction to mutual information these conclusions however were supported only with limited simulation evidence not with mathematical arguments upon revisiting these claims we prove that the mathematical definition of equitability proposed by reshef et al cannot be satisfied by any nontrivial dependence measure we also identify artifacts in the reported simulation evidence when these artifacts are removed estimates of mutual information are found to be more equitable than estimates of mic mutual information is also observed to have consistently higher statistical power than mic we conclude that estimating mutual information provides a natural and often practical way to equitably quantify statistical associations in large datasets

the upward bias in measures of information derived from limited data samples extracting information measures from limited experimental samples such as those normally available when using data recorded in vivo from mammalian cortical neurons is known to be plagued by a systematic error which tends to bias the estimate upward we calculate here the average of the bias under certain conditions as an asymptotic expansion in the inverse of the size of the data sample the result agrees with numerical simulations and is applicable as an additive correction term to measurements obtained under such conditions moreover we discuss the implications for measurements obtained through other usual procedures

estimation of entropy and mutual information we present some new results on the nonparametric estimation of entropy and mutual information first we use an exact local expansion of the entropy function to prove almost sure consistency and central limit theorems for three of the most commonly used discretized information estimators the setup is related to grenander s method of sieves and places no assumptions on the underlying probability measure generating the data second we prove a converse to these consistency theorems demonstrating that a misapplication of the most common estimation techniques leads to an arbitrarily poor estimate of the true information even given unlimited data this inconsistency theorem leads to an analytical approximation of the bias valid in surprisingly small sample regimes and more accurate than the usual 1 n formula of miller and madow over a large region of parameter space the two most practical implications of these results are negative 1 information estimates in a certain data regime are likely contaminated by bias even if bias corrected estimators are used and 2 confidence intervals calculated by standard techniques drastically underestimate the error of the most common estimation methods finally we note a very useful connection between the bias of entropy estimators and a certain polynomial approximation problem by casting bias calculation problems in this approximation theory framework we obtain the best possible generalization of known asymptotic bias results more interesting this framework leads to an estimator with some nice properties the estimator comes equipped with rigorous bounds on the maximum error over all possible underlying probability distributions and this maximum error turns out to be surprisingly small we demonstrate the application of this new estimator on both real and simulated data

properties of hermite and laguerre polynomials in matrix argument and their applications abstract this paper is concerned with the hermite polynomials in symmetric and rectangular matrix argument and the laguerre polynomials in symmetric matrix argument which are associated with the symmetric and rectangular matrix variate normal distributions and the wishart distribution respectively we derive the multivariate rodrigues formula a differential equation for each of the two kinds of hermite polynomials with respect to the associating p d f and the mutivariate differential operator i e the zonal polynomial in the matrix of differential operators which plays the role of the ordinary differential the multivariate rodrigues formulae are useful in asymptotic multivariate distribution theory for matrix variates we consider the general forms of the series edgeworth expansions for distributions of both symmetric and rectangular random matrices and then give an asymptotic edgeworth expansion for the distribution of a scale mixture matrix as a simple example as for the laguerre polynomials alternative forms of differential equations are derived we also present series expansions and recurrence relationships for the hermite and laguerre polynomials and some limit properties of the laguerre polynomials

estimating epistemic and aleatory uncertainties during hydrologic modeling an information theoretic approach 1 with growing interest in understanding the magnitudes and sources of uncertainty in hydrological modeling the difficult problem of characterizing model structure adequacy is now attracting considerable attention here we examine this problem via a model structureindependent approach based in information theory in particular we a discuss how to assess and compute the information content in multivariate hydrological data b present practical methods for quantifying the uncertainty and shared information in data while accounting for heteroscedasticity c show how these tools can be used to estimate the best achievable predictive performance of a model for a system given the available data and d show how model adequacy can be characterized in terms of the magnitude and nature of its aleatory uncertainty that cannot be diminished and is resolvable only up to specification of its density and its epistemic uncertainty that can in principle be suitably resolved by improving the model an illustrative modeling example is provided using catchment scale data from three river basins the leaf and chunky river basins in the united states and the chuzhou basin in china our analysis shows that the aleatory uncertainty associated with making catchment simulations using this data set is significant 50 further estimated epistemic uncertainties of the hymod sac sma and xinanjiang model hypotheses indicate that considerable room for model structural improvements remain

a normal law for the plug in estimator of entropy this paper establishes a sufficient condition for the asymptotic normality of the plug in estimator of shannon s entropy defined on a countable alphabet the sufficient condition covers a range of cases with countably infinite alphabets for which no normality results were previously known

an approach to quantifying the efficiency of a bayesian filter 1 data assimilation is the bayesian conditioning of uncertain model simulations on observations to reduce uncertainty about model states in practice it is common to make simplifying assumptions about the prior and posterior state distributions and to employ approximations of the likelihood function which can reduce the efficiency of the filter we propose metrics that quantify how much of the uncertainty in a bayesian posterior state distribution is due to i the observation operator ii observation error and iii approximations of bayes law our approach uses discrete shannon entropy to quantify uncertainty and we define the utility of an observation for reducing uncertainty about a model state as the ratio of the mutual information between the state and observation to the entropy of the state prior these metrics make it possible to analyze the efficiency of a proposed observation system and data assimilation strategy and provide a way to examine the propagation of information through the dynamic system model we demonstrate the procedure on the problem of estimating profile soil moisture from observations at the surface top 5 cm the results show that when synthetic observations of 5 cm soil moisture are assimilated into a three layer model of soil hydrology the ensemble kalman filter does not use all of the information available in observations

generalized hermite and laguerre polynomials in multiple symmetric matrix arguments and their applications abstract we investigate properties of the generalized hermite h m r x q b r and laguerre l u q r x q b r polynomials in q m m symmetric matrix arguments x 1 x q x q and r m m constant matrices b 1 b r b r and with q parameters u 1 u q u q q r and consider their applications in multivariate distribution theory the results extend those for the case of one symmetric matrix argument obtained by chikuse we derive the generalized multivariate rodrigues formula a differential equation for the polynomial h m r with respect to the associated joint multiple symmetric matrix variate normal p d f and the generalized multivariate differential operator i e the invariant polynomial in the multiple matrices of differential operators which plays the role of the ordinary differential the generalized multivariate rodrigues formula and the associated polynomials h m r are utilized to present the general form of the series edgeworth expansion for a distribution of multiple symmetric random matrices and then give a simple example alternative forms of differential equations are derived for the polynomial l u q r series expansions and recurrence relationships or some related results are obtained for the polynomials h m r and l u q r

bias analysis in entropy estimation we consider the problem of finite sample corrections for entropy estimation new estimates of the shannon entropy are proposed and their systematic error the bias is computed analytically we find that our results cover correction formulae of current entropy estimates recently discussed in the literature the trade off between bias reduction and the increase of the corresponding statistical error is analysed

geometrizing rates of convergence ii consider estimating a functional t f of an unknown distribution f in mathbf f from data x 1 cdots x n i i d f let omega varepsilon denote the modulus of continuity of the functional t over mathbf f computed with respect to hellinger distance for well behaved loss functions l t we show that inf t n sup mathbf f e fl t n t f is equivalent to l omega n 1 2 to within constants whenever t is linear and mathbf f is convex the same conclusion holds in three nonlinear cases estimating the rate of decay of a density estimating the mode and robust nonparametric regression we study the difficulty of testing between the composite infinite dimensional hypotheses h 0 t f leq t and h 1 t f geq t delta our results hold in the cases studied because the difficulty of the full infinite dimensional composite testing problem is comparable to the difficulty of the hardest simple two point testing subproblem

least squares independent component analysis accurately evaluating statistical independence among random variables is a key element of independent component analysis ica in this letter we employ a squared loss variant of mutual information as an independence measure and give its estimation method our basic idea is to estimate the ratio of probability densities directly without going through density estimation thereby avoiding the difficult task of density estimation in this density ratio approach a natural cross validation procedure is available for hyperparameter selection thus all tuning parameters such as the kernel width or the regularization parameter can be objectively optimized this is an advantage over recently developed kernel based independence measures and is a highly useful property in unsupervised learning problems such as ica based on this novel independence measure we develop an ica algorithm named least squares independent component analysis

multivariate meixner classes of invariant distributions abstract we define multivariate meixner classes of invariant distributions of random matrices as those whose generating functions for the associated orthogonal polynomials are of certain special integral or summation forms generalizing the univariate meixner classes of distributions which were first characterized by meixner 21 characterization theorems and properties of these multivariate meixner classes are established the zonal polynomials the extended invariant polynomials with matrix arguments and their related results in multivariate distribution theory are utilized in the discussion

finite sample concentration of the multinomial in relative entropy we show that the moment generating function of the kullback leibler divergence relative entropy between the empirical distribution of n independent samples from a distribution p over a finite alphabet of size k i e a multinomial distribution and p itself is no more than that of a gamma distribution with shape k 1 and rate n the resulting exponential concentration inequality becomes meaningful less than 1 when the divergence varepsilon is larger than k 1 n whereas the standard method of types bound requires varepsilon frac 1 n cdot log binom n k 1 k 1 geq k 1 n cdot log 1 n k 1 thus saving a factor of order log n k in the standard regime of parameters where n gg k as a consequence we also obtain finite sample bounds on all the moments of the empirical divergence equivalently the discrete likelihood ratio statistic which are within constant factors depending on the moment of their asymptotic values our proof proceeds via a simple reduction to the case k 2 of a binary alphabet i e a binomial distribution and has the property that improvements in the case of k 2 directly translate to improvements for general k in particular we conjecture a bound on the binomial moment generating function that would almost close the quadratic gap between our finite sample bound and the asymptotic moment generating function bound from wilks theorem which does not hold for finite samples