estimating functions of probability distributions from a finite set of samples this paper addresses the problem of estimating a function of a probability distribution from a finite set of samples of that distribution a bayesian analysis of this problem is presented the optimal properties of the bayes estimators are discussed and as an example of the formalism closed form expressions for the bayes estimators for the moments of the shannon entropy function are derived then numerical results are presented that compare the bayes estimator to the frequency counts estimator for the shannon entropy we also present the closed form estimators all derived elsewhere for the mutual information mathrm ensuremath chi 2 covariance and some other statistics c 1995 the american physical society

the hellinger correlation in this paper the defining properties of any valid measure of the dependence between two continuous random variables are revisited and complemented with two original ones shown to imply other usu

consistent distribution free k sample and independence tests for univariate random variables a popular approach for testing if two univariate random variables are statistically independent consists of partitioning the sample space into bins and evaluating a test statistic on the binned data the partition size matters and the optimal partition size is data dependent while for detecting simple relationships coarse partitions may be best for detecting complex relationships a great gain in power can be achieved by considering finer partitions we suggest novel consistent distribution free tests that are based on summation or maximization aggregation of scores over all partitions of a fixed size we show that our test statistics based on summation can serve as good estimators of the mutual information moreover we suggest regularized tests that aggregate over all partition sizes and prove those are consistent too we provide polynomial time algorithms which are critical for computing the suggested test statistics efficiently we show that the power of the regularized tests is excellent compared to existing tests and almost as powerful as the tests based on the optimal yet unknown in practice partition size in simulations as well as on a real data example

remarks on j langer and d a singer decomposition theorem for diffeomorphisms of the circle we give a simple proof that allc4 diffeomorphisms of the torus can be factorized into a finite number of diffeomorphisms commuting with reflection

estimation of mutual information using kernel density estimators mutual information is useful for investigating the dependence between two experimental time series it is often used to establish an appropriate time delay in phase portrait reconstruction from time series data a histogram based approach has been used so far to estimate the probabilities it is shown here that kernel density estimation of the probability density functions needed in estimating the average mutual information across two coordinates can be more effective than the histogram method of fraser and swinney phys rev a 33 1134 1986

demystifying fixed k nearest neighbor information estimators estimating mutual information from independent identically distributed samples drawn from an unknown joint density function is a basic statistical problem of broad interest with multitudinous applications the most popular estimator is the one proposed by kraskov stogbauer and grassberger ksg in 2004 and is nonparametric and based on the distances of each sample to its k mathrm th nearest neighboring sample where k is a fixed small integer despite of its widespread use part of scientific software packages theoretical properties of this estimator have been largely unexplored in this paper we demonstrate that the estimator is consistent and also identify an upper bound on the rate of convergence of the ell 2 error as a function of a number of samples we argue that the performance benefits of the ksg estimator stems from a curious correlation boosting effect and build on this intuition to modify the ksg estimator in novel ways to construct a superior estimator as a by product of our investigations we obtain nearly tight rates of convergence of the ell 2 error of the well known fixed k nearest neighbor estimator of differential entropy by kozachenko and leonenko

estimation of the entropy based on its polynomial representation estimating entropy from empirical samples of finite size is of central importance for information theory as well as the analysis of complex statistical systems yet this delicate task is marred by intrinsic statistical bias here we decompose the entropy function into a polynomial approximation function and a remainder function the approximation function is based on a taylor expansion of the logarithm given n observations we give an unbiased linear estimate of the first n power series terms based on counting sets of k coincidences for the remainder function we use nonlinear bayesian estimation with a nearly flat prior distribution on the entropy that was developed by nemenman shafee and bialek our simulations show that the combined entropy estimator has reduced bias in comparison to other available estimators

estimating mutual information we present two classes of improved estimators for mutual information m x y from samples of random points distributed according to some joint probability density mu x y in contrast to conventional estimators based on binnings they are based on entropy estimates from k nearest neighbor distances this means that they are data efficient with k 1 we resolve structures down to the smallest possible scales adaptive the resolution is higher where data are more numerous and have minimal bias indeed the bias of the underlying entropy estimates is mainly due to nonuniformity of the density at the smallest resolved scale giving typically systematic errors which scale as functions of k n for n points numerically we find that both families become exact for independent distributions i e the estimator m x y vanishes up to statistical fluctuations if mu x y mu x mu y this holds for all tested marginal distributions and for all dimensions of x and y in addition we give estimators for redundancies between more than two random variables we compare our algorithms in detail with existing algorithms finally we demonstrate the usefulness of our estimators for assessing the actual independence of components obtained from independent component analysis ica for improving ica and for estimating the reliability of blind source separation

relative performance of mutual information estimation methods for quantifying the dependence among short and noisy data commonly used dependence measures such as linear correlation cross correlogram or kendall s tau cannot capture the complete dependence structure in data unless the structure is restricted to linear periodic or monotonic mutual information mi has been frequently utilized for capturing the complete dependence structure including nonlinear dependence recently several methods have been proposed for the mi estimation such as kernel density estimators kde k nearest neighbors knn edgeworth approximation of differential entropy and adaptive partitioning of the xy plane however outstanding gaps in the current literature have precluded the ability to effectively automate these methods which in turn have caused limited adoptions by the application communities this study attempts to address a key gap in the literature specifically the evaluation of the above methods to choose the best method particularly in terms of their robustness for short and noisy data based on comparisons with the theoretical mi estimates which can be computed analytically as well with linear correlation and kendall s tau here we consider smaller data sizes such as 50 100 and 1 000 where this study considers 50 and 100 data points as very short and 1 000 as short we consider a broader class of functions specifically linear quadratic periodicmore and chaotic contaminated with artificial noise with varying noise to signal ratios the case studies presented here are motivated by domain consideration in the earth sciences where the data are short and noisy our results indicate kde as the best choice for very short data at relatively high noise to signal levels whereas the performance of knn is the best for short data at relatively low noise levels as well as for short data consistently across noise levels in addition the optimal smoothing parameter of a gaussian kernel appears to be the best choice for kde while three nearest neighbors appear optimal for knn thus in situations where the approximate data sizes are known in advance and exploratory data analysis and or domain knowledge can be used to provide a priori insights on the noise to signal ratios the results in the paper point to a way forward for automating the process of mi estimation less

copula index for detecting dependence and monotonicity between stochastic signals abstract this paper introduces a nonparametric copula based index for detecting the strength and monotonicity structure of linear and nonlinear statistical dependence between pairs of random variables or stochastic signals our index termed copula index for detecting dependence and monotonicity cim satisfies several desirable properties of measures of association including most of renyi s properties the data processing inequality dpi and consequently self equitability synthetic data simulations reveal that the statistical power of cim compares favorably to other state of the art measures of association that are proven to satisfy the dpi simulation results with real world data reveal cim s unique ability to detect the monotonicity structure among stochastic signals to find interesting dependencies in large datasets additionally simulations show that cim compares favorably to estimators of mutual information when discovering markov network structure

measuring dependence powerfully and equitably given a high dimensional data set we often wish to find the strongest relationships within it a common strategy is to evaluate a measure of dependence on every variable pair and retain the highest scoring pairs for follow up this strategy works well if the statistic used a has good power to detect non trivial relationships and b is equitable meaning that for some measure of noise it assigns similar scores to equally noisy relationships regardless of relationship type e g linear exponential periodic in this paper we define and theoretically characterize two new statistics that together yield an efficient approach for obtaining both power and equitability to do this we first introduce a new population measure of dependence and show three equivalent ways that it can be viewed including as a canonical smoothing of mutual information we then introduce an efficiently computable consistent estimator of our population measure of dependence and we empirically establish its equitability on a large class of noisy functional relationships this new statistic has better bias variance properties and better runtime complexity than a previous heuristic approach next we derive a second related statistic whose computation is a trivial side product of our algorithm and whose goal is powerful independence testing rather than equitability we prove that this statistic yields a consistent independence test and show in simulations that the test has good power against independence taken together our results suggest that these two statistics are a valuable pair of tools for exploratory data analysis

symbolic dynamics and the discrete variational principle we show how to construct symbolic dynamics for the class of 2d dimensional twist mappings generated by piecewise strictly convex concave generating functions the method is constructive and gives an efficient way to find all periodic orbits of these high dimensional symplectic mappings it is illustrated with the cardioid and the stadium billiard

analyzing neural responses to natural signals maximally informative dimensions we propose a method that allows for a rigorous statistical analysis of neural responses to natural stimuli that are nongaussian and exhibit strong correlations we have in mind a model in which neurons are selective for a small number of stimulus dimensions out of a high dimensional stimulus space but within this subspace the responses can be arbitrarily nonlinear existing analysis methods are based on correlation functions between stimuli and responses but these methods are guaranteed to work only in the case of gaussian stimulus ensembles as an alternative to correlation functions we maximize the mutual information between the neural responses and projections of the stimulus onto low dimensional subspaces the procedure can be done iteratively by increasing the dimensionality of this subspace those dimensions that allow the recovery of all of the information between spikes and the full unprojected stimuli describe the relevant subspace if the dimensionality of the relevant subspace indeed is small it becomes feasible to map the neuron s input output function even under fully natural stimulus conditions these ideas are illustrated in simulations on model visual and auditory neurons responding to natural scenes and sounds respectively

parametric inference in the large data limit using maximally informative models motivated by data rich experiments in transcriptional regulation and sensory neuroscience we consider the following general problem in statistical inference when exposed to a high dimensional signal s a system of interest computes a representation r of that signal which is then observed through a noisy measurement m from a large number of signals and measurements we wish to infer the filter that maps s to r however the standard method for solving such problems likelihood based inference requires perfect a priori knowledge of the noise function mapping r to m in practice such noise functions are usually known only approximately if at all and using an incorrect noise function will typically bias the inferred filter here we show that in the large data limit this need for a precharacterized noise function can be circumvented by searching for filters that instead maximize the mutual information i m r between observed measurements and predicted representations moreover if the correct filter lies within the space of filters being explored maximizing mutual information becomes equivalent to simultaneously maximizing every dependence measure that satisfies the data processing inequality it is important to note that maximizing mutual information will typically leave a small number of directions in parameter space unconstrained we term these directions diffeomorphic modes and present an equation that allows these modes to be derived systematically the presence of diffeomorphic modes reflects a fundamental and nontrivial substructure within parameter space one that is obscured by standard likelihood based inference

partially monotonic decision trees abstract in multicriteria decision tasks certain features are linearly ordered according to the decision and are called criteria whereas others called regular attributes are not in practice regular attributes and criteria coexist in most classification tasks in this paper we propose a rank inconsistent rate that distinguishes attributes from criteria furthermore it represents the directions of the monotonic relationships between criteria and decisions we design a partially monotonic decision tree algorithm to extract decision rules for partially monotonic classification tasks experimental results show that the proposed algorithm is effective and efficient

functional associations at global brain level during perception of an auditory illusion by applying maximal information coefficient abstract maximal information coefficient mic is a recently introduced information theoretic measure of functional association with a promising potential of application to high dimensional complex data sets here we applied mic to reveal the nature of the functional associations between different brain regions during the perception of binaural beat bb bb is an auditory illusion occurring when two sinusoidal tones of slightly different frequency are presented separately to each ear and an illusory beat at the different frequency is perceived we recorded sixty four channels eeg from two groups of participants musicians and non musicians during the presentation of bb and systematically varied the frequency difference from 1 hz to 48 hz participants were also presented non binuaral beat nbb stimuli in which same frequencies were presented to both ears across groups as compared to nbb i bb conditions produced the most robust changes in the mic values at the whole brain level when the frequency differences were in the classical alpha range 8 12 hz and ii the number of electrode pairs showing nonlinear associations decreased gradually with increasing frequency difference between groups significant effects were found for bbs in the broad gamma frequency range 34 48 hz but such effects were not observed between groups during nbb altogether these results revealed the nature of functional associations at the whole brain level during the binaural beat perception and demonstrated the usefulness of mic in characterizing interregional neural dependencies

statistical validation of mutual information calculations comparison of alternative numerical algorithms given two time series x and y their mutual information i x y i y x is the average number of bits of x that can be predicted by measuring y and vice versa in the analysis of observational data calculation of mutual information occurs in three contexts identification of nonlinear correlation determination of an optimal sampling interval particularly when embedding data and in the investigation of causal relationships with directed mutual information in this contribution a minimum description length argument is used to determine the optimal number of elements to use when characterizing the distributions of x and y however even when using partitions of the x and y axis indicated by minimum description length mutual information calculations performed with a uniform partition of the xy plane can give misleading results this motivated the construction of an algorithm for calculating mutual information that uses an adaptive partition this algorithm also incorporates an explicit test of the statistical independence of x and y in a calculation that returns an assessment of the corresponding null hypothesis the previously published fraser swinney algorithm for calculating mutual information includes a sophisticated procedure for local adaptive control of the partitioning process when the fraser and swinney algorithm and the algorithm constructed here are compared they give very similar numerical results less than 4 difference in a typical application detailed comparisons are possible when x and y are correlated jointly gaussian distributed because an analytic expression for i x y can be derived for that case based on these tests three conclusions can be drawn first the algorithm constructed here has an advantage over the fraser swinney algorithm in providing an explicit calculation of the probability of the null hypothesis that x and y are independent second the fraser swinney algorithm is marginally the more accurate of the two algorithms when large data sets are used with smaller data sets however the fraser swinney algorithm reports structures that disappear when more data are available third the algorithm constructed here requires about 0 5 of the computation time required by the fraser swinney algorithm

a method for accurate stability bounds in a small denominator problem the author considers the problem of obtaining realistic lower bounds for the siegel radius recent advances of the analysis of siegel disks allows one to give a very accurate numerical algorithm based on rigorous results he finds that for non quadratic polynomial maps the maximal siegel radius might correspond to rotation numbers different from the golden mean

a consistent multivariate test of association based on ranks of distances summary we consider the problem of detecting associations between random vectors of any dimension few tests of independence exist that are consistent against all dependent alternatives we propose a powerful test that is applicable in all dimensions and consistent against all alternatives the test has a simple form is easy to implement and has good power

cleaning up the record on the maximal information coefficient and equitability although we appreciate kinney and atwal s interest in equitability and maximal information coefficient mic we believe they misrepresent our work we highlight a few of our main objections below fig 1 equitability of mic and mutual information under a range of noise models the equitability of mic and mutual information across a subset of noise models analyzed in refs 1 and 4 for each noise model the relationships tested are as in ref 4 in each regarding our original paper 1 kinney and atwal 2 state mic is said to satisfy not just the heuristic notion of equitability but also the mathematical criterion of r2 equitability the latter being their formalization of the heuristic notion that we introduced this statement is simply false we were explicit in our paper that our claims regarding mic s performance were based on large scale simulations we tested mic s equitability through simulations these show that for a large collection of test functions with varied sample sizes noise levels and noise models mic roughly equals the coefficient of determination r2 relative to each respective noiseless function although we mathematically proved several things about mic none of our claims imply that it satisfies kinney and atwal s r2 equitability which would require that mic exactly equal r2 in the infinite data limit thus their proof that no dependence measure can satisfy r2 equitability although interesting does not uncover any error in our work and their suggestion that it does is a gross misrepresentation kinney and atwal seem ready to toss out equitability as a useful criterion based on their theoretical result we argue however that regardless of whether perfect equitability is possible approximate notions of equitability remain the right goal for many data exploration settings just as the theory of np completeness does not suggest we stop thinking about np complete problems but instead that we look for approximations and solutions in restricted cases an impossibility result about perfect equitability provides focus for further research but does not mean that useful solutions are unattainable similarly as others have noted 3 kinney and atwal s proof requires a highly permissive noise model and so the attainability of r2 equitability under more limited noise models such as those in our work remains an open question finally the authors argue that mutual information is more equitable than mic however they provide as justification only a single noise model only at limiting sample sizes n 5 000 as we ve shown in follow up work 4 which they themselves cite but fail to address mic is more equitable than mutual information estimation under many other realistic noise models even at a sample size of 5 000 kinney and atwal have stated it matters how one defines noise 5 and a useful statistic must indeed be robust to a wide range of noise models equally importantly we ve established in both our original and follow up work that at sample size regimes less than 5 000 mic is more equitable than mutual information estimates across all noise models tested mic s superior equitability in these settings is not an artifact we neglected as kinney and atwal suggest but rather a weakness of mutual information estimation and an important consideration for practitioners we expect that the understanding of equitability and mic will improve over time and that better methods may arise however accurate representations of the work thus far will allow researchers in the area to most productively and collectively move forward

equitability mutual information and the maximal information coefficient how should one quantify the strength of association between two random variables without bias for relationships of a specific form despite its conceptual simplicity this notion of statistical equitability has yet to receive a definitive mathematical formalization here we argue that equitability is properly formalized by a self consistency condition closely related to data processing inequality mutual information a fundamental quantity in information theory is shown to satisfy this equitability criterion these findings are at odds with the recent work of reshef et al reshef dn et al 2011 science 334 6062 1518 1524 which proposed an alternative definition of equitability and introduced a new statistic the maximal information coefficient mic said to satisfy equitability in contradistinction to mutual information these conclusions however were supported only with limited simulation evidence not with mathematical arguments upon revisiting these claims we prove that the mathematical definition of equitability proposed by reshef et al cannot be satisfied by any nontrivial dependence measure we also identify artifacts in the reported simulation evidence when these artifacts are removed estimates of mutual information are found to be more equitable than estimates of mic mutual information is also observed to have consistently higher statistical power than mic we conclude that estimating mutual information provides a natural and often practical way to equitably quantify statistical associations in large datasets

a simple proof of a particular case of c siegel s center theorem we give an elementary proof of a particular case of c siegel s center theorem based on a method of m herman even if the proof has less generality than the standard one it is simpler and provides sharper bounds

the upward bias in measures of information derived from limited data samples extracting information measures from limited experimental samples such as those normally available when using data recorded in vivo from mammalian cortical neurons is known to be plagued by a systematic error which tends to bias the estimate upward we calculate here the average of the bias under certain conditions as an asymptotic expansion in the inverse of the size of the data sample the result agrees with numerical simulations and is applicable as an additive correction term to measurements obtained under such conditions moreover we discuss the implications for measurements obtained through other usual procedures

improved kam estimates for the siegel radius for the siegel center problem we explore the possibility of improving the kam estimates with a view to possible extensions to hamiltonian systems the use of a suitable norm and explicit perturbative computations allow estimates to within a factor 2 of the siegel radius for the quadratic map

r2 equitability is satisfiable kinney and atwal 1 make excellent points about mutual information the maximal information coefficient 2 3 and equitability one of their central claims however is that no nontrivial dependence measure can satisfy r 2 r2 equitability we argue that this is the result of a poorly constructed definition which we quote a dependence measure d x y d x y is r 2 r2 equitable if and only if when evaluated on a joint probability distribution p x y p x y that corresponds to a noisy functional relationship between two real random variables x and y the following relation holds d x y g r 2 f x y d x y g r2 f x y here g is a function that does not depend on p x y p x y and f is the function defining the noisy functional relationship i e y f x y f x

composite coefficient of determination and its application in ultrahigh dimensional variable screening in this article we propose to measure the dependence between two random variables through a composite coefficient of determination ccd of a set of nonparametric regressions these regressions ta

linearization of generalized interval exchange maps a standard interval exchange map is a one to one map of the interval that is locally a translation except at nitely many singularities we dene for such maps in terms of the rauzy veech continuous fraction algorithm a diophantine arithmetical condition called restricted roth type which is almost surely satised in parameter space let t0 be a standard interval exchange map of restricted roth type and let r be an integer 2 we prove that amongst c r 3 deformations of t0 that are c r 3 tangent to t0 at the singularities those that are conjugated to t0 by a c r dieomorphism close to the identity form a c 1 submanifold of codimension g 1 2r 1 s here g is the genus and s is the number of marked points of the translation surface obtained by suspension of t0 both g and s can be computed from the combinatorics of t0

estimation of entropy and mutual information we present some new results on the nonparametric estimation of entropy and mutual information first we use an exact local expansion of the entropy function to prove almost sure consistency and central limit theorems for three of the most commonly used discretized information estimators the setup is related to grenander s method of sieves and places no assumptions on the underlying probability measure generating the data second we prove a converse to these consistency theorems demonstrating that a misapplication of the most common estimation techniques leads to an arbitrarily poor estimate of the true information even given unlimited data this inconsistency theorem leads to an analytical approximation of the bias valid in surprisingly small sample regimes and more accurate than the usual 1 n formula of miller and madow over a large region of parameter space the two most practical implications of these results are negative 1 information estimates in a certain data regime are likely contaminated by bias even if bias corrected estimators are used and 2 confidence intervals calculated by standard techniques drastically underestimate the error of the most common estimation methods finally we note a very useful connection between the bias of entropy estimators and a certain polynomial approximation problem by casting bias calculation problems in this approximation theory framework we obtain the best possible generalization of known asymptotic bias results more interesting this framework leads to an estimator with some nice properties the estimator comes equipped with rigorous bounds on the maximum error over all possible underlying probability distributions and this maximum error turns out to be surprisingly small we demonstrate the application of this new estimator on both real and simulated data

rigorous estimates for a computer assisted kam theory nonautonomous hamiltonian systems of one degree of freedom close to integrable ones are considered let e be a positive parameter measuring the strength of the perturbation and denote by e c the critical value at which a given kam kolmogorov arnold moser torus breaks down a computer assisted method that allows one to give rigorous lower bounds for e c is presented this method has been applied in celletti falcolini porzio to be published in ann inst h poincare to the escande and doveil pendulum yielding a bound which is within a factor 40 2 of the value indicated by numerical experiments