Estimating mutual information from i.i.d. samples drawn from an unknown joint density function is a basic statistical problem of broad interest with multitudinous applications. The most popular estimator is one proposed by Kraskov and St "ogbauer and Grassberger (KSG) in 2004, and is nonparametric and based on the distances of each sample to its @math nearest neighboring sample, where @math is a fixed small integer. Despite its widespread use (part of scientific software packages), theoretical properties of this estimator have been largely unexplored. In this paper we demonstrate that the estimator is consistent and also identify an upper bound on the rate of convergence of the bias as a function of number of samples. We argue that the superior performance benefits of the KSG estimator stems from a curious "correlation boosting" effect and build on this intuition to modify the KSG estimator in novel ways to construct a superior estimator. As a byproduct of our investigations, we obtain nearly tight rates of convergence of the @math error of the well known fixed @math nearest neighbor estimator of differential entropy by Kozachenko and Leonenko.
In some cases, the parametric rate of convergence of @math of @math error is achieved: of particular interest is the result of @cite_60 where the parametric rate is achieved for differential entropy estimation via KDE of density followed by the resubstitution estimator when the dimension is no more than 6. Numerical evidence suggests the hypothesis that the lower bounds derived in Theorem below could perhaps be improved when the dimension is more than 4 and estimators constrained to only use fixed @math -NN distances. Under certain very strong conditions on the density class (that are relevant in certain applications on graphical model selection @cite_55 ), exponential rate of convergence can be demonstrated @cite_66 @cite_33 . Recent works @cite_63 @cite_0 have studied the performance of the leave-one-out (LOO) approach where all but the sample of resubstitution are used for KDE, involving techniques such as von Mises expansion methods.
@cite_33 @cite_60 @cite_55 @cite_0 @cite_63 @cite_66
2160184181 2029140344 2143908925 2188155116 1673728053 2963540279
We derive the mean squared error convergence rates of kernel density-based plug-in estimators of mutual information measures between two multidimensional random variables @math and @math for two cases: 1) @math and @math are both continuous; 2) @math is continuous and @math is discrete. Using the derived rates, we propose an ensemble estimator of these information measures for the second case by taking a weighted sum of the plug-in estimators with varied bandwidths. The resulting ensemble estimator achieves the @math parametric convergence rate when the conditional densities of the continuous variables are sufficiently smooth. To the best of our knowledge, this is the first nonparametric mutual information estimator known to achieve the parametric convergence rate for this case, which frequently arises in applications (e.g. variable selection in classification). The estimator is simple to implement as it uses the solution to an offline convex optimization problem and simple plug-in estimators. A central limit theorem is also derived for the ensemble estimator. Ensemble estimators that achieve the parametric rate are also derived for the first case ( @math and @math are both continuous) and another case 3) @math and @math may have any mixture of discrete and continuous components. We investigate the minimization of Newton's functional for the problem of the body of minimal resistance with maximal height @math cite butt in the class of convex developable functions defined in a disc. This class is a natural candidate to find a (non-radial) minimizer in accordance with the results of cite lrp2. We prove that the minimizer in this class has a minimal set in the form of a regular polygon with @math sides centered in the disc, and numerical experiments indicate that the natural number @math is a non-decreasing function of @math . The corresponding functions all achieve a lower value of the functional than the optimal radially symmetric function with the same height @math . A citing sentence is one that appears in a scientific article and cites previous work. Citing sentences have been studied and used in many applications. For example, they have been used in scientific paper summarization, automatic survey generation, paraphrase identification, and citation function classification. Citing sentences that cite multiple papers are common in scientific writing. This observation should be taken into consideration when using citing sentences in applications. For instance, when a citing sentence is used in a summary of a scientific paper, only the fragments of the sentence that are relevant to the summarized paper should be included in the summary. In this paper, we present and compare three different approaches for identifying the fragments of a citing sentence that are related to a given target reference. Our methods are: word classification, sequence labeling, and segment classification. Our experiments show that segment classification achieves the best results. We consider nonparametric estimation of L2, Renyi-α and Tsallis-α divergences between continuous distributions. Our approach is to construct estimators for particular integral functionals of two densities and translate them into divergence estimators. For the integral functionals, our estimators are based on corrections of a preliminary plug-in estimator. We show that these estimators achieve the parametric convergence rate of n-1 2 when the densities' smoothness, s, are both at least d 4 where d is the dimension. We also derive minimax lower bounds for this problem which confirm that s > d 4 is necessary to achieve the n-1 2 rate of convergence. We validate our theoretical guarantees with a number of simulations.