Dimension-free empirical entropy estimation

1 Introduction . Estimating the entropy of a discrete distribution based on a finite iid sample is a classic problem with theoretical and practical ramifications . Considerable progress has been made in the case of a finite alphabet , and the countably infinite case has also attracted a fair amount of attention in recent years . A less-addressed issue is one of empirical accuracy estimates : data-dependent bounds adaptive to the particular distribution being sampled . Our point of departure is the simpler ( to analyze ) problem of estimating a discrete distribution µ in total variation norm ‖·‖TV = 12 ‖·‖1 , where the most recent advance was made by Cohen et al . [ 2020 ] ; see therein for a literature review . If µ is a distribution on N and µ̂n is its empirical realization based on a sample of size n , then Theorem 2.1 of Cohen et al . states that with probability at least 1− δ , ‖µ− µ̂n‖1 ≤ 2√ n ∑ j∈N √ µ̂n ( j ) + 6 √ log ( 2/δ ) 2n . ( 1 ) ∗A major part of this research was conducted when the author was graduate student at Ben-Gurion University of The Negev , Israel . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . This bound has the advantage of being valid for all distributions on N , without any prior assumptions , and being fully empirical : it yields a risk estimate that is computable based on the observed sample , not depending on any unknown quantities . ( Additionally , Cohen et al . argue that ( 1 ) is near-optimal in a well-defined sense . ) The question we set out to explore in this paper is : What analogues of ( 1 ) are possible for discrete entropy estimation ? When µ has support size d < ∞ , an answer to our question is readily provided by combining ( 1 ) with Cover and Thomas [ 2006 , Theorem 17.3.3 ] , which asserts that , for ‖µ− ν‖1 ≤ 1/2 , we have |H ( µ ) −H ( ν ) | ≤ ‖µ− ν‖1 log d ‖µ− ν‖1 , ( 2 ) where H ( · ) is the entropy functional defined in ( 3 ) . Indeed , taking µ as in ( 1 ) and ν to be µ̂n yields a fully empirical estimate on |H ( µ ) −H ( µ̂n ) | . For fixed d < ∞ , no technique relying on the plug-in estimator can yield minimax rates [ Wu and Yang , 2016 ] . The plug-in is , however , minimax optimal for fixed d < ∞ [ Paninski , 2003 ] as well as strongly universally consistent even for d =∞ [ Antos and Kontoyiannis , 2001a ] , and is among the few methods for which explicitly computable finite-sample risk bounds are known . The thrust of this paper is to replace the restrictive finite-support assumption with considerably more general moment conditions . It is well-known that when estimating the mean of some random variable X , the first-moment assumption E|X| ≤M is not sufficient to yield any finite-sample information.2 Strengthening the assumption to E|X|α ≤ M , for any α > 1 , immediately yields finite-sample empirical estimates on ∣∣EX − 1n ∑n i=1Xi ∣∣ via the von Bahr and Esseen [ 1965 ] inequality.3 In this sense , a bound on the ( 1 + ε ) th moment is a minimal requirement for empirical mean estimation . However , it is not immediately obvious how to apply this insight to the entropy estimation problem : the corresponding random variable is X = − logµ ( I ) , where I ∼ µ , but rather than being given iid samples of X , we are only given draws of I . Our contribution . In Theorem 1 , we provide a dimension-free analogue of ( 2 ) , which , combined with ( 1 ) , allows for empirical accuracy bounds on the plug-in entropy estimator under a minimalistic moment assumption . Moreover , for this rich class of distributions , the plug-in estimator turns out to be asymptotically optimal , as we show in Theorem 4 . Our moment assumption is natural and essentially the weakest one that makes any empirical bounds feasible , as we argue in Theorem 3 . As we demonstrate in Section 6 , the rates provided by our empirical bound compare favorably against the state of the art . 2 Definitions and notation . Our logarithms will always be base e by default . For discrete distributions , there is no loss of generality in taking the domain to be the natural numbers N = { 1 , 2 , 3 , . . . } . For k ∈ N , we write [ k ] : = { i ∈ N : i ≤ k } . The set of all probability distributions on N will be denoted by ∆N . For d ∈ N , we write ∆d ⊂ ∆N to denote those µ whose support is contained in [ d ] . We define the operator ( · ) ↓ , which maps any µ ∈ ∆N to its non-increasing rearrangement µ↓ . The set of all non-increasing distributions will be denoted by ∆↓N : = { µ↓ : µ ∈ ∆N } . We write R+ : = [ 0 , ∞ ) . For any ξ : N→ R+ and α ≥ 0 , define H ( α ) ( ξ ) : = ∑ j∈N : ξ ( j ) > 0 ξ ( j ) |log ξ ( j ) |α . ( 3 ) For ξ ∈ RN , denote by |ξ| ∈ RN+ the elementwise application of |·| to ξ . When ξ ∈ ∆N and α = 1 , ( 3 ) recovers the standard definition of entropy , which we denote by H ( ξ ) : = H ( 1 ) ( ξ ) . For general 2Even distinguishing , for X ≥ 0 , between EX = 0 and EX = M based on a finite sample is impossible with any degree of confidence . Of course , 1 n ∑n i=1Xi → EX almost surely , by the strong law of large numbers . 3Put Y = X − EX ; then E|Y | ≤ 2M . For 1 < α < 2 , a sharper version of the Bahr-Esseen inequality [ Pinelis , 2015 ] states that E [ ∣∣∑n i=1 Yi ∣∣α ] ≤ 2n ( 2M ) α , which implies tail bounds via Markov ’ s inequality . Better rates are available via the median-of-means estimator , see Lugosi and Mendelson [ 2019 ] . α > 0 , this quantity may be referred to as the αth moment of information . For h ≥ 0 , define ∆ ( α ) N [ h ] = { µ ∈ ∆N : H ( α ) ( µ ) ≤ h } and also ∆ ( α ) N : = ⋃ h≥0 ∆ ( α ) N [ h ] and ∆ ↓ ( α ) N [ h ] : = ∆ ↓ N ∩∆ ( α ) N [ h ] . For n ∈ N and µ ∈ ∆N , we write X = ( X1 , . . . , Xn ) ∼ µn to mean that the components of the vector X are drawn iid from µ . The empirical measure µ̂n ∈ ∆N induced by the sample X is defined by µ̂n ( j ) = 1 n ∑ i∈ [ n ] 1 [ Xi = j ] . For any ξ ∈ RN and 0 < p < ∞ , the ` p ( pseudo ) norm is defined by ‖ξ‖pp = ∑ j∈N |ξ ( j ) |p and ‖ξ‖∞ = supj∈N |ξ ( j ) | . For α , h > 0 , and n ∈ N , define the L1 minimax risk for the αth moment by R ( α ) n ( h ) : = inf Ĥ sup µ∈∆ ( α ) N [ h ] E|Ĥ ( X1 , . . . , Xn ) −H ( µ ) | , ( 4 ) where the infimum is over all mappings Ĥ : Nn → R+ . 3 Main results . Our first result is a dimension-free analogue of ( 2 ) : Theorem 1 . For all α > 1 , H : ∆ ( α ) N → R+ is uniformly continuous under ` 1 . In particular , for all µ , ν ∈ ∆ ( α ) N satisfying ‖µ− ν‖∞ < 1/2 , we have |H ( µ ) −H ( ν ) | ≤ ‖µ− ν‖1−1/α1 ( 2αα + H ( α ) ( µ ) + H ( α ) ( ν ) ) 1/α ≤ ‖µ− ν‖1−1/α1 ( 2α+ H ( α ) ( µ ) 1/α + H ( α ) ( ν ) 1/α ) . The requirement in Theorem 1 that α > 1 can not be dispensed with , as the function H : ∆ ( α ) N [ h ] → R+ is not continuous under ` 1 for α = 1 ( see Remark following Lemma 5 ) , and , a fortiori , is not uniformly continuous . Thus , there can be no function F : R2+ → R+ satisfying |H ( µ ) −H ( ν ) | ≤ F ( ‖µ− ν‖1 , h ) , h > 0 , µ , ν ∈ ∆ ( 1 ) N [ h ] with the additional property that for any two sequences µn , νn ∈ ∆N satisfying εn : = ‖µn − νn‖1 → 0 , it holds that F ( εn , h ) → 0 . Perhaps surprisingly,4 it turns out that H : ∆ ( α ) N [ h ] → R+ is uniformly continuous under ` p for all α > 1 , p ∈ [ 1 , ∞ ] : Theorem 2 . There is a function F : R4+ → R+ such that |H ( µ ) −H ( ν ) | ≤ F ( ‖µ− ν‖p , h , α , p ) , h > 0 , α > 1 , p ∈ [ 1 , ∞ ] , µ , ν ∈ ∆ ( α ) N [ h ] with the additional property that whenever εn : = ‖µn − νn‖p → 0 , we have F ( εn , h , α , p ) → 0 . Remark . Although Theorem 2 establishes uniform continuity , it gives no hint as to the functional dependence of the modulus of continuity F on α , p , h , and ‖µ− ν‖p . We leave this as a fascinating open problem — even though the practical applications are likely to be limited : it follows from Wyner and Foster [ 2003 ] and Theorem 4 that for p = α = 2 and fixed h , F ( ‖µ− ν‖2 , h , 2 , 2 ) can not decay at a faster rate than 1/ log ( 1/ ‖µ− ν‖2 ) . Combining Theorem 1 with ( 1 ) yields an empirical ( under moment assumptions ) bound for the plug-in entropy estimator : 4Since ` 1 dominates all of the ` p norms , continuity of a function under ` p trivially implies continuity under ` 1 , but the reverse implication is generally not true . Corollary 1 . For all α > 1 , h > 0 , δ ∈ ( 0 , 1 ) , n ≥ 2 log 4δ , and µ ∈ ∆ ( α ) N [ h ] , we have that |H ( µ ) −H ( µ̂n ) | ≤ ( 2αα + h+ H ( α ) ( µ̂n ) ) 1/α 2 ‖µ̂n‖ 1/2 1/2√ n + 6 √ log ( 4/δ ) 2n 1−1/α holds with probability at least 1− δ . In Section 6 , we compare the rates implied by Corollary 1 to the state of the art on various distributions . Next , we examine the optimality of the plug-in estimate by analyzing the minimax risk , defined in ( 4 ) . It was known [ Silva , 2018 , Appendix A ] that assuming H ( µ ) < ∞ does not suffice to yield a minimax rate for the L2 risk : inf Ĥ : Nn→R+ sup µ∈∆ ( 1 ) N E ( Ĥ ( X1 , . . . , Xn ) −H ( µ ) ) 2 =∞ . This technique yields an analogous result for the L1 risk as well . We strengthen these results in two ways : ( i ) by lower-bounding the L1 risk ( rather than L2 , which is never smaller ) , and ( ii ) by restricting µ to ∆ ( 1 ) N [ h ] and obtaining a finitary , quantitative lower bound : Theorem 3 . For α = 1 , there is a universal constant C > 0 such that for all h > 1 and n ∈ N , we haveR ( 1 ) n ( h ) ≥ Ch . Remark . The above result complements — but is not directly comparable to — Antos and Kontoyiannis [ 2001a , Theorem 4 ] . Ours gives a quantitative dependence on h but constructs an adversarial distribution for each sample size n ; theirs is asymptotic only but a single adversarial distribution suffices for all n. Remark . Our technique immediately yields a lower bound of Ch2 on the L2 minimax risk . In contradistinction to the α = 1 case , where no minimax rate exists , we show that the plug-in estimator is minimax for all α > 1 : Theorem 4 . The following bounds hold for the L1 minimax risk : ( a ) Upper bound : for all h > 0 , α > 1 , R ( α ) n ( h ) ≤ 1 + log n√ n + 2α−1h logα−1 n , n ∈ N ; further , this bound is achieved by the plug-in estimate H ( µ̂n ) . ( b ) Lower bound : for each α > 0 , n ∈ N there is an h > 0 such that R ( α ) n ( h ) ≥ h 4 · 3α logα−1 n . Open problem . Close the gap in the dependence on α in the upper and lower bounds .

Summary: The main contribution of the paper is to give a continuity bound on entropy: how far the entropies of two categorical distributions are, based on the L1-distance between these distributions. This bound does not depend on dimension, and is instead expressed in terms of a higher information-moment, which is similar to the expression of entropy, but with the log taken to a power alpha. Let’s call this alpha-entropy (note: not Renyi entropy).  Using high-probability bounds on the L1-distance between empirical and true distributions, this gives a parallel bound on the accuracy of the empirical plug-in entropy estimator. The paper shows that, over the class of bounded alpha-entropy distributions, the plug-in estimator is minimax optimal. For the class of finite-dimensional distributions, the plug-in estimator is not minimax optimal, but the paper also shows that for sub-classes with small alpha-entropy its rate is not that far-off from the optimal rate.