Dimension-free empirical entropy estimation

1 Introduction . Estimating the entropy of a discrete distribution based on a finite iid sample is a classic problem with theoretical and practical ramifications . Considerable progress has been made in the case of a finite alphabet , and the countably infinite case has also attracted a fair amount of attention in recent years . A less-addressed issue is one of empirical accuracy estimates : data-dependent bounds adaptive to the particular distribution being sampled . Our point of departure is the simpler ( to analyze ) problem of estimating a discrete distribution µ in total variation norm ‖·‖TV = 12 ‖·‖1 , where the most recent advance was made by Cohen et al . [ 2020 ] ; see therein for a literature review . If µ is a distribution on N and µ̂n is its empirical realization based on a sample of size n , then Theorem 2.1 of Cohen et al . states that with probability at least 1− δ , ‖µ− µ̂n‖1 ≤ 2√ n ∑ j∈N √ µ̂n ( j ) + 6 √ log ( 2/δ ) 2n . ( 1 ) ∗A major part of this research was conducted when the author was graduate student at Ben-Gurion University of The Negev , Israel . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . This bound has the advantage of being valid for all distributions on N , without any prior assumptions , and being fully empirical : it yields a risk estimate that is computable based on the observed sample , not depending on any unknown quantities . ( Additionally , Cohen et al . argue that ( 1 ) is near-optimal in a well-defined sense . ) The question we set out to explore in this paper is : What analogues of ( 1 ) are possible for discrete entropy estimation ? When µ has support size d < ∞ , an answer to our question is readily provided by combining ( 1 ) with Cover and Thomas [ 2006 , Theorem 17.3.3 ] , which asserts that , for ‖µ− ν‖1 ≤ 1/2 , we have |H ( µ ) −H ( ν ) | ≤ ‖µ− ν‖1 log d ‖µ− ν‖1 , ( 2 ) where H ( · ) is the entropy functional defined in ( 3 ) . Indeed , taking µ as in ( 1 ) and ν to be µ̂n yields a fully empirical estimate on |H ( µ ) −H ( µ̂n ) | . For fixed d < ∞ , no technique relying on the plug-in estimator can yield minimax rates [ Wu and Yang , 2016 ] . The plug-in is , however , minimax optimal for fixed d < ∞ [ Paninski , 2003 ] as well as strongly universally consistent even for d =∞ [ Antos and Kontoyiannis , 2001a ] , and is among the few methods for which explicitly computable finite-sample risk bounds are known . The thrust of this paper is to replace the restrictive finite-support assumption with considerably more general moment conditions . It is well-known that when estimating the mean of some random variable X , the first-moment assumption E|X| ≤M is not sufficient to yield any finite-sample information.2 Strengthening the assumption to E|X|α ≤ M , for any α > 1 , immediately yields finite-sample empirical estimates on ∣∣EX − 1n ∑n i=1Xi ∣∣ via the von Bahr and Esseen [ 1965 ] inequality.3 In this sense , a bound on the ( 1 + ε ) th moment is a minimal requirement for empirical mean estimation . However , it is not immediately obvious how to apply this insight to the entropy estimation problem : the corresponding random variable is X = − logµ ( I ) , where I ∼ µ , but rather than being given iid samples of X , we are only given draws of I . Our contribution . In Theorem 1 , we provide a dimension-free analogue of ( 2 ) , which , combined with ( 1 ) , allows for empirical accuracy bounds on the plug-in entropy estimator under a minimalistic moment assumption . Moreover , for this rich class of distributions , the plug-in estimator turns out to be asymptotically optimal , as we show in Theorem 4 . Our moment assumption is natural and essentially the weakest one that makes any empirical bounds feasible , as we argue in Theorem 3 . As we demonstrate in Section 6 , the rates provided by our empirical bound compare favorably against the state of the art . 2 Definitions and notation . Our logarithms will always be base e by default . For discrete distributions , there is no loss of generality in taking the domain to be the natural numbers N = { 1 , 2 , 3 , . . . } . For k ∈ N , we write [ k ] : = { i ∈ N : i ≤ k } . The set of all probability distributions on N will be denoted by ∆N . For d ∈ N , we write ∆d ⊂ ∆N to denote those µ whose support is contained in [ d ] . We define the operator ( · ) ↓ , which maps any µ ∈ ∆N to its non-increasing rearrangement µ↓ . The set of all non-increasing distributions will be denoted by ∆↓N : = { µ↓ : µ ∈ ∆N } . We write R+ : = [ 0 , ∞ ) . For any ξ : N→ R+ and α ≥ 0 , define H ( α ) ( ξ ) : = ∑ j∈N : ξ ( j ) > 0 ξ ( j ) |log ξ ( j ) |α . ( 3 ) For ξ ∈ RN , denote by |ξ| ∈ RN+ the elementwise application of |·| to ξ . When ξ ∈ ∆N and α = 1 , ( 3 ) recovers the standard definition of entropy , which we denote by H ( ξ ) : = H ( 1 ) ( ξ ) . For general 2Even distinguishing , for X ≥ 0 , between EX = 0 and EX = M based on a finite sample is impossible with any degree of confidence . Of course , 1 n ∑n i=1Xi → EX almost surely , by the strong law of large numbers . 3Put Y = X − EX ; then E|Y | ≤ 2M . For 1 < α < 2 , a sharper version of the Bahr-Esseen inequality [ Pinelis , 2015 ] states that E [ ∣∣∑n i=1 Yi ∣∣α ] ≤ 2n ( 2M ) α , which implies tail bounds via Markov ’ s inequality . Better rates are available via the median-of-means estimator , see Lugosi and Mendelson [ 2019 ] . α > 0 , this quantity may be referred to as the αth moment of information . For h ≥ 0 , define ∆ ( α ) N [ h ] = { µ ∈ ∆N : H ( α ) ( µ ) ≤ h } and also ∆ ( α ) N : = ⋃ h≥0 ∆ ( α ) N [ h ] and ∆ ↓ ( α ) N [ h ] : = ∆ ↓ N ∩∆ ( α ) N [ h ] . For n ∈ N and µ ∈ ∆N , we write X = ( X1 , . . . , Xn ) ∼ µn to mean that the components of the vector X are drawn iid from µ . The empirical measure µ̂n ∈ ∆N induced by the sample X is defined by µ̂n ( j ) = 1 n ∑ i∈ [ n ] 1 [ Xi = j ] . For any ξ ∈ RN and 0 < p < ∞ , the ` p ( pseudo ) norm is defined by ‖ξ‖pp = ∑ j∈N |ξ ( j ) |p and ‖ξ‖∞ = supj∈N |ξ ( j ) | . For α , h > 0 , and n ∈ N , define the L1 minimax risk for the αth moment by R ( α ) n ( h ) : = inf Ĥ sup µ∈∆ ( α ) N [ h ] E|Ĥ ( X1 , . . . , Xn ) −H ( µ ) | , ( 4 ) where the infimum is over all mappings Ĥ : Nn → R+ . 3 Main results . Our first result is a dimension-free analogue of ( 2 ) : Theorem 1 . For all α > 1 , H : ∆ ( α ) N → R+ is uniformly continuous under ` 1 . In particular , for all µ , ν ∈ ∆ ( α ) N satisfying ‖µ− ν‖∞ < 1/2 , we have |H ( µ ) −H ( ν ) | ≤ ‖µ− ν‖1−1/α1 ( 2αα + H ( α ) ( µ ) + H ( α ) ( ν ) ) 1/α ≤ ‖µ− ν‖1−1/α1 ( 2α+ H ( α ) ( µ ) 1/α + H ( α ) ( ν ) 1/α ) . The requirement in Theorem 1 that α > 1 can not be dispensed with , as the function H : ∆ ( α ) N [ h ] → R+ is not continuous under ` 1 for α = 1 ( see Remark following Lemma 5 ) , and , a fortiori , is not uniformly continuous . Thus , there can be no function F : R2+ → R+ satisfying |H ( µ ) −H ( ν ) | ≤ F ( ‖µ− ν‖1 , h ) , h > 0 , µ , ν ∈ ∆ ( 1 ) N [ h ] with the additional property that for any two sequences µn , νn ∈ ∆N satisfying εn : = ‖µn − νn‖1 → 0 , it holds that F ( εn , h ) → 0 . Perhaps surprisingly,4 it turns out that H : ∆ ( α ) N [ h ] → R+ is uniformly continuous under ` p for all α > 1 , p ∈ [ 1 , ∞ ] : Theorem 2 . There is a function F : R4+ → R+ such that |H ( µ ) −H ( ν ) | ≤ F ( ‖µ− ν‖p , h , α , p ) , h > 0 , α > 1 , p ∈ [ 1 , ∞ ] , µ , ν ∈ ∆ ( α ) N [ h ] with the additional property that whenever εn : = ‖µn − νn‖p → 0 , we have F ( εn , h , α , p ) → 0 . Remark . Although Theorem 2 establishes uniform continuity , it gives no hint as to the functional dependence of the modulus of continuity F on α , p , h , and ‖µ− ν‖p . We leave this as a fascinating open problem — even though the practical applications are likely to be limited : it follows from Wyner and Foster [ 2003 ] and Theorem 4 that for p = α = 2 and fixed h , F ( ‖µ− ν‖2 , h , 2 , 2 ) can not decay at a faster rate than 1/ log ( 1/ ‖µ− ν‖2 ) . Combining Theorem 1 with ( 1 ) yields an empirical ( under moment assumptions ) bound for the plug-in entropy estimator : 4Since ` 1 dominates all of the ` p norms , continuity of a function under ` p trivially implies continuity under ` 1 , but the reverse implication is generally not true . Corollary 1 . For all α > 1 , h > 0 , δ ∈ ( 0 , 1 ) , n ≥ 2 log 4δ , and µ ∈ ∆ ( α ) N [ h ] , we have that |H ( µ ) −H ( µ̂n ) | ≤ ( 2αα + h+ H ( α ) ( µ̂n ) ) 1/α 2 ‖µ̂n‖ 1/2 1/2√ n + 6 √ log ( 4/δ ) 2n 1−1/α holds with probability at least 1− δ . In Section 6 , we compare the rates implied by Corollary 1 to the state of the art on various distributions . Next , we examine the optimality of the plug-in estimate by analyzing the minimax risk , defined in ( 4 ) . It was known [ Silva , 2018 , Appendix A ] that assuming H ( µ ) < ∞ does not suffice to yield a minimax rate for the L2 risk : inf Ĥ : Nn→R+ sup µ∈∆ ( 1 ) N E ( Ĥ ( X1 , . . . , Xn ) −H ( µ ) ) 2 =∞ . This technique yields an analogous result for the L1 risk as well . We strengthen these results in two ways : ( i ) by lower-bounding the L1 risk ( rather than L2 , which is never smaller ) , and ( ii ) by restricting µ to ∆ ( 1 ) N [ h ] and obtaining a finitary , quantitative lower bound : Theorem 3 . For α = 1 , there is a universal constant C > 0 such that for all h > 1 and n ∈ N , we haveR ( 1 ) n ( h ) ≥ Ch . Remark . The above result complements — but is not directly comparable to — Antos and Kontoyiannis [ 2001a , Theorem 4 ] . Ours gives a quantitative dependence on h but constructs an adversarial distribution for each sample size n ; theirs is asymptotic only but a single adversarial distribution suffices for all n. Remark . Our technique immediately yields a lower bound of Ch2 on the L2 minimax risk . In contradistinction to the α = 1 case , where no minimax rate exists , we show that the plug-in estimator is minimax for all α > 1 : Theorem 4 . The following bounds hold for the L1 minimax risk : ( a ) Upper bound : for all h > 0 , α > 1 , R ( α ) n ( h ) ≤ 1 + log n√ n + 2α−1h logα−1 n , n ∈ N ; further , this bound is achieved by the plug-in estimate H ( µ̂n ) . ( b ) Lower bound : for each α > 0 , n ∈ N there is an h > 0 such that R ( α ) n ( h ) ≥ h 4 · 3α logα−1 n . Open problem . Close the gap in the dependence on α in the upper and lower bounds .

Summary: This paper proves an inequality for estimating the entropy by means of plugging in the empirical distribution that depends only empirical quantities, in the setting of iid samples from a discrete alphabet with potentially infinite size. This enables the statistician to certify directly in terms of the sample how well the entropy is estimated by the simple plug-in estimator. The result is proven under a mild moment condition which is known to be essential for consistent estimation, and further the authors show strong lower bounds on the risk when the moment assumption fails to hold. Further, minimax lower bounds are shown for estimating the entropy under this moment assumption that are sharp up to constant factors. In the final section these bounds are compared to pre-existing ones in various settings.