Formal Limitations on the Measurement of Mutual Information

Motivated by applications to unsupervised learning , we consider the problem of measuring mutual information . Recent analysis has shown that naive kNN estimators of mutual information have serious statistical limitations motivating more refined methods . In this paper we prove that serious statistical limitations are inherent to any measurement method . More specifically , we show that any distributionfree high-confidence lower bound on mutual information can not be larger than O ( lnN ) where N is the size of the data sample . We also analyze the DonskerVaradhan lower bound on KL divergence in particular and show that , when simple statistical considerations are taken into account , this bound can never produce a high-confidence value larger than lnN . While large high-confidence lower bounds are impossible , in practice one can use estimators without formal guarantees . We suggest expressing mutual information as a difference of entropies and using cross entropy as an entropy estimator . We observe that , although cross entropy is only an upper bound on entropy , cross-entropy estimates converge to the true cross entropy at the rate of 1/ √ N . 1 INTRODUCTION . Motivated by maximal mutual information ( MMI ) predictive coding ( McAllester , 2018 ; Stratos , 2018 ; Oord et al. , 2018 ) , we consider the problem of measuring mutual information . A classical approach to this problem is based on estimating entropies by computing the average log of the distance to the kth nearest neighbor in a sample ( Kraskov et al. , 2003 ) . It has recently been shown that the classical kNN methods have serious statistical limitations and more refined kNN methods have been proposed ( Gao et al. , 2014 ) . Here we establish serious statistical limitations on any method of estimating mutual information . More specifically , we show that any distribution-free high-confidence lower bound on mutual information can not be larger than O ( lnN ) where N is the size of the data sample . Prior to proving the general case , we consider the particular case of the Donsker-Varadhan lower bound on KL divergence ( Donsker & Varadhan , 1983 ; Belghazi et al. , 2018 ) . We observe that when simple statistical considerations are taken into account , this bound can never produce a highconfidence value larger than lnN . Similar comments apply to lower bounds based on contrastive estimation . The contrastive estimation lower bound given in Oord et al . ( 2018 ) does not establish mutual information of more than ln k where k is number of negative samples used in the contrastive choice . The difficulties arise in cases where the mutual information I ( x , y ) is large . Since I ( x , y ) = H ( y ) −H ( y|x ) we are interested in cases where H ( y ) is large and H ( y|x ) is small . For example consider the mutual information between an English sentence and its French translation . Sampling English and French independently will ( almost ) never yield two sentences where one is a plausible translation of the other . In this case the DV bound is meaningless and contrastive estimation is trivial . In this example we need a language model for estimating H ( y ) and a translation model for estimating H ( y|x ) . Language models and translation models are both typically trained with crossentropy loss . Cross-entropy loss can be used as an ( upper bound ) estimate of entropy and we get an estimate of mutual information as a difference of cross-entropy estimates . Note that the upper-bound guarantee for the cross-entropy estimator yields neither an upper bound nor a lower bound guarantee for a difference of entropies . Similar observations apply to measuring the mutual information for pairs of nearby frames of video or pairs of sound waves for utterances of the same sentence . We are motivated by the problem of maximum mutual information predictive coding ( McAllester , 2018 ; Stratos , 2018 ; Oord et al. , 2018 ) . One can formally define a version of MMI predictive coding by considering a population distribution on pairs ( x , y ) where we think of x as past raw sensory signals ( images or sound waves ) and y as a future sensory signal . We consider the problem of learning stochastic coding functions Cx and Cy so as to maximize the mutual information I ( Cx ( x ) , Cy ( y ) ) while limiting the entropies H ( Cx ( x ) ) and H ( Cy ( y ) ) . The intuition is that we want to learn representations Cx ( x ) and Cy ( y ) that preserve “ signal ” while removing “ noise ” . Here signal is simply defined to be a low entropy representation that preserves mutual information with the future . Forms of MMI predictive coding have been independently introduced in ( McAllester , 2018 ) under the name “ information-theoretic cotraining ” and in ( Oord et al. , 2018 ) under the name “ contrastive predictive coding ” . It is also possible to interpret the local version of DIM ( DIM ( L ) ) ( Hjelm et al. , 2018 ) as a variant of MMI predictive coding . A closely related framework is the information bottleneck ( Tishby et al. , 2000 ) . Here one again assumes a population distribution on pairs ( x , y ) . The objective is to learn a stochastic coding function Cx so as to maximize I ( Cx ( x ) , y ) while minimizing I ( Cx ( x ) , x ) . Here one does not ask for a coding function on y and one does not limit H ( Cx ( x ) ) . Another related framework is INFOMAX ( Linsker , 1988 ; Bell & Sejnowski , 1995 ; Hjelm et al. , 2018 ) . Here we consider a population distribution on a single random variable x . The objective is to learn a stochastic coding function Cx so as to maximize the mutual information I ( x , Cx ( x ) ) subject to some constraint or additional objective . As mentioned above , in cases where I ( Cx ( x ) , Cy ( y ) ) is large it seems best to train a model of the marginal distribution of P ( Cy ) and a model of the conditional distribution P ( Cy|Cx ) where both models are trained with cross-entropy loss . Section 5 gives various high confidence upper bounds on cross-entropy loss for learned models . The main point is that , unlike lower bounds on entropy , high-confidence upper bounds on cross-entropy loss can be guaranteed to be close to the true cross entropy . Out theoretical analyses will assume discrete distributions . However , there is no loss of generality in this assumption . Rigorous treatments of probability ( measure theory ) treat integrals ( either Riemann or Lebesgue ) as limits of increasingly fine binnings . A continuous density can always be viewed as a limit of discrete distributions . Although our proofs are given for discrete case , all our formal limitations on the measurement of mutual information apply to continuous case as well . See Marsh ( 2013 ) for a discussion of continuous information theory . Additional comments on this point are given in section 4 . 2 THE DONSKER-VARADHAN LOWER BOUND . Mutual information can be written as a KL divergence . I ( X , Y ) = KL ( PX , Y , PXPY ) Here PX , Y is a joint distribution on the random variables X and Y and PX and PY are the marginal distributions on X and Y respectively . The DV lower bound applies to KL-divergence generally . To derive the DV bound we start with the following observation for any distributions P , Q , and G on the same support . Our theoretical analyses will assume discrete distributions . KL ( P , Q ) = Ez∼P ln P ( z ) Q ( z ) = Ez∼P ln ( G ( z ) Q ( z ) P ( z ) G ( z ) ) = Ez∼P ln G ( z ) Q ( z ) +KL ( P , G ) ≥ Ez∼P ln G ( z ) Q ( z ) ( 1 ) Note that ( 1 ) achieves equality for G ( z ) = P ( z ) and hence we have KL ( P , Q ) = sup G Ez∈P ln G ( z ) Q ( z ) ( 2 ) Here we can let G be a parameterized model such that G ( z ) can be computed directly . However , we are interested inKL ( PX , Y , PXPY ) where our only access to the distribution P is through sampling . If we draw a pair ( x , y ) and ignore y we get a sample from PX . We can similarly sample from PY . So we are interested in a KL-divergenceKL ( P , Q ) where our only access to the distributions P and Q is through sampling . Note that we can not evaluate ( 1 ) by sampling from P because we have no way of computing Q ( z ) . But through a change of variables we can convert this to an expression restricted to sampling from Q . More specifically we define G ( z ) in terms of an unconstrained function F ( z ) as G ( z ) = 1 Z Q ( z ) eF ( z ) Z = ∑ z Q ( z ) eF ( z ) = Ez∼Q e F ( z ) ( 3 ) Substituting ( 3 ) into ( 2 ) gives KL ( P , Q ) = sup F Ez∼P F ( z ) − lnEz∼Q eF ( z ) ( 4 ) Equation ( 4 ) is the Donsker-Varadhan lower bound . Applying this to mutual information we get I ( X , Y ) = KL ( PX , Y , PXPY ) = sup F Ex , y∼PX , Y F ( x , y ) − lnEx∼PX , y∼PY eF ( x , y ) ( 5 ) This is the equation underlying the MINE approach to maximizing mutual information ( Belghazi et al. , 2018 ) . It would seem that we can estimate both terms in ( 5 ) through sampling and be able to maximize I ( X , Y ) by stochastic gradient ascent on this lower bound . 3 STATISTICAL LIMITATIONS OF KL-DIVERGENCE LOWER BOUNDS . In this section we show that the DV bound ( 4 ) can not be used to measure KL-divergences of more than tens of bits . In fact we will show that no high-confidence distribution-free lower bound on KL divergence can be used for this purpose . As a first observation note that ( 4 ) involves Ez∼Q eF ( z ) . This expression has the same form as the moment generating function used in analyzing large deviation probabilities . The utility of expectations of exponentials in large deviation theory is that such expressions can be dominated by extremely rare events ( large deviations ) . The rare events dominating the expectation will never be observed by sampling from Q . It should be noted that the optimal value for F ( z ) in ( 4 ) is ln ( P ( z ) /Q ( z ) ) in which case the right hand side of ( 4 ) simplifies to KL ( P , Q ) . But for large KL divergence we will have that F ( z ) = ln ( P ( z ) /Q ( z ) ) is typically hundreds of bits and this is exactly the case where Ez∼Q eF ( z ) can not be measured by sampling from Q . If Ez∼Q eF ( z ) is dominated by events that will never occur in sampling from Q then the optimization of F through the use of ( 4 ) and sampling from Q can not possibly lead to a function F ( z ) that accurately models the desired function ln ( P ( z ) /Q ( z ) ) . To quantitatively analyze the risk of unseen outlier events we will make use of the following simple lemma where we write Pz∼Q ( Φ [ z ] ) for the probability over drawing z from Q that the statement Φ [ z ] holds . Outlier Risk Lemma : For a sample S ∼ QN with N ≥ 2 , and a property Φ [ z ] such that Pz∼Q ( Φ [ z ] ) ≤ 1/N , the probability over the draw of S that no z ∈ S satisfies Φ [ z ] is at least 1/4 . Proof : The probability that Φ [ z ] is unseen in the sample is at least ( 1−1/N ) N which is at least 1/4 for N ≥ 2 and where we have limN→∞ ( 1− 1/N ) N = 1/e . Q.E.D . We can use the outlier risk lemma to perform a quantitative risk analysis of the DV bound ( 4 ) . We can rewrite ( 4 ) as KL ( P , Q ) ≥ B ( P , Q , F ) B ( P , Q , F ) = Ez∼P F ( z ) − lnEz∼Q eF ( z ) We can try to estimate B ( P , Q , G ) from samples SP and SQ , each of size N , from the population distributions P and Q respectively . B̂ ( SP , SQ , F ) = 1 N ∑ z∈SP F ( z ) − ln 1 N ∑ z∈SQ eF ( z ) While B ( P , Q , F ) is a lower bound on KL ( P , Q ) , the sample estimate B̂ ( SP , SQ , F ) is not . To get a high confidence lower bound on KL ( P , Q ) we have to handle unseen outlier risk . For a fair comparison with our analysis of cross-entropy estimators in section 5 , we will limit the outlier risk by bounding F ( z ) to the interval [ 0 , Fmax ] . The largest possible value of B̂ ( SP , Sq , F ) occurs when F ( z ) = Fmax for all z ∈ SP and F ( z ) = 0 for all z ∈ SQ . In this case we get B̂ ( SP , SQ , F ) = Fmax . But by the outlier risk lemma there is still at least a 1/4 probability that Ez∼Q e F ( z ) ≥ 1 N eFmax . ( 6 ) Any high confidence lower bound B̃ ( SP , SQ , F ) must account for the unseen outlier risk . In particular we must have B̃ ( SP , SQ , F ) ≤ Fmax − ln eFmax N = lnN Our negative results can be strengthened by considering the preliminary bound ( 1 ) where G ( z ) is viewed as a model of P ( z ) . We can consider the extreme case of perfect modeling of the population P with a model G ( z ) where G ( z ) is computable . In this case we have essentially complete access to the distribution P . But even in this setting we have the following negative result . Theorem 1 Let B be any distribution-free high-confidence lower bound on KL ( P , Q ) computed with complete knowledge of P but only a sample from Q . More specifically , let B ( P , S , δ ) be any real-valued function of a distribution P , a multiset S , and a confidence parameter δ such that , for any P , Q and δ , with probability at least ( 1− δ ) over a draw of S from QN we have KL ( P , Q ) ≥ B ( P , S , δ ) . For any such bound , and for N ≥ 2 , with probability at least 1 − 4δ over the draw of S from QN we have B ( P , S , δ ) ≤ lnN . Proof . Consider distributions P and Q and N ≥ 2 . Define Q̃ by Q̃ ( z ) = ( 1− 1 N ) Q ( z ) + 1 N P ( z ) . We now have KL ( P , Q̃ ) ≤ lnN . We will prove that from a sample S ∼ QN we can not reliably distinguish between Q and Q̃ . We first note that by applying the high-confidence guarantee of the bound to Q̃ have PS∼Q̃N ( B ( P , S , δ ) ≤ KL ( P , Q̃ ) ) ≥ 1− δ . The distribution Q̃ equals the marginal on z of a distribution on pairs ( s , z ) where s is the value of Bernoulli variable with bias 1/N such that if s = 1 then z is drawn from P and otherwise z is drawn from Q . By the outlier risk lemma the probability that all coins are zero is at least 1/4 . Conditioned on all coins being zero the distributions Q̃N and QN are the same . Let Pure ( S ) represent the event that all coins are 0 and let Small ( S ) represent the event that B ( P , S , δ ) ≤ lnN . We now have PS∼QN ( Small ( S ) ) = PS∼Q̃N ( Small ( S ) |Pure ( S ) ) = PS∼Q̃N ( Pure ( S ) ∧ Small ( S ) ) PS∼Q̃N ( Pure ( S ) ) ≥ PS∼Q̃N ( Pure ( S ) ) − PS∼Q̃N ( ¬Small ( S ) ) PS∼Q̃N ( Pure ( S ) ) ≥ PS∼Q̃N ( Pure ( S ) ) − δ PS∼Q̃N ( Pure ( S ) ) = 1− δ PS∼Q̃N ( Pure ( S ) ) ≥ 1− 4δ .

Summary: This paper studied the Donsker-Varadhan lower bound of KL-divergence. The authors show that with high probability, the DV lower bound is upper bounded by log of the sample size, so if the true KL-divergence is very large, then exponential sample size is needed to make the DV lower bound tight. The same argument holds true for any distribution-free high-confidence lower bound (such as DV lower bound) for KL divergence. Then the authors proposed to use an upper bound for entropy instead of lower bound for mutual information.