Improved Mutual Information Estimation

We propose a new variational lower bound on the KL divergence and show that the Mutual Information ( MI ) can be estimated by maximizing this bound using a witness function on a hypothesis function class and an auxiliary scalar variable . If the function class is in a Reproducing Kernel Hilbert Space ( RKHS ) , this leads to a jointly convex problem . We analyze the bound by deriving its dual formulation and show its connection to a likelihood ratio estimation problem . We show that the auxiliary variable introduced in our variational form plays the role of a Lagrange multiplier that enforces a normalization constraint on the likelihood ratio . By extending the function space to neural networks , we propose an efficient neural MI estimator , and validate its performance on synthetic examples , showing advantage over the existing baselines . We then demonstrate the strength of our estimator in large-scale self-supervised representation learning through MI maximization . 1 INTRODUCTION . Mutual information ( MI ) is an ubiquitous measure of dependency between a pair of random variables , and is one of the corner stones of information theory . In machine learning , the information maximization principle for learning representation from unlabeled data through self-supervision ( Bell & Sejnowski , 1995 ) motivated the development of many MI estimators and applications ( Hjelm et al. , 2019 ; Noroozi & Favaro , 2016 ; Kolesnikov et al. , 2019 ; Doersch et al. , 2015 ; van den Oord et al. , 2018b ; Hu et al. , 2017 ) . The information bottleneck ( Tishby et al. , 1999 ; Kolchinsky et al. , 2017 ) is another principle that triggered recent interest in mutual information estimation . MI is also used to understand the information flow in neural networks , in learning clusters ( Krause et al. , 2010 ) and in regularizing the training of Generative Adversarial Networks ( GANs ) ( Chen et al. , 2016 ) . In many of those machine learning applications and other scientific fields , one has to estimate MI given samples from the joint distribution of high dimensional random variables . This is a challenging problem and many methods have been devised to address it . Since MI is defined as the KullbackLeibler ( KL ) divergence between the joint distribution and the product of marginals , one can leverage non parametric estimators of f-divergences ( Nguyen et al. , 2008 ; Nowozin et al. , 2016 ; Sriperumbudur et al. , 2009 ) . Specifically of interest to us is the Donsker-Varadhan ( DV ) representation of the KL divergence ( Donsker & Varadhan , 1976 ) that was used recently with neural networks estimators ( Belghazi et al. , 2018 ; Poole et al. , 2019 ) . Other approaches to estimating the MI are through finding lower bounds using variational Bayesian methods ( Alemi et al. , 2016 ; 2017 ; Barber & Agakov , 2003 ; Blei et al. , 2017 ) , as well as through geometric methods like binning ( Kraskov et al. , 2004 ) . In this paper we propose a new estimator of MI that can be used in direct MI maximization or as a regularizer , thanks to its unbiased gradients . Our starting point is the DV lower bound of the KL divergence that we represent equivalently via a joint optimization that we call η-DV on a witness function f and an auxiliary variable η in Section 2 . In Section 3 , we show that when the witness function f is learned in a Reproducing Kernel Hilbert Space ( RKHS ) the η-DV problem is jointly convex in both f and η . The dual of this problem sheds the light on this estimator as a constrained ratio estimation where η plays the role of a Lagrange multiplier that ensures proper normalization of the likelihood ratio . We also show how the witness function can be estimated as a neural network akin to Belghazi et al . ( 2018 ) . We specify our estimator for MI in Section 4 , and show how it compares to alternatives in the literature ( Nguyen et al. , 2008 ; Belghazi et al. , 2018 ; Poole et al. , 2019 ) . The experiments are presented in Section 5 . On synthetic data , we validate our estimators by estimating MI on Gaussian variables and by regularizing GAN training as in Chen et al . ( 2016 ) . On real data , we explore our estimator in deep MI maximization for learning representation from unlabeled data . Figure 1 shows an overview of all the bounds and related MI estimators discussed in this paper . 2 LOWER BOUNDS ON KL DIVERGENCES AND MUTUAL INFORMATION . Consider two probability distributions P and Q , where P is absolutely continuous w.r.t . Q . Let p and q be their respective densities defined on X ⊂ Rd . Their KL divergence is defined as KL ( P , Q ) = Ex∼P log ( p ( x ) q ( x ) ) = ∫ p ( x ) log ( p ( x ) q ( x ) ) dx . We are interested in the MI between two random variables X , Y where X is defined on X ⊂ Rdx , and Y on Y ⊂ Rdy . Let pxy be their joint densities and px , py the marginals of X and Y respectively . The MI is defined as follows : I ( X ; Y ) = KL ( pxy , pxpy ) , ( 1 ) which is the KL divergence between the joint density and the product of marginals . Non-parametric estimation of MI from samples is an important problem in science and machine learning . In what follows , we review variational lower bounds on KL to enable such estimation . Variational Characterization of KL divergence . Let H be any function space mapping X to R. The first variational characterization of the KL divergence goes back to Donsker & Varadhan ( 1976 ) : KL ( P , Q ) ≥ DHDV ( P , Q ) = sup { Ex∼Pf ( x ) − log ( Ex∼Qef ( x ) ) : f ∈H } , ( 2 ) where the equality holds if and only if ( iif ) f∗ = log ( p/q ) ∈H . We refer to this bound as the DV bound . The second variational representation was introduced in Nguyen et al . ( 2008 ) ; Nowozin et al . ( 2016 ) and derived through convex duality to be finally stated as follows : KL ( P , Q ) ≥ DHf ( P , Q ) = 1 + sup { Ex∼Pf ( x ) − Ex∼Qef ( x ) : f ∈H } , ( 3 ) with equality iif f∗ = log ( p/q ) ∈H . We call this bound the f -div bound ( as in f -divergence ) . From Eq . 2 and Eq . 3 we see that the variational bounds are attempting to estimate the log-likelihood ratio f∗= log ( p/q ) , and the tightness of the bound depends on the representation power of H . In order to compare these two lower bounds , observe that log ( t ) ≤ t − 1 , t > 0 . Therefore log ( Ex∼Qef ( x ) ) ≤ Ex∼Qef ( x ) − 1 which means that for any function space H we have : KL ( P , Q ) ≥ DHDV ( P , Q ) ≥ DHf ( P , Q ) , ( 4 ) from which we conclude that the DV bound is tighter than the f -div bound . Now , given samples { xi , i= 1 . . . N , xi ∼ P } , { yi , i= 1 . . . N , yi ∼ Q } , estimating the KL divergence can be done by computing the variational bound from Monte-Carlo simulation . Specifically , for the DV bound we have the following estimator : D̂HDV ( P , Q ) = sup { 1 N N∑ i=1 f ( xi ) − log ( 1 N N∑ i=1 ef ( yi ) ) : f ∈H } . ( 5 ) Note that the Mutual Information Neural Estimator ( MINE ) ( Belghazi et al. , 2018 ) considered the above formulation with the hypothesis class H being a neural network . For the f -div bound we have similarly the following estimator : D̂Hf ( P , Q ) = 1 + sup { 1 N N∑ i=1 f ( xi ) − 1 N N∑ i=1 ef ( yi ) : f ∈H } , ( 6 ) for which Nguyen et al . ( 2008 ) introduced and studied a convex estimator with H being an RKHS . While DV bound is tighter than the f -div bound , in order to learn the function f using stochastic gradient optimization , f -div is a better fit because the cost function is linear in the expectation , whereas in the DV bound we have a log non-linearity being applied to the expectation . This nonlinearity introduces biases in the mini-batch estimation of the cost function as noted in Belghazi et al . ( 2018 ) . In the following , we show how to alleviate this problem and remove the non-linearity at the price of an additional auxiliary variable that will enable better optimization for the DV bound . An η-trick for the DV Bound . We start with the following elementary Lemma , that gives a variational characterization of the log . All proofs are given in the Appendix A. Lemma 1 . Let x > 0 , we have : log ( x ) = minη e−ηx+ η − 1 . Using Lemma 1 we can now linearize the log in the DV bound . Lemma 2 ( η−Donsker-Varadhan ) . Let H be any function space mapping X to R : KL ( P , Q ) ≥ DHη-DV ( P , Q ) = − inf { L ( f , η ) : f ∈H , η ∈ R } ( 7 ) L ( f , η ) = e−ηEx∼Qef ( x ) − Ex∼Pf ( x ) + η − 1 , ( 8 ) We refer to this bound as η-DV bound . Note that for η=0 we recover the f -div bound . Using Lemma 2 , we can now rewrite the estimator for the DV bound in Eq . 5 as follows : D̂HDV ( P , Q ) = − inf f , η L̂ ( f , η ) , where L̂ ( f , η ) = e−η 1 N N∑ i=1 ef ( yi ) − N∑ i=1 f ( xi ) + η − 1 , ( 9 ) enabling unbiased stochastic gradient optimization of the function f . We note that similar variational tricks of non-linearities have been devised for g ( η ) = √ η in Argyriou et al . ( 2008 ) ; Bach et al . ( 2011 ) . 3 η-DV BOUND , CONVEXITY , MMD RATIO ESTIMATES AND LAGRANGIANS We established that the DV bound is tighter than the f -div and derived η-DV as an alternative to DV , enabling unbiased stochastic optimization . However , the role of η in improving the lower bound remains unclear . To highlight its role , we consider the η-DV bound from Eq . 7 , and restrict the hypothesis function class to an RKHS , similarly to Nguyen et al . ( 2008 ) . Working in the dual , we can compare η-DV estimator to the f -div estimator and further understand this hierarchy of lower bounds . See Figure 2 for an illustration . For simplicity , and without loss of generality , we let RKHS be a finite dimensional feature map , i.e. , H = { f |f ( x ) = 〈w , Φ ( x ) 〉 , Φ : X → Rm , w ∈ Rm } . Now for f ∈H , the loss given in Eq . 8 for η-DV can be rewritten as follows : L ( f , η ) =∆ L ( w , η ) = e−ηEx∼Qe〈w , Φ ( x ) 〉 − 〈w , Ex∼PΦ ( x ) 〉+ η − 1 . ( 10 ) Following Nguyen et al . ( 2008 ) , we consider the following regularized loss : L ( w , η ) = L ( w , η ) + Ω ( w ) , and the corresponding sample-based formulation L̂ ( w , η ) = L̂ ( w , η ) + Ω ( w ) , where Ω ( w ) is a convex regularizer , e.g. , Ω ( w ) = λ2 ‖w‖ 2 2 . The η-DV primal problem can now be defined as : η-DV–P : min w , η L ( w , η ) + Ω ( w ) . ( 11 ) In the following , we show that L ( w , η ) is jointly convex in ( w , η ) and derive its dual , which will shed light on the nature of the likelihood ratio estimate as well as the role of η. Convex Estimate in RKHS . In Lemma 3 we first establish the convexity of the η-DV loss function . Lemma 3 . L and L̂ are jointly convex in w and η. L̂ε ( u , η ) = L̂ ( u , η ) +Ω ( u ) +εe−η.Our KL estimate is therefore given as D̂HDV ( P , Q ) = −L̂ ( w∗ , η∗ ) , where ( w∗ , η∗ ) = arg min ( u , η ) L̂ε ( u , η ) . η-DV Dual is a Constrained Likelihood Ratio Estimation . Another angle for estimating the KL divergence is through likelihood ratio estimation ( Mohamed & Lakshminarayanan , 2016 ) . Given a ratio estimate r̂ of p/q , the KL divergence can be computed as EQr̂ log ( r̂ ) , which can be easily estimated using samples from Q . In Theorem 1 , proven in Appendix A , we show that the dual problem ( denoted D ) , corresponding to the primal minimization problem ( denoted P ) of η-DV , reduces to a constrained likelihood ratio estimation problem ( denoted C ) . Theorem 1 . Let Ω∗ ( . ) be the Fenchel conjugate of Ω ( . ) . The η-DV bound restricted to an RKHS , amounts to the following regularized convex minimization : P = − ( minw , η L ( w , η ) + Ω ( w ) ) , that has the following dual form : D = min r > 0 max η ∫ X r ( y ) log ( r ( y ) ) q ( y ) dy + ( η − 1 ) ( ∫ X r ( y ) q ( y ) dy − 1 ) + Ω∗ ( ∆ ( r ) ) , where ∆ ( r ) = ∫ X Φ ( x ) p ( x ) dx− ∫ X r ( y ) q ( y ) Φ ( y ) dy . Noticing that η − 1 plays the role of a Lagrangian multiplier , this is equivalent to the following likelihood ratio estimation problem : C = min r > 0 ∫ X r ( y ) log ( r ( y ) ) q ( y ) dy + Ω∗ ( ∫ X Φ ( x ) p ( x ) dx− ∫ X r ( y ) q ( y ) Φ ( y ) dy ) subject to : ∫ X r ( y ) q ( y ) dy = 1 . Therefore , we have P = D = C. Let ( w∗ , η∗ ) be an optimizer of P . Let r∗ be an optimizer of D , then the KL estimate is given by : DHDV ( P , Q ) = ∫ X r ∗ ( y ) log ( r∗ ( y ) ) q ( y ) dy = L ( w∗ , η∗ ) . The regularizer Ω ( w ) = λ2 ‖w‖ 2 can now be given the following interpretation based on the results of Theorem 1 . Recall the definition of the MMD ( Gretton et al. , 2012 ) between distributions using an RKHS : MMDΦ ( P , Q ) = ‖Ex∼PΦ ( x ) − Ey∼QΦ ( y ) ‖ . Replacing the Fenchel conjugate Ω∗ ( . ) by its expression in Theorem 1 , we see that η-DV is equivalent to the following dual ( η-DV–D ) η-DV–D : min r > 0 max η ∫ X r ( y ) log ( r ( y ) ) q ( y ) dy+ ( η−1 ) ( ∫ X r ( y ) q ( y ) dy − 1 ) + 1 2λ MMD2Φ ( rq , p ) ( 12 ) which can be written as a constrained ratio estimation problem ( η-DV–C ) η-DV–C : min r > 0 ∫ X r ( y ) log ( r ( y ) ) q ( y ) dy + 1 2λ MMD2Φ ( rq , p ) , subject to : ∫ X r ( y ) q ( y ) dy = 1 . ( 13 ) Hence , it is clear now that the η-DV optimization problem is equivalent to the constrained likelihood ratio estimation problem r given in Eq . 13 , where the ratio is estimated using the MMD distance in the RKHS between rq and p. It is also easy to see that p/q is a feasible point and for the universal feature map MMDΦ ( rq , p ) = 0 iif r = p/q , therefore , for a universal kernel , p/q is optimal and we recover the KL divergence for r = p/q . When comparing η-DV–D from Eq . 12 and η-DV–C from Eq . 13 , we see that η−1 plays the role of a Lagrangian that ensures that rq is indeed a normalized distribution . In practice , we solve the primal problem ( P ) , while the dual problem ( D ) and its equivalent constrained form ( C ) explain why this formulation estimates the KL divergence and the role of η as a Lagrangian multiplier that enforces a normalization constraint . Let r∗ be the solution , then we have : DHDV ( P , Q ) = ∫ X r∗ ( y ) log ( r∗ ( y ) ) q ( y ) dy = −L ( w∗ , η∗ ) . ( 14 ) For comparison , the f -div bound , restricted to an RKHS , is equivalent to the following ratio estimation ( follows from the proof of Theorem 1 by eliminating max on η , and setting η= 0 ) , and is consistent with the results of Nguyen et al . ( 2008 ) ( see Eq . 51 therein ) : f -div : min r > 0 ∫ X r ( y ) log ( r ( y ) ) q ( y ) dy + 1− ∫ X r ( y ) q ( y ) dy + 1 2λ MMD2Φ ( rq , p ) . ( 15 ) Let r∗ be the optimum , then the KL divergence can be estimated as follows Nguyen et al . ( 2008 ) : DHf ( P , Q ) = ∫ X r∗ ( y ) log ( r∗ ( y ) ) q ( y ) dy + ( 1− ∫ X r∗ ( y ) q ( y ) dy ) . ( 16 ) Comparing DHDV ( P , Q ) and DHf ( P , Q ) we see that they both have a relative entropy term but the f -div bound does not impose the normalization constraint on the ratio , which biases the estimate . We end with the following remark : In Theorem 1 , if we replace the loss L by its empirical counterpart L̂ from Eq . 9 , the equivalent Dual takes the following form : min ri > 0 1 N N∑ i=1 ri log ( ri ) + Ω ∗ ( 1 N N∑ i=1 Φ ( xi ) − 1 N N∑ i=1 riΦ ( yi ) ) , subject to : 1 N N∑ i=1 ri = 1 , and the KL estimate is given by : D̂HDV ( P , Q ) = 1N ∑N i=1 r ∗ i log ( r ∗ i ) = −L̂ ( w∗ , η∗ ) . From RKHS to Neural Estimation . One shortcoming of the RKHS approach is that the method depends on the choice of feature map Φ . We propose to learn Φ as a deep neural network as in MINE ( Belghazi et al. , 2018 ) . Given samples xi from P , yi from Q , the KL estimation problem becomes : D̂NNDV ( P , Q ) = − ( min Φ min η , w e−η 1 N N∑ i=1 e〈w , Φ ( yi ) 〉 − N∑ i=1 〈w , Φ ( xi ) 〉+ η − 1 ) , ( 17 ) which can be solved using BCD on ( w , η , Φ ) . Note that if Φ ( · ) is known and fixed , then optimization problem in Eq . 17 becomes convex in η and w. We refer to this bound as η-DV–convex . What Do Neural Estimators of KL or MI Learn ? We conclude with the following observations : 1 . Ratio estimation via Feature Matching . KL divergence estimation with variational bounds boils down to a ratio estimation using a form of MMD matching . 2 . Choice of Feature/Architecture . The choice of the feature space or the architecture of the neural network introduces a bias in the ratio estimation ; also observed in Tschannen et al . ( 2019 ) . 3 . Ratio Normalization . η-DV bound introduces a normalization constraint on the ratio , that ensures a better consistent estimate . 4 . Impact of Regularizers . The choice of the regularizer dictates the choice of the metric . For instance , Ozair et al . ( 2019 ) used gradient penalties as a regularizer , and this corresponds to a ratio matching in the Sobolev discrepancy sense . Please see Appendix A.1 for more details . 4 η-DV MUTUAL INFORMATION ESTIMATION We now specify the KL estimators given in Section 3 for the MI estimation problem . Given a function space H defined on X × Y , I ( X ; Y ) ≥ IHDV ( X ; Y ) = sup f∈H Epxyf ( x , y ) − log ( EpxEpyef ( x , y ) ) ≥ IHf -div ( X ; Y ) , ( 18 ) where IHf -div ( X ; Y ) =supf∈H Epxyf ( x , y ) − EpxEpyef ( x , y ) . Equivalently , with the η-trick we have : IHDV ( X ; Y ) = − inf f∈H , η e−ηEpxpyef ( x , y ) − Epxyf ( x , y ) + η − 1 . ( 19 ) Now , given iid samples from marginals xi ∼ px and ỹi ∼ py , i = 1 . . . N , and samples from the joint ( xi , yi ) ∼ pxy , i = 1 . . . N , we can estimate the MI as follows : ÎHDV ( X , Y ) = D̂ H DV ( px , y , pxpy ) = − inf f∈H , η e−η 1 N N∑ i=1 ef ( xi , ỹi ) − 1 N N∑ i=1 f ( xi , yi ) + η − 1 , ( 20 ) when H is an RKHS , η can be seen as a Lagrangian ensuring that the ratio estimation r ( x , y ) of pxy pxpy is normalized when integrated on pxpy , i.e. , η is a Lagrangian associated with the constraint∫ X×Y r ( x , y ) px ( x ) py ( y ) dxdy = 1 . In Table 1 we review other variational bounds for MI based on f -div , DV , and η-DV bounds ( a-MINE from Poole et al . ( 2019 ) is discussed next ) . Improved Donsker-Varadhan for Mutual Information and a-DV Bound Discussion . In the following Lemma we show that for the particular case of MI estimation , the DV bound can be made tighter : Lemma 4 . For any function g that maps X × Y to R+ we have : KL ( px , y , pxpy ) + Epx log ( Epyg ( x , y ) ) ≥ Epx , y log ( g ( x , y ) ) Therefore : I ( X ; Y ) = supf∈G ( X×Y ) Epx , yf ( x , y ) − Epy log ( Epxef ( x , y ) ) where G ( X × Y ) is the entire space of functions defined on X × Y . For any function space H map X × Y to R we have : I ( X ; Y ) ≥ IHa-DV ( X ; Y ) : = sup f∈H Epx , yf ( x , y ) − Epy log ( Epxef ( x , y ) ) . Using Jensen ’ s inequality , we can easily derive the following hierarchy of lower bounds : I ( X ; Y ) ≥ IHa-DV ( X ; Y ) ≥ IHη-DV ( X ; Y ) ≥ IHf -div ( X ; Y ) . ( 21 ) In Theorem 2 , we apply the η-trick of Lemma 1 to IHa-DV ( X ; Y ) , where we replace each log ( Epxef ( x , y ) ) using the variational form of log which leads to η ( y ) . The interchangeability between minimization and expectation and going from scalar η for each y to a function η ( . ) needs some care and we prove it in the Appendix B.1 , appealing to the interchangeability principle ( Lemma 1 in Dai et al . ( 2016 ) ) . Theorem 2 ( η-trick for Improved DV Bound ) . Let H be a space of bounded functions defined on X ×Y we have : I ( X ; Y ) ≥ − inff∈H , η∈G ( Y ) Epx.pyef ( x , y ) −η ( y ) +Epyη ( y ) − 1−Epx , y ( f ( x , y ) ) , where G ( Y ) is the entire space of functions defined on Y . Let Hη be a function space subset of G ( Y ) we obtain : I ( X ; Y ) ≥ IHa-DV ( X ; Y ) = sup f∈H , η ( . ) ∈Hη Epx , yf ( x , y ) − Epx.pyef ( x , y ) −η ( y ) − Epyη ( y ) + 1 . ( 22 ) Hence , we recover the lower bound given in Poole et al . ( 2019 ) ( given in Table 1 ) , that was derived through other variational principles . In the following , we make a few remarks about IHa-DV ( X ; Y ) : 1 . The IHa-DV ( X ; Y ) bound , as given in Poole et al . ( 2019 ) , can be also obtained by setting η ( y ) = log ( a ( y ) ) , a ( y ) > 0 . While this seems equivalent , the restriction of a to R+ ( η is in R ) changes the landscape of the optimization problem and the interchangeability principle does not apply in a straightforward way . 2 . Poole et al . ( 2019 ) refers to a ( y ) as a “ critic ” ( as in the reinforcement learning literature ) . As shown in our analysis in Section 3 , when we restrict H to an RKHS , η ( y ) plays a role of a Lagrangian multiplier or a dual potential in a ratio estimation problem . In this case it is enforcing multiple constraints ∫ X r ( x , y ) p ( x ) dx = 1 for each y , instead of imposing a single constraint∫ X ∫ Y r ( x , y ) p ( x ) p ( y ) dxdy=1 as in the I H η-DV ( X ; Y ) bound . 3 . No Free Lunch : while IHa-DV ( X ; Y ) is tighter than our I H η-DV ( X ; Y ) bound , it comes at the price of estimating a function η ( y ) together with the witness function f . This triggers a higher computational cost and larger estimation errors . 4 . We show in Appendix C how IHa-DV ( X ; Y ) can be extended to conditional MI estimation . 5 . Generality : η-DV bound is applicable for general probability distributions P and Q : KL ( P , Q ) ≥ DHη-DV ( P , Q ) , while IHa-DV ( X ; Y ) only works for P = pxy and Q = pxpy . Finally , in Algorithm 1 we outline the steps of η-MINE for MI estimation . Algorithm 1 η-MINE ( Stochastic BCD ) Inputs : X , Y dataset X ∈ RN×dx , Y ∈ RN×dy , such that ( xi = Xi , . , yi = Yi , . ) ∼ pxy Hyperparameters : αη , αθ ( learning rates ) , nc ( number of critic updates ) Initialize η , θ parameter of the neural network fθ for i = 1 . . .Maxiter do for j = 1 . . . nc do Fetch a minibatch of size N ( xi , yi ) ∼ pxy Fetch a minibatch of size N ( xi , ỹi ) ∼ pxpy { ỹi obtained by permuting rows of Y } Evaluate L̂ ( fθ , η ) = e−η 1N ∑N i=1 e fθ ( xi , ỹi ) − 1N ∑N i=1 fθ ( xi , yi ) + η − 1 Stochastic Gradient step on θ : θ ← θ − α∂L̂ ( fθ , η ) ∂θ { We use ADAM } end for Update η : η ← η − αη ∂L̂ ( fθ , η ) ∂η end for Output : fθ , η , ÎHη-DV ( X , Y ) = 1 N ∑N i=1 fθ ( xi , yi ) − e−η 1N ∑N i=1 e fθ ( xi , ỹi ) − η + 1

Summary: This paper develops variations on the Donsker-Varadhan (DV) lower bound on KL divergence which yields a lower bound on mutual information as a special case.  The paper is primarily theoretical with an emphasis on the case where the witness function f is drawn from an RKHS (a support vector machine).  They discuss, and present experimental results where the feature map of the RKHS is computed by a neural network, although the theoretical results largely do not apply to optimization of the neural network.