Understanding the Limitations of Variational Mutual Information Estimators
Authors: Jiaming Song
Stefano Ermon
Venue:  (2020)

Abstract: Variational approaches based on neural networks are showing promise for estimating mutual information (MI) between high dimensional variables. However, they can be difficult to use in practice due to poorly understood bias/variance tradeoffs. We theoretically show that, under some conditions, estimators such as MINE exhibit variance that could grow exponentially with the true  amount of underlying MI. We also empirically demonstrate that existing estimators fail to satisfy basic self-consistency properties of MI, such as data processing and additivity under independence. Based on a unified perspective of variational approaches, we develop a new estimator that focuses on variance reduction. Empirical results on standard benchmark tasks demonstrate that our proposed estimator exhibits improved bias-variance trade-offs on standard benchmark tasks.

Published as a conference paper at ICLR 2020
UNDERSTANDING THE LIMITATIONS OF VARIATIONAL MUTUAL INFORMATION ESTIMATORS
Jiaming Song & Stefano Ermon Stanford University (tsong, ermon)@cs stanford.edu
ABSTRACT
Variational approaches based on neural networks are showing promise for estimat- ing mutual information (MI) between high dimensional variables. However, they can be difficult to use in practice due to poorly understood bias/variance tradeoffs. We theoretically show that, under some conditions, estimators such MINE ex- hibit variance that could grow exponentially with the true amount of underlying MI. We also empirically demonstrate that existing estimators fail satisfy basic self-consistency properties of MI, such as data processing and additivity under in- dependence. Based on unified perspective of variational approaches, we develop a new estimator that focuses on variance reduction. Empirical results on standard benchmark tasks demonstrate that our proposed estimator exhibits improved bias- variance trade-offs on standard benchmark tasks.
INTRODUCTION
Mutual information (MI) estimation and optimization are crucial to many important problems in machine learning, such as representation learning (Chen al., 2016; Zhao al., 2018b; Tishby & Zaslavsky, 2015; Higgins et al., 2018) and reinforcement learning (Pathak et al., 2017; van den Oord al., 2018). However, estimating mutual information from samples challenging (McAllester & Statos, 2018) and traditional parametric and non-parametric approaches (Nemenman et al., 2004; Gao et al., 2015; 2017) struggle scale up to modern machine learning problems, such as estimating the MI between images and learned representations.
Recently, there has been a surge of interest in MI estimation with variational approaches (Barber & Agakov, 2003; Nguyen et al., 2010; Donsker & Varadhan, 1975), which can be naturally com- bined with deep learning methods (Alemi et al., 2016; van den Oord et al., 2018; Poole et al., 2019). Despite their empirical effectiveness in downstream tasks such as representation learning (Hjelm et al., 2018; Velicković et al., 2018), their effectiveness for MI estimation remains unclear. In par- ticular, higher estimated MI between observations and learned representations do cate improved predictive performance when the representations are used for downstream supervised learning tasks (Tschannen et al., 2019).
to indi-
In this paper, we discuss two limitations of variational approaches to MI estimation. First, we theoretically demonstrate that the variance of certain estimators, such MINE (Belghazi et al., 2018), could grow exponentially with the ground truth MI, leading to poor bias- varianai trade-offs. Second, we propose set of self-consistency tests over basic properties of MI. and empirically demonstrate that all considered variational estimators fail to satisfy critical properties of MI, such as data processing and additivity under independence. These limitations challenge the effectiveness of these methods for estimating optimizing MI.
To mitigate these issues, we propose unified perspective over variational estimators treating vari- ational MI estimation as optimization problem over (valid) density ratios. This view highlights the role of partition functions estimation, which is the culprit of high variance issues in MINE. To address this issue, propose improve MI estimation via variance reduction techniques for partition function estimation. Empirical results demonstrate that our estimators have much better bias-variand variance trade-off compared to existing methods standard benchmark tasks.

---

Published as a conference paper at ICLR 2020
2 BACKGROUND AND RELATED WORK
2.1 NOTATIONS
We use uppercase letters to denote probability measure (e.g.. P, Q) and corresponding lowercase letters to denote its density' functions (e.g., P. unless specified otherwise. We use X, Y to denote random variables with separable sample spaces denoted as X and y respectively, and P(X) (or P(J)) to denote the set of all probability measures over the Borel o-algebra on X (or 3).
Under Q E P(()), the p-norm of function r X R is defined as |||||p == (ʃ Ir(r
with ||r|] oo = limp-+co/1t |/rl/l The set locally p-integrable functions is defined as (P(Q) == (r X + R |r]pp ). The space of probability measures wrt. Q is defined A(Q)
{r (2) |r = 1, 2 0}; we also call this the space of "valid density ratios" wrt. Q. We use
P < Q denote that P is absolutely continuous with respect to Q. We use Ig to denote an estimator for I where we replace expectations with sample averages.
2.2 VARIATIONAL MUTUAL INFORAMTION ESTIMATION
The mutual information between two random variables X and Y is the KI. divergence between the joint and the product of marginals:
I(: = (K(X,
(1)
which we wish to estimate using samples from P(X, Y); in certain cases we may know the density of marginals (e.g. P.X)). There are wide range of variational approaches variational MI esti- mation. Variational information maximization uses the following result (Barber & Agakov, 2003);
Lemma (Barber-Agakov (BA)). For random variables X and Y: I(X;Y) sup (Erxx,) [log (o(l))) log (o=)( =: [BA()
(2)
where qs y + P(X) is valid conditional distribution over X' given E y and p(x:) is the probability density function of the marginal distribution P(X).
Another family of approaches perform MI estimation through variational lower bounds to KL di- vergences. For example, the Mutual Information Neural Estimator (MINE, Belghazi et al. (2018)) applies the following lower bound to KL divergences (Donsker & Varadhan, 1975).
Lemma 2 (Donsker- Varadahn (DV)). VP,Q E P(2)) such that P & Q,
(PII))
sup TEL=(Q)
[Ep[T] log Eo [e"] [MINE(T))
One could set p= P(X,Y) and Q P(X) T parametrized neural network (e.g. (a y) parametrized by 0), and obtain the estimate by optimizing the above objective via stochastic gradient descent over mini-batches. However, the corresponding estimator ÍMIN (where we replace the expectations in Eq. (3) with sample averages) is biased, leading to biased gradient estimates; Belghazi et al. (2018) propose to reduce bias via estimating the partition function Eqle"" with exponential moving averages of mini-batches.
The variational /diivergence estimation approach (Nguyen et al., 2010; Nowozin et al., 2016) con- siders lower bounds on f-divergences which can be specialize to KL divergence, and subsequently to mutual information estimation:
Lemma (Nyugen et al. (NWJ)). VP, Q E P(X) such that P & Q
DKL(PIQ) sup TEL
{Ep [T)
BoleT-11 =xwo((T)
(4)
and (kl(())Q) = (T) when T = ldg dQ) 1
In the remainder of the paper, slightly overload "density" for discrete random variables.
2

---

Published as a conference paper at ICLR 2020
The supremum over T is invertible function of the density ratio dP/ dQ, so could use this approach to estimate density ratios by inverting the function (Nguyen et al., 2010; Nowozin et al,, 2016; Grover & Ermon, 2017). The corresponding mini-batch estimator (denoted as (Nw.) is unbi- ased, unlike MINE, this approach does not require special care to reduce bias in gradients.
Contrastive Predictive Coding (CPC, van den Oord et al. (2018)) considers the following objective:
1
log fo y:)
[fpc(fa)= == (X,))
11 Eド== fo(o,, ( 8j)
==1
where So X' x' 20 is neural network parametrized by 0 and P"(X, X Y) denotes the joint pdf for i..d. random variables sampled from P(X, Y). CPC generally has less variance but is more biased because its estimate does not exceed log n. where n is the batch size (van den Oord al., 2018; Poole et al., 2019). While one further reduce the bias with larger n, the number of evaluations needed for estimating each batch with fo is n², which scales poorly. To address the high-bias issue of CPC, Poole al. proposed interpolation between LCPC and ΣNwJ to obtain more fine-grained bias-variance trade-offs.
3 VARIATIONAL MUTUAL INFORMATION ESTIMATION AS OPTIMIZATION OVER DENSITY RATIOS
In this section, we unify several existing methods for variational mutual information estimation. We first show that variational mutual information estimation can be formulated as a constrained optimization problem, where the feasible set is A(Q), i.e. the valid density ratios with respect to Q- Theorem 1, VP, Q E P(X) such that P & Q we have
DKL (PIIQ) = sup Ep
(6)
rEA(Q)
where the supremum is achived when dP/ dQ.
We defer the proof in Appendix A. The above argument works for KL divergence between general distributions, but in this paper focus the special case of mutual information estimation. For the remainder of the paper, we use P to represent the short-hand notation for the joint distribution P(X, Y) and use Q to represent the short-hand notation for the product of marginals P(X)
3.1 A SUMMARY OF EXISTING VARIATIONAL. METHODS
From Theorem 1, we can describe a general approach to variational MI estimation:
1. Obtain density ratio estimate denote the solution as r:
2. Project to be close to A(Q) in practice only have samples from Q, denote the solution as [(r; Q.). where Qn is the empirical distribution i.i.d. samples from Q:
3. Estimate mutual information with Epllog I(r; Qn)).
We illustrate two examples of variational mutual information estimation that can be summarized with this approach. In the case of Barber-Agakov, the proposed density ratio estimate is PBa (s(mly) / /(x:) (assuming that p(æ)) is known), which is guaranteed to be in A(Q) because
EEQ (ao(
= go (
1, IAA Qn) = TBA (7)
for all conditional distributions qø- In the case of MINE/ Donsker- Varadah, the logarithm of the density ratio is estimated with Tæ y); the corresponding density ratio might not be normalized, so one could apply the following normalization for samples:
Equ. To /EQ./E [o =
IMINE(e2e (n.) = eg [e
(8)
where Eq, To (the sample average) is an unbiased estimate of the partition function EglcTo];;
I IDv (e Qw.) E A(Q) is only true when n - oc. Similarly, we show ICpc is a lower bound to MI in Corollary 2, Appendix A, providing alternative proof to the one in Poole et al. (2019).
3

---

Published as a conference paper at ICLR 2020
These examples demonstrate that different mutual information estimators be obtained in pro- cedural manner by implementing the above steps, and one could involve different objectives at each step. For example, could estimate density ratio via logistic regression (Hjelm et al., 2018; Poole et al., 2019; Mukherjee et al., 2019) while using JNwJ or MINE to estimate MI. While logistic re- gression does not optimize for lower bound for KL. divergence, it provides density ratio estimates between P' and Q which could be used for subsequent steps.
Table 1: Summarization of variational estimators of mutual information. The E A(Q) column denotes whether the estimator is valid density ratio wrt. Q. (V) means any parameterization is valid; (n + 20) means any parameterization valid as the batch size grows infinity; (tr -> 00) means only the optimal parametrization is valid (infinite training cost).
Category
Estimator
Params
Qn):Qn)
E A(Q)
v
Gen.
faM (Eq. (9))
po: Ps. Pe
po(ar, /) (2)
tr - 00
ÍMINE
Tg
cTo(..)) /Eq, (eT
Disc.
fcPc
So
fo(æ, y) T8 (x) [Fo v)] (T( /Bq.. (
/
ISMILE (Eq. (17))
Tg. T
n2 T 00
3.2 GENERATIVE AND DISCRIMINATIVE APPROACHES TO MI ESTIMATION
The above discussed variational mutual information methods can be summarized into two broad categories based on how the density ratio is obtained.
The discriminative approach estimates the density ratio dP/dQ directly; examples include the MINE, NWJ and CPC estimators.
The generative approach estimates the densities of P and Q separately; examples include the BA estimator where a conditional generative model learned. In addition, we describe generative approach that explicitly learns generative models (GM) for P(X,Y) P(X) and P(Y):
(po; P4: Po) Ep.log po y) log (a) log Pa (y)),
where pa. Ps. Py are maximum likelihood estimates of P(X,Y) P(X) and P(Y) respec- tively. We can learn the three distributions with generative models, such VAE (Kingma Welling, 2013) or Normalizing flows (Dinh et al., 2016), from samples.
We summarize various generative and discriminative variational estimators in Table 1.
Differences between two approaches While both generative and discriminative approaches can be summarized with the procedure in Section 3.1, they imply different choices in modeling, estima- tion and optimization.
On the modeling side, the generative approaches might require stringent assumptions on the architectures (e.g. likelihood or evidence lower bound is tractable), whereas the discriminative approaches do not have such restrictions.
On the estimation side, generative approaches do not need to consider samples from the product of marginals P(X) (since it can model P(X, Y), P(X), P(Y) separately), yet the discriminative approaches require samples from P(X) P(Y); if we consider a mini- batch of size n, the number of evaluations for generative approaches is 2(n) whereas that for discriminative approaches it could be (())).
On the optimization side, discriminative approaches may need additional projection steps to close to (QQ) (such as IMIE), while generative approaches might not need to perform this step (such J.
4

---

Published as a conference paper at ICLR 2020
4 LIMITATIONS OF EXISTING VARIATIONAL ESTIMATORS
4.1 GOOD DISCRIMINATIVE ESTIMATORS REQUIRE EXPONENTIALLY LARGE BATCHES
In the iNWJ and IMINE estimators, one needs to estimate the "partition function" Eq[r] for some density ratio estimator r: for example, MINE needs this in order to perform the projection step [MINE(T, Qx) in Eq (8). Note that the [NwJ and IMINE lower bounds maximized when r takes the optimal value T d dQ. However, the sample averages IMIE and INWJ of Eq(r*] could have a variance that scales exponentially with the ground-trute MI; we show this in Theorem 2.
Theorem 2. Assume that the ground truth density ratio r* = dP/ dQ and Varo(r") exist. Let Qn denote the empirical distribution of n. i.i.d. samples from Q and let Eq, denote the sample average over Qn. Then under the randommess of the sampling procedure, we have:
(Dx((P1Q)
Varq Eq
(10)
lim naraoloog llog Eq,, [r*]] (xL(P110) -1. 200
(11)
We defer the proof to Appendix A. Note that in the theorem above, we assume the ground truth density ratio r" already obtained, which is the optimal ratio for NWJ and MINE estimators. As a natural consequence, the NWJ and MINE estimators under the optimal solution could exhibit variances that grow exponentially with the ground truth MI (recall that our context MI is KL divergence). One could achieve smaller variances with some of r', but this guarantees looser bounds and higher bias.
Corollary 1. Assume that the assumptions in Theorem 2 hold. Let P.n and Qn be the empirical distributions of m ii.d samples from P and n i.idd samples from Q, respectively. Define
Eq_ [r*] I I == Er [log r"] ] log Eq, [r*] MINE
NWJ = Pm, [log r*
(12)
(13)
where r" = aP/dQ. Then under the randomness of the sampling procedure, we have Vm E N:
Varp NwJ] 2 (e -1)nn
(14)
lim n Var [ MINE (xa(P11() -
(15)
R-+00
This high variance phenomenon has been empirically observed in Poole et al. (2019) (Figure 3) for INwJ under various batch sizes, where the log-variance scales linearly with MI. We also demonstrate this in Figure (Section 6) In order to keep the variance of IIIN and INw. relatively constant with growing MI, one would need batch size of n ((ePPa.((P1))) Ícpc has small variance, but it would need 2 (Paa(PlIQ)) to have small bias, as its estimations are bounded by log n.
4.2 S#LF-CONSISTENCY ISSUES FOR MUTUAL INFORMATION ESTIMATORS
we consider X, y to be gigh-dimnnsional, estimation mutual information becomes more diffi- cult. The density ratio between P(X,Y) Y) and P(X)P(Y) could be very difficult estimate from fi- nite samples without proper parametric assumptions (McAllester & Statos, 2018; Zhao et al., 20) Additionally, the exact value of mutual information is dependent on the definition of the sample space; given finite samples, whether the underlying random variable is assumed to be discrete or continuous will lead to different measurements of mutual information (corresponding to entropy and differential entropy, respectively).
In machine learning applications. however, we are often more interested in maximizing or min- imizing mutual information (estimates), rather than estimating its exact value. For example. if an estimator is off by a constant factor, it would still useful for downstream applications, even though it can be highly biased. To this end, we propose a set of self-consistency tests for any MI estimator i. based properties of mutual information:
1. (Independence) if X and Y are independent, then f(X;Y) 0:
5

---

Published as a conference paper at ICLR 2020
2. (Data processing) for all functions g. h, ((; 2 I(g(X)))(Y)) and f(X;Y) x
T((X, (YY)) where [;-]] denotes concatenation.
3. (Additivity) denote X,, X2 as independent random variables that have the same distribution as X (similarly define Y1, Y2), then f([X1, X2]; [Y,, Y2]) x 2 Y(.
These properties holds under both entropy and differential entropy, so they do not depend on the choice of the sample space. While these conditions are necessary but obviously not sufficient for accurate mutual information estimation, we argue that satisfying them is highly desirable for appli- cations such as representation learning (Chen al., 2016) and information bottleneck (Tishby & Zaslavsky, 2015). Unfortunately, none of the MI estimators we considered above pass all the self- consistency tests when X, Y are images, as we demonstrate below Section 6.2. particular, the generative approaches perform poorly when MI is low (failing in independence and data process- ing), whereas discriminative approaches perform poorly when MI is high (failing in additivity).
IMPROVED MI ESTIMATION VIA CLIPPED DENSITY RATIOS
To address the high-variance issue in the INwJ and IIIN estimators, we propose to clip the density ratios when estimating the partition function. We define the following clip function:
clip(e, L, u) = max(min((())) u).
(16)
For an empirical distribution of n samples Qn instead of estimating the partition function via Eq.. [r), we instead consider Eq. [clip(r, e e-,,"" e²)] where 2 is a hyperparameter, this is equiva- lent to clipping the log density ratio estimator between - and т.
We can then obtain following estimator with smoothed partition function estimates:
ISMILE (To, -) = E. (To( y) log (clip(( ( (o(m.)))
,e"]]
(17)
where T3 is a neural network that estimates the log-density ratio (similar to the role of l in ÍMINE).
We term this the Smoothed Mutual Information "Lower-bound" Estimator (SMILE) with hyperpa- rameter т; ISMILE converges to IMIEE when t - oc. In our experiments, we consider learning the density ratio with logistic regression, similar to the procedure in Deep InfoMax (Hjelm al., 2018).
The selection of r affects the bias-van variance trade-off when estimating the partition function; with a smaller т, variance is reduced at the cost of (potentially) increasing bias. In the following theorems, we analyze the bias and variance in the worst case for density ratio estimators whose actual partition function is S for some 5 E (0, co).
Theorem 3. Let r(x) X' + Rz0 be any non-negative measurable function such that frQ S,
E (0, oc) and r(æ) E [0. ). Define rz( clip(r(z), eT, _e²) for finite, non-negative If
T < K, then the bias for using r. to estimate the partition function of satisfies:
- [ain]- S max (e-)1- Se
ck e S(c - ek -
if 2 K, then
Eq [r] Eglr+]l S (e Se-K).
Theorem 4. The variance of the estimator EQ,, [r-] (using n samples from Q) satisfies:
Var [r>]] e- --
(18)
4n
We defer the proofs to Appendix A. Theorems and 4 suggest that as we decrease т, variance is decreased at the cost of potentially increasing bias. However, if S is close to 1, then we could use small 5 values to obtain estimators where both variance and bias are small. We further discuss the bias-variand trade-off for fixed over changes of in Theorem 3 and Corollary 3.

---

Published as a conference paper at ICLR 2020
EXPERIMENTS
6.1 BENCHMARKING ON MULTIVARIATE GAUSSIANS
First, evaluate the performance of MI bounds Belghazi et al., 2018), where the ground truth MI is tractable. The first task (Gaussian) where (x, y) are drawn from 20-d Gaussian distribution with correlation p. and the second task (Cubic) is the same as Gaussian but we apply the transformation y + y². We consider three discrimina- tive approaches (Iccc, INWJ, ISMILE) and one generative approach (IGM). For the discriminative approaches, we consider the joint critic in (Belghazi et al., 2018) and the separate critic in (van den Oord al., 2018). For IGM we consider invertible flow models (Dinh al., 2016). We train all models for 20k iterations, with the ground truth mutual information increasing by 2 per 4k itera- tions. More training details are included in Appendix B
toy tasks detailed in (Poole et al., 2019;
5000
15
19.00
MM
I5KO 2008
500 JM0 1900 200
V0 JOWN 1506 2800
1890 (560 2008
530 3H08 1906 2000
CFC
100)
SHILE tt-
SYILE 5.00
SNLE (c
GN (Pov
Figure 1: Performance of mutual information estimation approaches Gaussian (top row) and Cubic (bottom row). Left two columns are IcPc and INWJ, next three columns are ISMILE with 1.0, 5.0, oo and the right column is IGM with flow models.
Figure 1 shows the estimated mutual information over the number of iterations. both tasks, ICpC has high bias and JNwJ has high variance when the ground truth MI is high, whereas ISMILE has relatively low bias and low variance across different architectures and tasks. Decreasing in the SMILE estimator decreases variances consistently but has different effects over bias; for example, under the joint critic bias is higher for r = 5.0 in Gaussian but lower in Cubic. IGM with flow models has the best performance on Gaussian, yet performs poorly on Cubic, illustrating the importance of model parametrization in the generative approaches.
Cubic
Variance
10
Blos
MSE
20
10-
10
MI
Figure 2:
Bias/ / Variance MSE of various estimators on Cubic (right). We display more results for Gaussian in Appendix B.
In Figure 2, compare the bias, variance and mean squared error (MSE) of the discriminative methods. We observe that the variance of INWJ increases exponentially with mutual information, On the other hand, the SMILE estimator is able
which is consistent with our theory in Corollary to achieve much lower variances with small - values; in comparison the variance of SMILE when oo is similar to that of JNwJ in Cubic. In Table 2, we show that ISMILE can have nearly two orders of magnitude smaller variance than INwJ while having similar bias. Therefore ISMILE enjoys lower MSE in this benchmark MI estimation task compared /NWJ and ICPC.
We release code in

---

Published as a conference paper at ICLR 2020
6.2 SELF-CONSISTENCY TESTS ON IMAGES
same image
different images
X =
5 ~
X
X =
S a
Y =
Y
Y
Sotting (baseline)
Setting 2 (data processing)
Setting (additivity)
Figure 3: Three settings in the self-consistency experiments.
Next, we perform our proposed self-consistency tests high-dimensional images (MNIST and CIFARIO) 10) under three settings, where the ground truth MI is difficult to obtain (if not impossible). These settings are illustrated in Figure 3.
1. The first setting is where X is an image and Y is the same image where we mask the bot- tom rows, leaving the top rows from X selected before evaluation). The rationale behind this choice of Y is twofold: 1) [(X; Y') should be non-decreasing with t; 2) it is eas- ier (compared to low-d representations) gain intuition about the amount of information remaining in Y.
2. In the second setting, X corresponds to two identical images, and Y the top t1, t2 rows of the two images (f1 2 t2); this considers the "data- "itta-processin" property.
3. In the third setting, X corresponds both; this considers the "additivity" property.
independent images, and Y the top rows of
We compare four approaches: IcPc, IMINE, ISMILE and IGM. We use the same CNN architecture for /cPc. MINE and /SMILE, and use VAEs (Kingma & Welling, 2013) 3) for ICM. We include more experimental details and alternative image processing approaches in Appendix B.
MNIST (Y X(rows))
CIFAR10 (Y X(rows))
Rcicenage
0.75
0.75
0.50
0.50
CPC
0.25
(.00)
0.25
(VAS)
Accunacy
(V46)
Accunacy
25
20
used
Rows used
Figure 4: Evaluation of ((x;))/i(x;, X an image and Y contains the top rows of X.
Baselines evaluate the first setting with Y having varying number of rows in Figure 4, where the estimations are normalized by the estimated ((x.X) Most methods (except for IGM) predicts zero MI when X and Y are independent, passing the first self-consistency test. Moreover, the estimated MI is dec-eescg with increasing t, but with different slopes. As a reference, we show the validation accuracy of predicting the label where only the top rows are considered.
Data-procsssing In the second setting we set f2 t1 - Ideally, the estimator should satisfy i[[x, X]; [Y, h(Y)))/((X Y) z 1, as additional processing should not increase information. We show the above ratio in Figure 5 under varying t values. All methods except for IMINE and IGM performs well both datasets; IGM performs poorly in CIFARIO (possibly due to limited capacity of VAE), whereas JMINE performs poorly in MNIST (possibly due numerical stability issues).
8

---

Published as a conference paper at ICLR 2020
MNIST (Data processing)
CIFAR10 (Data processing)
2.5
2.0
+-
CPC
SHI [T
2.0
+-
NIRE
GM WWAE)
SHIF 5.0)
10c
1.5
Ranoo
1.0
0.5
1.0
10
15
20
25
10
15
20
25
30
Rows used
Rows used
Figure 5: Evaluation of ((XXXXXXX.,XX))(((XX)) where the ideal value is 1.
MNIST (Additivity)
CIFAR10 (Additivity)
SMILE (5
NINE
SMBLE
Idcal
Rabio
1.5
1.0-
0.5
++
CPC
GM (VAE)
SMILE 5.00
NERE
SMILE (c
Ideal
0.0
10
25
Rows used
Rows used
Figure 6: Evaluation of 7([X1, X2]: [V, Y2)//(X()) where the ideal value is 2.
Additivity In the third setting, the estimator should double its value compared to the baseline with the same i.e. i([Xi, X2]; Y2)) values of L. None of the discriminative approaches worked well in this case except when / is very small, when large this ratio converges (possibly due to initialization and saturation of the training objective). LcM however, performs near perfectly on this test for all values of t.
2. Figure shows the above ratio under different
7 DISCUSSION
In this work, we discuss generative and discriminative approaches to variational mutual information estimation and demonstrate their limitations. We show that estimators based on ΣNwJ and IMINE are prone to high variances when estimating with mini-batches, inspiring our JSMILE estimator that improves performances benchmark tasks. However, none of the approaches pass the self-connistency tests. The generative approaches perform poorly when MI is small (failing independence and data-processing tests) while the discriminative approaches perform poorly when MI is large (failing additivity tests).
good enough to
These empirical evidences suggest that optimization essarily related to optimizing MI, so the empirical successes with these estimators might have little connections to optimizing mutual information. Therefore, would helpful to acknowledge these limitations and consider alternative measurements of information that are more suited for modern machine learning applications (Ozair et al., 2019; Tschannen et al., 2019).
these variational estimators are not
ACKNOWLEDGEMENTS
This research was supported by AFOSR (F4-050--1-1100024), NSF (#1651565, #1522054, #1733686), ONR, and FLI. The authors would like to thank Shengjia Zhao, Yilun Xu and Lantao Yu for helpful discussions.
REFERENCES
Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information bottleneck. arXiv preprint arXiv:1612.000 004100, December 2016.
David Barber and Felix V Agakov. The IM algorithm: variational approach information max- imization. In Advances neural information processing systems, pp. None. researchgate.net 2003.

---

Published as a conference paper at ICLR 2020
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeswar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and R Devon Hjelm. MINE: Mutual information neural estimation. arXiv preprint arXiv:1801.04062, January 2018.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. InfoGAN: Interpretable represeniation learning by information maximizing generative adversarial nets. In D D Lee, M Sugiyama, U Luxburg. Guyon, and R Garnett (eds.), Advances in Neural Infor- mation Processing Systems 29, pp. 2172-2180. Curran Associates, Inc., 2016.
Dinh, D Krueger, and Y Bengio. NICE: Non-linear independent components estimation. arXiv
preprint arXiv:1410.8514: 2014.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real NVP. arXiv May 2016.
preprint
Monroe D Donsker and SR Srinivasa Varadhan. Asymptotic evaluation of certain markov process expectations for large time, i. Communications on Pure and Applied Mathematics, 28(1):1-41) 1975.
Shuyang Gao, Greg Ver Steeg, and Aram Galstyan. Efficient estimation of mutual information for strongly dependent variables. In Artificial intelligence and statistics, pp. 277-286, 2015.
Weihao Gao, Sreeram Kannan, Sewoong Oh, and Pramod Viswanath. Estimating mutual informa- tion for Discrete-Contiuuoss mixtures, arXiv preprint arXiv: 122.122 September 2017.
Aditya Grover and Stefano Ermon. Boosted generative models. arXiv preprint arXiv: urXiv:1702.08485 February 2017.
Irina Higgins, David Amos, David Pfau, Sehastien Racaniere, Loic Matthey, Danilo Rezende, and Alexander Lerchner. Towards a definition of disentangled representations. arXiv preprint arXiv:1812.02233, 2018.
Devon Hjelm, Alex Fedorov, Samuel Lavoie Maroic-Marchildon. Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint Ttt.:/8808.06777 2018.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint
1rXiv:1412.6968. December 2014.
Diederik Kingma and Max Welling. Auto-Encoding variational bayes. 1:Xx:13312.6114v10 December 2013.
arXiv preprint
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. ImageNet classification with deep con- volutional neural networks. In F Pereira, C C Burges, L Bottou, and K Q Weinberger (eds.), Advances in Neural Information Processing Systems 25, pp. 1097-1105. Curran Associates, Inc., 2012.
Y LeCunn, I eon Bottou, Y Bengio, and others. Gradient-Bart Based learning applied to document recog- nition. PROC. OF THE, 1998.
David McAllester and Karl Statos. Formal limitations the measurement of mutual information. arXiv preprint 1.X.::8111.042555 2018.
Sudipto Mukherjee. Himanshu Asnani, and Sreeram Kannan. Cemi: Classifier based conditional mutual information estimation. arXiv preprint (.X.::/906.0//000 2019.
Ilya Nemenman, William Bialek, and Rob De Ruyter Van Steveninck. Entropy and information in neural spike trains: Progress the sampling problem. Physical Review E, 69(5):0561 1, 2004.
X Nguyen, MI J Wainwright, and M I Jordan. Estimating divergence functionals and the likelihood ratio by convex risk minimization. IEEE transactions information theory / Professional Tech- nical Group on Information Theory, 56(11):5847-58)) November 2010. ISSN 2018-9448. doi: 10.1 109/TTTT201122068888
10

---

Published as a conference paper at ICLR 2020
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-GAN: Training generative neural sam- plers using variational divergence minimization. arXiv preprint arXiv: 00709, June 2016.
Sherjil Ozair, Corey Lynch, Yoshua Bengio, Aaron van den Oord, Sergey Levine, and Pierre Sermanet. Wasserstein dependency measure for representation learning. arXiv preprint arXiv: 11700 March 2019.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 16-17, 2017.
Ben Poole, Sherjil Ozair, Aaron van den Oord, Alexander A Alemi, and George Tucker. On varia- tional bounds of mutual information. arXiv preprint arXiv:1905.0692), May 2019.
Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. arXiv preprint arXiv:1503 0044000, March 2015.
Michael Tschannen, Josip Djolonga, Paul Rubenstein, Sylvain Gelly, and Mario Lucic. On mutual information maximization for representation learning. arXiv preprint urXiv: 13625, July 2019.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint arXiv:1807.03748, July 2018.
Petar Velicković, William Fedus, William L Hamilton, Pietro Liò, Yoshua Bengio, and R De- von Hjelm. Deep graph infomax. arXiv preprint arXiv: 1809,001 September 2018.
Shengjia Zhao, Hongyu Ren, Arianna Yuan, Jiaming Song, Noah Goodman, and Stefano Ermon. Bias and generalization deep generative models: An empirical study. In Advances in Neural Information Processing Systems, pp. 10792-10801, 2018a.
Shengjia Zhao, Jiaming Song, and Stefano Ermon. The information autoencoding family: A la- grangian perspective on latent variable generative models. arXiv preprint arXiv: 18806005544 June 2018b.

---

Published as a conference paper at ICLR 2020
A PROOFS
A.1 PROOFS IN SECTION
Theorem 1. VP, Q P(X) such that P < Q we have
DRL. (PI
sup rEA(Q)
Ep logr]
where the supremum is achived when = dP/
Proof. For every T E (Q), define VT = Eg then PT E A(Q) and from the Donsker-
Varadhan inequality (Donsker & Varadhan, 1975)
(PIl() sup Ep(T] log Eolc²] TEL*(Q)
(19)
e sup rT()(
sup TE)=(())
Ep log EoleT]
Epllogryi
(20)
Moreover, we have:
dP log dQ
DKL (PllQ) = Er[[log dP log
Ep
(21)
which completes the proof.
Corollary 2. VP,Q E P()) such that P < Vfo: :2' - Rgo have
fo Vi) log
I( (cpo(fo) Ex,,)x,
(22)
72
Σラ= fa Ui)
Proof.
fo (a 3/
nlcro(fo) = Epa(x,))
Σ log
(23)
ΣØ= fo (I Mi)
( vi) Σ 1 fo (c My))
(I,,
Epp(xXY)
log
(24)
Since
n&fl ( 8) 1= fo yj))
EP(X)P-(M)
1,
(25)
can apply Theorem obtain:
10 yi) j=1 fo M1)
nTcre(fo) Ep-(x,))
Σ log i=1
(26)
no (1i yi: log
Σ Ep(x..))
(27)
ΣØ= fo yi)
Y(x:
(28)
1=1
where Y denotes the concatenation of independent random variables VY....,,Y,, and P(X,,Y?) P(x,, (XX)
the joint distribution of P(X,, Y
12

---

Published as a conference paper at ICLR 2020
A.2 PROOFS IN SECTION
Theorem 2. Assume that the ground truth density ratio r² / dQ and Varo(r") exist. Let Qu denote the empirical distribution of i.i.d. samples from and let Egg denote the sample average over Qn Then under the randomness of the sampling procedure, we have:
Varo[Ea.Ir"]] (
(10)
7
lim Varolloo BQ, [r*]] 2 (xI(P(I)) -.
(11)
Proof. Consider the variance of r* (x) when Q:
dP 27
dP
Varę [aro[r"]==0 = Es
EQQ
(29)
(Q)
dQ
ddP
Ep
(30)
dQ
Er flog aq
(31)
= DRa (Q)
(32)
where (29) uses the definition of variance, (30) uses the definition of Radon-Nikodyy derivative to change measures, (31) uses Jensen's 's inequality over log, and (32) uses the definition of KL diver-
gences.
The variance of the mean of i.i.d. random variables then gives us: Varfr] (Px(()))))
Varq [Q [r]
(33)
n
which is the first part of the theorem.
As oo, Varq Bq,, [r]] + 0, we can apply the delta method:
Varolf(X)) = ((((X)))
(34)
Applying f = log and [[[] = gives us the second part of the theorem:
lim n Varq [o, [r]] = lim n Var [r]] 2 (xu(PlIQ) n
(35)
n-
which describes the variance in the asymptotic sense.
Corollary 1. Assume that the assumptions in Theorem 2 hold. Let Pm and Qn be the empirical distributions of m i.i.d. samples from and n. i.i.d. samples from Q, respectively. Define
NWS = Ep,, [logr* r* m Er.. [log r*] log Eq,, [r*]
Eqq, [r*]
(12)
(13)
where r- = dP/dQ. Then under the randomness of the sampling procedure, we have Vm E N:
Varp 2 (KL(()))))
- 11)/n
(14)
lim n Varp. MINE 2 (Pu((P 1. 1=000
(15)
Proof.
Since Pm and Qn are independent, we have
Var 2 Var [r]]]
(36)
Var[Eq, [r+]] (PI(()
(37)
and
lim n /a n00
> lim Var EQ, [r*]] 2 Dku(P(IQ) -1 n
(38)
which completes the proof.
13

---

Published as a conference paper at ICLR 2020
A.3 PROOFS IN SECTION
Theorem 3. Let r(t)) X + Ryo be any non-negative measurable function such that rrd S. S E (0, 0o) and r(æ) E [0,e*] Define r-(x:) = clip(r(ar), e², ) for finite, non-negative T. If
T K, then the bias for using Ty to estimate the partition f ofr satisfies:
[Eob]] - [ S max (e (e Se
c c S(c* -
ek - E
T 2 K, then
[F]r]] Rolrr]l S S - Se-R)
Proof. We establish the upper bounds by finding worst case r to find the largest [Bg|r] - -Eg(r First, without loss of generality, we may assume that r(wx E (-oc, [e 0o) for all æ E X. Otherwise, denote X (r) = {I E :e < r(xx) < } as the (measurable) set where the r(x)
values are between and et. Let
(+(r)= S r(æ)dx e (e (1))
(39)
be the integral of r over X²(r). We can transform r(æ: for all x E Xx(r) to have values only in (e=T,e") and still integrate to (r),,, so the expectation under Q is not changed.
Then we show that we can rescale all the values above ex and below ex to the same value without changing the expected value under Q. We denote
K1 = loog| / [(r(x) Se ()(())d(())) llg / [(r(x) Se se T)Q(())
(40)
K₂ log / I(r(x)) 2 (=)))(()( - log / I(r(m) 2 ed)(((())
(41)
where ex and K2 represents the mean of r(x) for all r(x:) Se- and and)(x) 2 e respectively. We then have:
Eolr] = cki / [(r(a)) Se (dQ()) kz f I(r(a)) 2 ((Q)(()))
(42)
1= / (r((z) Se )dQ(()+ + / 1(r(z)2 (d)()()))
2
(43)
so we can parametrize Bo(r]] via K1 and K2. Since Eq = S by assumption we have:
/ [(((x)= S (e-))d((a = .k -K Ki
e²² - S
(44)
and from the definition of rr (x):
Eq[r=]= = ckze- ST + Se" c- -Kpp
== g(K1 K2)
(45)
ck - c K₁
We can obtain an upper bound once we find max g(K,, K2) and min g (K1, K2). First, we have:
0g(K1, K2)
e- e - -e-Ki K E Ki (e T= -S - + Se - -e ET
0Ki
(ekz e-K
<Kie ( 5) 20
(46)
(e - c k2
0g(K1,K2
- -e-Ki) - (e - Se + Se -KiE)
0K₂
(ek2 KKi
ckaee" - (e- 5)
< 0
(47)
(e e -k1
14

---

Published as a conference paper at ICLR 2020
Therefore, 9(K1, K2) is largest when K1 - 00, K2 and smallest when K1 T, K2 -
1 e- c² +((c -e
maxg K2)
lim 400
-
(48)
e" e k
k e - 1 + S(EJ -6 =
g(i, K2)
lim
(49)
ek
e-
Therefore,
Belr] - Bo[r>]] max( max (K1, K2) - -1,,15 min K2)l) (2)l)
(50)
-ax (le Se 2,,1 Se cT
(51)
The proof for Theorem 3 simply follows the above analysis for fixed K. When T < K, we consider the case when K1 + o0, K2 7 and K1 т, K2 K; when be clipped, so the increased value is no larger than the case where Ki 0o, K2 = K:
only the smaller values will
e S
ex e c
(1-
(52)
where eK 2 from the fact that f dQ S.
Theorem 4. The variance of the estimator Eq,. [r+] (using n samples from Q) satisfies:
Var Eq, [r>]] 4n
(18)
Proof. Since r, (a) is bounded between e² and e eT we have
Var[r-] S e² - -
(53)
Taking the mean of n independent random variables gives us the result.
Combining Theorem 3 and 4 with the bias-variance trade-off argument, we have the following: Corollary 3. Let r(x): : + R2o be any non-negative measurable function such that f rdQ = S SE (0, oo) and r(x) E [0, e1. Define r (x) = clip(r (x), (x) e _ finite, non-negative t and (p)pr(()),
Eq, the sample average of i.i.d. samples from Q. If K, then
Eol(r -Eq, (r.,))))) max (e-11-Se) Se 1,
K e + S(eK -00)
e" e
ek -
4n
1 2 K, then:
Bel(r Be,lr=l)² 22- = Se -K= 22= -
(54)
4n
15

---

Published as a conference paper at ICLR 2020
B ADDITIONAL EXPERIMENTAL DETAILS
B.1
BENCHMARK TASKS
Tasks We sample each dimension of (æ. y) independently from correlated Gaussian with mean and correlation of where A' = = RR200 The true mutual information is computed as:
d log -1- (a, y) = 2
(55)
The initial mutual information is 2, and we increase the mutual information by 2 every 4k iterations, so the total training iterations is 20k:
Architecture and training procedure For all the discriminative methods, we consider two types of architectures joontt and separable. The joint architecture concatenates the inputs y. and then passes through a two layer MLP with 256 neurons in each layer with ReLU activations at each layer. The separaable architecture learns separate neural networks for and y (denoted g(x) and hy)) and predicts g(x) hyy): and are two neural networks, each a two layer MI. with 256 neurons each layer with ReLU activations each layer; the output and are 32 dimensions.
For the generative method, we consider the invertible flow architecture described in (Dinh et al., 2014; 2016). Po: Ps are flow models with 5 coupling layers (with scaling), where each layer contains a neural network with 2 layers of 100 neurons and ReLU activation. For all the cases, we use with the Adam optimizer (Kingma & Ba, 2014) with learning rate x 10 and 31 0.9, 82 = 0.999 and train for 20k iterations with a batch size of 64, following the setup in Poole et al. (2019).
Additional results We show the bias, variance and mean squared error of the discriminative ap- proaches in Table 2. We include additional results for ISMILE with = 10.0.
Gaussian
Cubic
MI
2
8
10
2
4
6
8
10
CPC
0.25
0.99
2.31
4.00
5.89
0.72
1.48
2.63
4.20
5.99
NWJ
0.12
0.30
0.75
2.30
2.97
0.66
1.21
2.04
3.21
4.70
Bias
SMILE (7 1.0)
0.15
0.30
0.32
0.18
0.03
0.47
0.77
1.16
1.64
2.16
SMILE (T 5.0)
0.13
0.11
0.19
0.54
0.86
0.71
1.22
1.55
1.84
2.16
SMILE (7 = 10.0)
0.14
0.21
0.22
0.11
0.19
0.70
1.28
1.83
2.44
3.02
SMILL (r 0C)
0.15
0.21
0.22
0.12
0.22
0.71
1.29
1.82
2.35
2.81
GM (Flow)
0.11
0.14
0.15
0.14
0.17
1.02
0.47
1.85
2.93
3.55
CPC
0.04
0.04
0.02
0.01
0.00
0.03
0.04
0.03
0.01
0.01
NWJ
0.06
0.22
1.36
16.50
99.0
0,04
0.10
0.41
0.93
3.23
Var
SMILE 1.0)
0.05
0.12
0.20
0.28
0.34
0.04
0.10
0.14
0.20
0.30
SMILE (r 5.0)
0.05
0.19
0.31
0.51
0.07
0.12
0.18
0.26
SMILE (7 = 10.0)
0.05
0.13
0.31
0.69
1.35
0.03
0.10
0.21
0.46
0.79
SMILE (T oo)
0.05
0.14
0.36
0.75
1.54
0.03
0.12
0.24
0.65
0.94
GM (Flow)
0.05
0.10
0.13
0.16
0.19
0.56
0.72
0.92
1.02
1.02
CPC
0.10
1.02
5.33
16.00
34.66
0.55
2.22
6.95
17.62
35.91
NWJ
0.07
0.32
2.19
33.37
28.43
0.47
1.55
4.56
11.13
27.00
MSE
SMILE 1.0)
0.08
0.21
0.30
0.32
0.31
0.26
0.69
1.49
2.90
4.98
SMILE (T = 5.0)
0.07
0.13
0.22
0.57
1.26
0.54
1.56
2.53
3.58
4.92
SMILE (T = 10.0)
0.07
0.18
0.36
0.67
1.33
0.52
1.75
3.54
6.41
9,91
SMILE (T oo)
0.08
0.19
0.40
0.76
1.62
0.54
1.75
3.55
6.09
8.81
GM (Flow)
0.07
0.11
0.14
0.17
0.22
1.65
0.91
4.36
9.70
13.67
Table 2: Bias, Variance and MSE of the estimators under the joint critic.
We show the bias, variance and MSE results in Figure 7. We also evaluate the variance of estimating Eq,, [r.] (partition function with clipped ratios) for different values of in the SMILE estimator in Figure With smaller we see visible decrease in terms of variance in this term; this is consistent with the variance estimates in Figure 7, as there the variance of Ep, [logr] is also considered.
16

---

Published as a conference paper at ICLR 2020
Varlance
10
20
(76x)
Cubic
107
30-
Variance
12-1
10-2
10
30
Figure 7: Bias / Variance / MSE of various estimators. Gaussian (top) and Cubic (down).
MINE (Gsussian)
MINE (Cubic)
Gaussian
16a
SPEC Dea (raa
CIEC
5-
2-
5000
10000
15000
20000
5000
10000
15000
20100
(h) Variance of Bq, [r+]
(a) Additional benchmark results on MINE estimator.
Figure 8: Additional benchmark results.
B.2 SELF-CONSISSENCY EXPERIMENTS
Tasks We consider three tasks with the mutual information estimator î:
1. f(X;Y) where X is an image from MNIST (LeCun et al., 1998) or CIFARIO (Krizhevsky et al., 2012) and Y the top rows of X. To simplify architecture designs, we simply mask out the bottom rows to be zero, see Figure 3.
2. i([x, ((())) where X is an image, Y is the top rows of X, A(Y) is the top (t 3) rows of Y and [ denotes concatenation. Ideally, the prediction should be close
to f(X;Y).
3. f(x,, X2), [Y, Y2)) where X, and X₂ are independent images from MNIST CIFAR 10,
Vi and Y2 are the top & rows of X, and X₂ respectively. Ideally, this prediction should be close to 2 Y(:;Y).
Architecture and training procedure We consider the same architecture for all the discriminative approaches. The first layer is convolutional layer with 64 output channels, kernel size of 5, stride of 2 and padding of 2; the second layer is convolutional layer with 128 output channels, kernel size of 5, stride of 2 and padding of This is followed another fully connected layer with 1024 neurons and finally linear layer that produces an output of 1. All the layers (except the last one) use ReLU activations. We stack variables over the channel dimension to perform concatenation.
For the generative approach, we consider the following VAE architectures. The encoder architec- ture identical to the discriminative approach except the last layer has 20 outputs that predict the mean and standard deviations of 10 Gaussians respectively. The decoder for MNIST is two layer MLP with 400 neurons each; the decoder for CIFARIO is the corresponding transposed convolution network for the encoder. All the layers (except the last layers for encoder and decoder) use ReLU ac- tivations. For concatenation we stack variables over the channel dimension. For all the cases, we use with the Adam optimizer (Kingma & Ba, 2014) with learning rate 10-1 and B 0.9, B2 0.999.
17

---

Published as a conference paper at ICLR 2020
For ICM train for 10 epochs, and for the discriminative methods, train for 2 epochs, due to numerical stability issues of IMINE.
Additional experiments on scaling, rotation and translation We consider additional benchmark experiments on MNIST where instead of removing rows, we apply alternative transformation such as random scaling, rotation and translations. For random scaling, we upscale the image randomly by Ix to 1.2x; for random rotation, we randomly rotate the image between +20 degrees: for random translation, we shift the image randomly by no more than pixels horizontally and vertically. We consider evaluating the data processing and additivity properties, where the ideal value for the former no more than 1, and the ideal value for the latter is 2. From the results Table 3, none of the considered approaches achieve good results in all cases.
CPC
MINE
GM (VAE)
SMILE (r 5.0)
SMILE (T oc)
Scaling
1.00
1.03
1.12
1.19
1.04
Data-Processing
Rotation
1.00
1.30
1.13
1.03
1.27
Translation
1.00
1.28
1.01
1.07
1.08
Scaling
1.00
1.55
1.89
1.04
1.18
Additivity
Rotation
1.00
2.09
1.58
1.50
1.78
Translation
1.00
1.41
1.28
1.32
1.33
Table 3: Self-consistency experiments other image transforms.
18


Reviews:
[{"review_id": "rkgLMqFl9r", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nAs the title suggests the paper focusses mostly on a negative result:\nMutual information (MI) estimators obtained by variational methods have severe\nlimitations that make them potentially not useful for down stream tasks.\nBesides highlighting the problems with variational MI estimators the authors\nsuggest a modification to slightly improve the performance of MI estimators\nbased on partition functions by reducing their variance when MI is high.\n\nThe authors give a good overview / introduction of various approaches to\nvariational MI estimation by discriminative and generative methods.\nGenerally, MI estimation involves the estimation of the KL divergence between\nthe joint distribution and the product of the marginals.\nThe authors present a unifying view on the different approaches that optimizes\nthe log density ratio required for the KL divergence over the space of\nlog density ratios.\nDiscriminative approaches model the density ratio directly (through e.g. neural\nnetwork models) and generative approaches model the separate densities\n(as generative models where it is possible to evaluate the (conditional)\nprobabilities / likelihoods of the data generating process).\n\nThe authors prove that discriminative approaches that are based on the partition\nfunction approach suffer from high variance where mutual information is high\n(Theorem 2).\nThe estimator based on a finite sample has high variance even if the density\nratio approximation is correct.\n(The partition function approach is a way of staying constrained to the log\ndensity ratio function space.)\nThis high variance problem is something that has previously been observed\nempirically and is the main theoretical point that is being made about\nlimitations of MI estimators.\n\nIn order to slightly alleviate the problem of high variance the authors suggest\na way of biasing MI estimators by clipping the density ratio estimates\nthrough a constant chosen as a hyper-parameter.\nThey prove that their clipping approach reduces variance and therefore\nintroduces a bias variance tradeoff.\n\nIn their later experiments the clipped version of the discriminative approach\nperforms much better in terms of variance than without clipping and also better\nthan a generative approach.\n\nIn order to empirically evaluate the quality of MI estimators the authors\nsuggest three criteria that they call self-consistency:\n(i) independence, (ii) data processing, (iii) additivity\n\nSelf-consistency is evaluated experimentally on images where mutual information\nis computed between original image and image with part covered.\n\nThe authors claim and experimentally show that discriminative approaches fail\nin (iii) and generative approaches fail in (i), (ii).\nOverall, variational MI approaches do not satisfy self-consistency.\n\nEvaluation:\nI suggest to accept the paper.\nThe theoretical contribution of showing the variance limitation of\ndiscriminative approaches seems significant.\nThat insight leads to the idea that clipping can be a useful bias that\nsignificantly reduces variance without making the already biased anyways\nresults much worst in the experiments.\nHowever, I also feel like the paper is not yet as focused as it could be.\nIt contains many concepts that could need a little bit more space.\n\nSuggestions:\n- Page 2: Nitpick, but in the definition of $L^p(Q)$ using $\\colon$ twice is\n\tnot super readable on the first read\n\n- Page 2: In the definition of $I_{BA}$ clearify whether $p(x)$ is a marginal\n\tor a joint density (as $P$ is the cumulative joint)\n\n- Page 3: In Theorem 1 what is the definition of $P \\ll Q$?\n\tThis suggests an order on the space of measures $\\mathcal P$?\n\n- Page 3: \"Obtain an density ratio estimate\" -> Obtain a density ratio estimate\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I did not assess the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "Sye0PZtoFr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper relates most existing variational mutual information estimators to density ratio estimation and uses this to show high variance of certain categories of estimators; this motivates a new SMILE estimator that uses clipping to explicitly control the variance of the estimator (while potentially increasing bias). The paper also conducts some experiments on how well mutual information estimators satisfy some of the core properties of mutual information in practice.\n\nThe core result of Theorem 2, although is proof is very simple once the problem has been phrased appropriately, is enlightening.\n\nThe self-consistency tests are also sensible and important, although it would be nice to understand better which of these properties are truly important for various uses of mutual information in the literature.\n\nThe proposed SMILE estimator is natural. But:\n\n- As $\\tau \\to \\infty$, the variance bound of Theorem 4 naturally becomes infinite (as we'd expect). But we'd hope that the bias would become 0, when in fact Theorem 3 only shows in this case that the bias is at most S. Of course it should be true that $\\mathbb E_Q[r] = \\mathbb E_Q[r_\\infty]$; can we fix Theorem 3 so that it recovers this fact?\n\nThen the biggest question not really addressed by this paper is: how do you set $\\tau$?\n\n- If we know the (approximate) value of $S$, then we can presumably trade off between the bias and variance terms of Theorem 3 and 4 to find the most effective value for $\\tau$. (Indeed, it might be nice to write a corollary showing that if you set $\\tau$ to something depending on $n$ and some upper bound on $S$, you achieve some rate in L2 risk.)\n\n- Is it possible to relate the value of $S$ to, say, the suboptimality in cross-entropy loss of the classifier used to obtain $r$ (when getting $r$ that way)?\n\n- Is it possible to get a probabilistic upper bound on $S$, either with some other technique or perhaps a simple Hoeffding/Bernstein-type bound on the SMILE estimator with some initial value of $\\tau$, and then use that to choose the rate for $\\tau$?\n\n- Is there some other practical scheme (even if it doesn't admit bounds) to set $\\tau$ to control the bias, given that it depends on $S$?\n\n\n\nSmaller points and minor suggestions:\n\n- In the paragraph following Theorem 2, it is first of all strange to say that \"alternative procedures to obtaining the density ratio estimates will not alleviate the high variance issue\" when indeed your proposal does just that. The last sentence of this paragraph clarifies the issue, but still makes $r \\ne r^*$ sound \"bad,\" when in fact it's a trade-off; this could be rephrased more clearly.\n\n- In Theorem 3: I think writing $e^{-2 \\tau} \\lvert S - e^\\tau \\rvert$ would be clearer than $\\lvert e^{-\\tau} - S e^{-2 \\tau} \\rvert$ as you have it.\n\n- In the multivariate Gaussian experiments, Section 6.1: why do you effectively warm-start the estimator for higher mutual informations? Is it just so you can put things in one plot per method? I think it would be more natural to train each different problem from scratch; you can still use the same plot format if you like (with additional vertical lines between problems or something), but it's not clear whether the behavior of the estimator is affected by this choice.\n\n- Why are the flow-based results not shown in Figure 2 / Table 2?\n\n- In Section 6.2 when setting up the three evaluation settings, it would be good to have a forward reference to Figure 3 (since these settings are perhaps easier to understand visually).", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "S1lVRYFatS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work summarizes the existing methods of mutual information estimation in a variational inference framework and describes the limitations in terms of bias-variance tradeoffs. Further, the authors care about the self-consistency, namely, independence, data processing, and additivity, which are properties of both entropy and differential entropy. Further, density ratio clipping is proposed to lessen a high-variance problem in estimating a partition function.\n\nIn regard to the clipped density rations, it is not clear why both variance and bias become small when S is close to 1 and using small $\\tau$ in Theorem 3 and 4. In Fig. 2 on the benchmark experiment, $\\tau=5.0$ showed lower variance than $\\tau=1.0$.\n\nThe comparison with MINE and BA is also expected on the benchmark experiment.\n\nAs for the self-consistency tests on images, especially, for the ‘data-processing’ property, it would better to apply affine transformation such as rotation, translation, and scaling, rather than rows deletion.\n\nIn Section 3.1, $\\Gamma$ is not defined.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a critical appraisal of variational mutual information estimators, and suggests a slight variance-reducing improvement based on clipping density ratio estimates, and prove that this reduces variance (at the cost of bias). They also propose a set of criteria they term "self-consistency" for evaluation of MI estimators and, and show convincingly that variational MI estimators fall short with respect to these.

Reviewers were generally positive about the contribution, and were happy with improvements made. While somewhat limited in scope, I believe this is nonetheless a valuable contribution to the conversation surrounding mutual information objectives that have become popular recently. I therefore recommend acceptance.
Decision: Accept (Poster)