DEMI: Discriminative Estimator of Mutual Information 
Authors: Ruizhi Liao
Daniel Moyer
Polina Golland
William M Wells
Venue:  (2021)
Keywords: Mutual information estimation
discriminative classification

Abstract: Estimating mutual information between continuous random variables is often intractable and extremely challenging for high-dimensional data. Recent progress has leveraged neural networks to optimize variational lower bounds on mutual information. Although showing promise for this difficult problem, the variational methods have been theoretically and empirically proven to have serious statistical limitations: 1) many methods struggle to produce accurate estimates when the underlying mutual information is either low or high; 2) the resulting estimators may suffer from high variance. Our approach is based on training a classifier that provides the probability that a data sample pair is drawn from the joint distribution rather than from the product of its marginal distributions. Moreover, we establish a direct connection between mutual information and the average log odds estimate produced by the classifier on a test set, leading to a simple and accurate estimator of mutual information. We show theoretically that our method and other variational approaches are equivalent when they achieve their optimum, while our method sidesteps the variational bound. Empirical results demonstrate high accuracy of our approach and the advantages of our estimator in the context of representation learning.


Under review as a conference paper at ICLR 2021
DEMI: DISCRIMINATIVE ESTIMATOR OF MUTUAL INFORMATION
Anonymous authors Paper under double-bling review
ABSTRACT
Estimating mutual information between continuous random variables is often in- tractable and extremely challenging for high-dimensional data. Recent progress has leveraged neural networks to optimize variational lower bounds on mutual in- formation. Although showing promise for this difficult problem, the variational methods have been theoretically and empirically proven to have serious statisti- cal limitations: 1) many methods struggle produce accurate estimates when the underlying mutual information is either low or high; 2) the resulting estimators may suffer from high variance. Our approach is based on training classifier that provides the probability that data sample pair is drawn from the joint dis- tribution rather than from the product of its marginal distributions. Moreover, we establish direct connection between mutual information and the average log odds estimate produced by the classifier on test set, leading to a simple and ac- curate estimator of mutual information. We show theoretically that our method and other variational approaches are equivalent when they achieve their optimum, while method sidesteps the variational bound. Empirical results demonstrate high accuracy of our approach and the advantages of our estimator in the context of representation learning.
INTRODUCTION
Mutual information (MI) measures the information that two random variables share. MI quantifies the statistical dependency linear and non-linear- between two variables. This property has made MI a crucial measure in machine learning. In particular, recent work in unsupervised representation learning has built on optimizing MI between latent representations and observations (Chen et al.)
2016 Zhao et al., 2018 Oord et al., 2018 Hjelm et al., 2018 Tishby & Zaslavsky 2015 Alemi 2018; Ver Steeg & Galstyan 2014). Maximization of MI has long been a default method for modallty image registration (Maes et al. 1997), especially in medical applications (Wells III et al.
ct al.
1996). though in most work the dimensionality of the random variables is very low. Here, coordinate transformations images are varied maximize their MI.
Estimating MI from finite data samples has been challenging and is intractable for most continuous probabilistic distributions. Traditional MI estimators (Suzuki al., [2008) Darbellay & Vajda 1999]
Kraskov et al.] [2004] Gao et al. 2015) do not scale well to modern machine learning problems with high-dimensional data. This impediment has motivated the construction of variational bounds for MI (Nguyen et al.) 2010 Barber & Agakov 2003); in recent years this has led to maximization procedures that use deep learning architectures parameterize the space of functions, exploiting
the expressive power of neural networks (Song & Ermon 2019 Belghazi et al. 2018 Oord et al.
2018 Mukherjee al. 2020).
Unfortunately, optimizing lower bounds on MI has serious statistical limitations. Specifically, McAllester & Stratos (2020) showed that any high-confidence distribution-fies lower bound can- not exceed O(log.V,, where N is the number of samples. This implies that if the underlying MI is high, it cannot be accurately and reliably estimated by variational methods like MINE (Belghazi) et al. 2018). Song & Ermon 2019) further categorized the state-of-the-an variational methods into "generative" and "discriminative" approaches, depending on whether they estimate the probabil- ity densities the density ratios. They showed that the "generative" approaches perform poorly

---

Under review as a conference paper at ICLR 2021
when the underlying MI is small and "discriminative" approaches perform poorly when MI is large; moreover, certain approaches like MINE (Belghazi et al. 2018 are prone to high variances.
We propose simple discriminative approach that avoids the limitations of previous discriminative methods that are based on variational bounds. Instead of estimating density or attempting to predict one data variable from another, method estimates the likelihood that sample is drawn from the joint distribution versus the product of marginal distributions. A similar classifier- approach was used by Lopez-Paz & Oquab (0117) for "two sample testing" hypothesis tests about whether two samples are from the same distribution or not. If the two distributions are the joint and product of the marginals, then the test is for independence. A generalization of this work was used by Sen et al. (2017) to test for conditional independence. We show that accurate performance on this clas- sification task provides an estimate of the log odds. This can greatly simplify the MI estimation task in comparison with generative approaches: estimating single likelihood ratio may be easier than estimating three distributions (the joint and the two marginals). Moreover, classification tasks are generally amicable to deep learning, while density estimation remains challenging many cases. Our approach avoids the estimation of the partition function, which induces large variance in most discriminative methods (Song & Ermon 2019). Our empirical results bear out these conceptual advantages.
Our approach, as well other sampling based methods such as MINE, uses the given joint/paired data with derived "unpaired" data that captures the product the marginal distributions p(z)p(y)- The unpaired data can be synthesized via permutations or resampling of the paired data. This con- struction, which synthesizes unpaired data and then defines a metric to encourage paired data points map closer than the unpaired data in the latent space, has previously been used in other machine learning applications, such as audio-video and image-text joint representation learning (Harwath) al. 2016 Chauhan al., 2020). Recent contrastive learning approaches (Tian et al. 2019 Hénaff et al.
2019 Chen et al., [020]] He et al., (2200) further leverage a machine learning model to dif- ferentiate paired and unpaired data mostly in the context of unsupervised representation learning- Simonovsky et al. (2016) used paired and unpaired data in conjunction with a classifier-based loss
function for patch-based image registration.
derive our approach to estimating MI. Sec- tion 2.4 discusses connections to related approaches, including MINE. This followed by empiri- cal evaluation in Section 3 Our experimental results synthetic and real image data demonstrate the advantages of the proposed discriminative classification basedd MI estimator, which has higher accuracy than the state-of-the-art variational approaches and good bias/variance tradeoff.
This paper is organized as follows. In Section
METHODS
Let E X and y E y be two random variables generated by joint distribution p x x y + R+ Mutual Information (MI)
(x:y) A Ep(r,,) log p(x.y) y) p(x)x(v)
(1)
is measure of dependence between and y. Let D { ys W be set of independent identically distributed (i.i.d.) samples from p(æ, v). The law of large numbers implies
(D) A log
p( vi) p(z))p(3))
I(x;)) as + 00,
which suggests simple estimation strategy via sampling. Unfortunately, the joint distribu- tion p(x, 3/) often unknown and therefore the estimate in Eq. 2 cannot be explicitly computed.
Here we develop an approach to accurately approximating the estimate (D) based on discrimina- tive learning.
In our development, we will find it convenient to define a Bernoulli random variable 2 E {0, 1] and to "lift" the distribution p(æ, y) the product space X' y x {0, 1). We thus define family of
2

---

Under review as a conference paper at ICLR 2021
distributions parametrized by E (0, 1) follows:
P. (z. y/2 a; =p(x.
(3)
(2 y/ = 0; = pp())p())
(4)
Pa(= = 1;0) = 1 1-p4 0; a.
(5)
Using Bayes' rule, we obtain
(= 1 v) p, 1) p=
p(z, y)
(6)
(2 -= 0 y) (, = 0) P. (z.
0) p. pz (== p(z)p(y)
a
which implies that the estimate
(2 can be alternatively expressed as
(= 11, log a
1
log
n i=
p=(2 = 0 10 9/)
logit [p. 1/x logit(a),
(8)
72
where logit(u) 2 log 1- is the log-odds function.
Our key idea is to approximate the latent posterior distribution (= 1/2, y) by classifier that is trained distinguish between the joint distribution p(r, y) and the product distribution p(z)p(y) as described below.
2.1 TRAINING SET CONSTRUCTION
We assume that we have access to a large collection D if... samples (a, y) from p(z, y) and define p(x, y; D), p(x; D), and p(y; D) to be the empirical joint and marginal distributions respectively induced by data set DD
We construct the training set T = [(x1. 20)} } of m i.i.d. samples from our empirical approxima-
tion to the distribution (2, 2). Each sample is generated independently of all others as follows.
First, a value z {0, 1} is sampled from the prior distribution p. (z) in (5). If z = 1, then pair (xr, y²) is sampled randomly from the empirical joint distribution p(x, y; D); otherwise value x is sampled randomly from the empirical marginal distribution p(z; D) and value y is sampled randomly from the empirical marginal distribution p(y; D), independently from x². This sampling easy to implement as it simply samples an element from set of unique values in the original collection D with frequencies adjusted to account for repeated appearances of the same value.
It is straightforward to verify that any individual sample in the training set T is generated from distribution p. (ar, y, 2) up the sampling of D. Where D is small, multiple samples may not be jointly from D but from some idiosyncratic subset; however, the empirical distribution induced by the set converges to p. (x, y, 2) as the size of available data D and the size m of the training set T becomes large.
2.2 CLASSIFIER TRAINING FOR MUTUAL INFORMATION ESTIMATION
Let q(z = 1 y/ 0, T) be a (binary) classifier parameterized by and derived from the training set T. If q(2 = 1|r, yr 0, T) accurately approximates the posterior distribution p.(z = 1|z, a), then we can use this classifier q instead of (_ = 1|x, yi a) in (8) to estimate MI.
We follow the widely used maximum likelihood approach to estimating the classifier's parameters 0 and form the cross-entropy loss function
m
((0:T) = m j=1
log ((2 ((22(zz) w ,²; T,
(9)
log (2 = 1, 0) (1=2))ogg) log(1 q(2² = 11 1 (10)
3

---

Under review as a conference paper at ICLR 2021
to be minimized to determine the optimal value of parameters 0. Once the optimization is completed, we form the estimate
1 " T)
TD Σ logit g((
1/2 ye: 0. T) - logit[a]
(11)
that approximates the estimate in (8) distinct from the training set T.
Note that the estimate computed using the data set D, which
2.3 ASYMPTOTIC ANALYSIS
As the size of available data and the size m of the training set T increase to infinity, the law of large numbers implies
E(0 T) + Ep (a.y.)) llog [<]|,, yı 0, T),,
(12)
and therefore
i - arg min min T) arg 0 p. (r.),)) [log <<22, y; 0, T)].
(13)
Thus, when the model capacity of the family <2122, y: 0) is large enough to include the original distribution p. (2(2,,)) y). Gibb's inequality implies
q x, yi 0, 7) + (=(2,, v) and ((D,T) - (xx;))
(14)
both the training data and testing data grow.
2.4 CONNECTIONS TO OTHER MUTUAL INFORMATION ESTIMATORS
MINE and SMILE Belghazi et al. 2018) introduced the Mutual Information Neural Estimation (MINE) method, wherein they proposed learning a neural network f(x, y; that maximizes the objective function J(f) Ep(z..) [xr y: 0)] log Ep(r)p(y) [e which is the Donsker- Varadhan (DV) lower bound for the Kullback Leibler (KL) divergence. For analysis purposes, we
define q(2, y: 0) - ((((=:)):0p(()))(()))) where = Ep(rp(() [e/().)))) By substituting into the
definition of J.)) and invoking Gibb's inequality, we obtain
((f = Ep(a,)) [log Úαz y; 0)] Ep(a,)) [log(())(())] < Ep(ay) [ogg(<r, 3)] Ep(z.)) [log p(x))())) I(x:)),
(15)
(16)
with equality if and only if ((x y: 0) = p(z, i...
p(x,1) + C, f(x.v) log p(z)p(u)
(17)
where is a constant that is absorbed into the partition function Z. Thus the objective function is lower bound on MI and is maximized when the unspecified "statistics network" f(r, y) is the log likelihood ratio of the joint distribution and the product of the marginals.
Song & Ermon (2019) introduced the Smoothed Mutual Information Lower Bound Estimator (SMILE) approach which is a modification of the MINE estimator. To alleviate the high variance of f(x, v) in practice, the tilting factor (x(x)) is constrained to the interval le-r e²), for tuned hyper-parameter As co, SMILE estimates converge to those produced by MINE.
The log likelihood ratio of the joint versus the marginals, which the f(x, v) network from both these methods approximates, is the optimal classifier function for the task defined our training set T above. Our parameterization of this ratio makes use of a classifier and the logit transformation. While analytically equivalent, the MINE and SMILE optimization procedures must instead search over ratio functions directly, optimizing f(r, y) ss p(z, v)/p(z)p(y) itself. Our experimental results demonstrate the advantage of using our estimator in
4

---

Under review as a conference paper at ICLR 2021
Oord et al. (2018) proposed contrastive predictive coding (CPC) method that also maxi- mizes lower bound
CPC
N
f(z., y1:0)
J(f) = Bp(a.y)
log
+ log N S I(x:)),
(18)
N
1=1 N E f My:
where f(, y; () is a neural network and N is the batch size. CPC is not capable of estimating high underlying MI accurately- it is constrained by their batch size N, and this constraint scales logarith- mically. In our approach, we do not estimate the likelihood ratio directly, instead we construct an auxiliary variable and "lift" the joint distribution, where we leverage the power of discriminative neural network classifier. The logit transformation of our classifier response is used to approximate the log likelihood ratio in Eq. (I.
CCMI Mukherjee et al. (2020) recently proposed classifier based (conditional) MI estimator (CCMI). The classifier g (z, y; trained on paired and unpaired sample pairs to yield the pos- terior probability that the joint distribution p(x: y) (rather than the product of marginals p(z)p(y))
generated sample pair (x, y). Unlike DEMI, the CCMI estimator
g(r, w)
f(, v) Ppt(a))) [logit [9(2r, v)] - log P(())(())
(19)
g y)
still relies on variational lower bound in Belghazi et al. 2018 The first term above employs paired sample pairs and is identical to our estimator in Eq. for g( y) _ q(= = 1|2, y; 0, T) and a 0.5. The second term depends on the unpaired samples and is asymptotically zero. Thus CCMI and DEMI are asymptotically equivalent, but for finite sample sizes CCMI is prone to higher error than DEMI, as we demonstrate empirically later in the paper.
EXPERIMENTS
We employ two setups widely used in prior work (Song & Ermon 2019 Belghazi al., 2018
Poole et al. 2019 Hjelm et al. (018) to evaluate the proposed estimator and to compare it to the state of the art approaches for estimating MI. In particular, we directly evaluate the accuracy of the resulting estimate in synthetic examples where the true value of MI can be analytically derived and also compare the methods' performance in representation learning task where the goal is to maximize MI.
Additional experiments that investigate self-consistency and long-run training behavior are reported in Appendices A and B respectively.
3.1 MI ESTIMATION
We sample jointly Gaussian variables x and y with known correlation and thus known MI values, which enables us to measure the accuracy of MI estimators when trained this data. We vary the dimensionality of x and y (20-d, 50-d, and 100-d), the underlying true MI, and the size of the training set (32K, 80K, and 160K) in order to characterize the relative behaviors of different MI estimators. In an additional experiment, we employ an element-wise cubic transformation (y: + y to generate non-linear dependencies in the data. Since deterministic transformations of and y preserve MI, we can still access ground truth values of MI in this setup. We generate a different set of 10240 samples held out for testing/estimating MI given each training set. We generate 10 independently drawn training and test sets for each two correlated Gaussian variables.
We assess the following estimators in this experiment:
DEMI, the proposed method, with three settings of the parameter (0.25, 0.5, 0.75) in
Eq. (5).
SMILE (Song Ermon 2019). with three settings of the clipping parameter J {1.0, 5.0, oc). The 7 = oo case (i.c., no clipping) is equivalent to the MINE (Belghazi et al. 2018) objective.
InfoNCE (Oord et al. 2018), the method used for contrastive predictive coding (CPC).
5

---

Under review as a conference paper at ICLR 2021
CCMI (Mukherjee et al. 2020
A generative model (GM), i... directly approximating p(z, y) and marginals log p(x:) and log ply) using a flow network. We note that it is difficult to make comparable param- eterizations between GM-flow networks and the rest of the methods, and that additionally because the "base" flow distribution is a Gaussian, these networks have a structural advan- tage for our synthetic tests. They are, in sense, correctly specified for the Gaussian case, which probably would not happen in real data.
Each estimator uses the same neural network architecture: a multi-layer perceptron with an initial concatenation layer for the æ and y inputs, then two fully connected layers with ReLU activations, then single output. This final layer uses a linear output for MINE, SMILE, InfoNCE, and CCMI, and logistic output for DEMI. We 256 hidden units for each of the fully connected layers. For the GM flow network we use the RealNVP scheme (Dinh et al.] 2016), which includes a "transfor- mation block" of two 256-unit fully connected layers, each with ReLU activations. This network outputs two parameters, scale and a shift, both wise This transformation block is repeated three times.
We train each MI estimator for 20 epochs and with the mini-batch size of 64. We employ the Adam optimizer with learning rate parameter 0. 0.0005. The architecture choices above and the optimization settings are comparable with Song & Ermon (019))
Results. Figure D reports the MI estimation error (I(x, ()- [xx, y)) versus the true underlying MI for the experiments with joint Gaussians and joint Gaussians with a cubic transformation, when the size of the training data size 160K. Appendix reports additional experimental results from 80K and 32K training samples. The results of DEMI with three different settings of a are very close. In we report the other two settings as well.
Figure [
we only show DEMI (a 0.5). In Appendix C
For all experiments, InfoNCE substantially underestimated MI. This is due to its log-batchsize (log N) maximum, which saturates quickly relative to the actual mutual information in these regimes. The limited training data setup leads to increased errors of CCMI shown in Appendix C which is consistent with our analysis in Section 2.4
Overall, for Gaussian variables, the GM method performed very well. This is somewhat expected, as its base distribution for the flow network is itself a Gaussian. This trend begins to fall off at higher MI values for the 100-d case. For the cubic case, however, the GM method performs quite poorly, perhaps due to the increased model flexibility required for the transformed distribution.
For 20-d Gaussian variables, MINE and SMILE with both parameter settings overestimated MI in comparison to DEMI, which provided estimates that were fairly close to the ground truth values. Appendix B further investigates this behavior. For the 50-d joint Gaussian case, DEMI again pro- duced accurate estimates of MI, while MINE and SMILE underestimated MI substantially. For the 100-d joint Gaussian case, all approaches underestimated MI, with DEMI and CCMI performing best.
For 20-d joint Gaussians with a cubic transformation, all approaches underestimated MI, SMILE (T = 5) and DEMI performed best. For the 50-d and 100-d cases, all approaches understimated MI, with DEMI performing the best.
In summary, DEMI performed best or very similar to the best baseline in all the experiments. It further was not sensitive to the setting of its parameter a. Its performance relative to the other MI estimators held up with the training data size decreased.
3.2 REPRESENTATION LEARNING
Our second experiment demonstrates the viability of DEMI the differentiable loss estimate in a representation learning task. Specifically, we train an encoder on CIFARIO and CI- 100 (Krizhevsky et al. 2009) data sets using the Deep InfoMax (Hjelm et al.) 2018) criterion. Deep InfoMax learns representations by maximizing mutual information between local features of an input and the output of an encoder, and by matching the representations to a prior distribution. To evaluate the effectiveness of different MI estimators, we only include the MI maximization as the representation learning objective, without prior matching. We compare DEMI with MINE,

---

Under review as a conference paper at ICLR 2021
SMILE, InfoNCE, and JSD (Hjelm al. 2018) for MI estimation and maximization required by Deep InfoMax.
As discussed in Hjelm al. 20188), evaluation of the quality of representation is case-driven and relies on various proxics. We use classification as proxy to evaluate the representations, i.c., we use Deep InfoMax train encoder and learn representations from data set without class labels, and then we freeze the weights of the encoder and train a small fully-connected neural network classifier using the representation as input. We use the classification accuracy as performance proxy to the representation learning and thus the MI estimators. We build two separate classifiers on the last convolutional layer (conv (conv(256,4,4)) and the following fully connected layer (fc(1024) for classification evaluation of the representations, similar to the setup in Hjelm et al. (2018). The size of the input images 32 x and the encoder the same architecture as one in Hjelm et al. 2018
Gaussian, dim=20, N=160k
Cubed Gaussian, dim=20, N=160k
Enor Bis (Actual -Ped.)
Emor Bts (Actual Pred.)
Bis
BBS
dim-50, N=160K
Cubed Gausslan, N-160k
Emrr Bitt (Aatual Prod)
Emrrr (Amual Pred.)
Actual MI Dits
Actual Dia
Gaussian, d0m.100, N-180k
Cubed Gaussian, dim-100, N-160k
Erarin in Bhs (fctual- Pred)
Enar Bhs (Mctual prec.)
30-
20
Actual MI in Bts
Actual MI Btz
SMILE tau=inf
SMILE tau
GM
DEMI alpha=0.5
Estimator Legend
SMILE tau=1
InfoNCE
CCMI
Figure 1: Mutual information estimation between multivariate Gaussian variables (left column) and between multivariate Gaussian variables with a cubic transformation (right column). Closer to Zero is better. The estimation error (T(x, far y)) versus the true underlying MI are reported. These estimates are based on training data size of 160K. We only show DEMI (a 0.5) since the results of the other two parameter settings are very close. A complete table of estimate results is reported in Appendix

---

Under review as a conference paper at ICLR 2021
Results. Table I reports the top classification accuracy of CIFARIO and CIFARIOO. DEMI is comparable to InfoNCE in out of tasks and outperforms the other MI estimators by significant margin. When the encoder is allowed to train with the class labels, it becomes fully-supervised task. We also report its classification accuracy as reference. The classification accuracy based on the representations learned by Deep InfoMax with DEMI is close to even surpasses the fully- supervised case. Note that the Deep InfoMax objective we use here does not include prior distribu- tion matching regularize the encoder.
Table I: Top classification accuracy as a proxy to representation learning (Deep InfoMax without prior matching) performance with different MI estimators.
CIFARTO
CIFARTOO
conv(256,4,4) fc(1024)
conv(256,4,4)
fc(1024)
Fully- supervisd
75.39
42.27
MINE
71.36
66.30
42.52
37.23
SMILE (T 5.0)
71.88
66.92
42.74
37.48
SMILE (T 1.0)
71.12
66.22
42.13
37.10
JSD
72.79
67.94
42.78
37.76
InfoNCE
73.73
69.77
44.91
39.95
DEMI
74.09
70.16
45.59
42.02
CONCLUSION
We described a simple approach for estimating MI from joint data that is based on a neural network classifier that is trained to distinguish whether sample pair is drawn from the joint distribution or the product transform of the classifier responses. Theoretically, the estimator converges MI when the data sizes grow to infinity and the neural network capacity is large enough to contain the corresponding true conditional probability.
its marginals. The resulting estimator the average over joint data of the logit
The accuracy of our estimator is governed by the ability of the classifier to predict the true posterior probability of items in the test set, which in depends (i) the number of training sample pairs and (ii) the capacity of the neural network used in training. Thus the quality of our estimates is subject the classical issues of model capacity and overfitting deep learning. We leave theoretical analysis (which is closely related to the classifier's convergence to the true separating boundary for general hypothesis class) for future work
We discussed close connections between our approach and the lower bound approaches of MINE and SMILE and InfoNCF(CPC).. Unlike the cnffereecce-oo-entrppiie (DoF) estimator described in (McAllester & Stratos) 2020), our approach does not make use of assumed distributions. We also demonstrate empirical advantages of our approach over the state of the art methods for estimat- ing MI in synthetic and real image data. Given its simplicity and promising performance, we believe that DEMI is a good candidate for use research that optimizes MI for representation learning.
REFERENCES
Alexander Alemi, Ben Poole, Ian Fischer, Joshua Dillon, Rif A Saurous, and Kevin Murphy. Fixing broken elbo. International Conference on Machine Learning, pp. 159-168. PMLR, 2018.
David Barber and Felix V Agakov. The im algorithm: variational approach to information maxi- mization. In Advances in neural information processing systems, pp. None, 2003.
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeswar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and R Devon Hjelm. Mine: mutual information neural estimation. arXiv preprint arXiv:1801.04062. 2018.
Geeticka Chauhan, Ruizhi Liao, William Wells, Jacob Andreas, Xin Wang, Seth Berkowitz, Steven Horng, Peter Szolovits, and Polina Golland. Joint modeling of chest radiographs and radiology
8

---

Under review as a conference paper at ICLR 2021
reports for pulmonary edema assessment. In International Conference Medical Image Com- puting and Computer-Assister Intervention, pp. $29-539. Springer, 2020.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint 2tti::2000005099 2020.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in neural information processing systems, pp. 2172-2180 2016.
Georges A Darbellay and Igor Vajda. Estimation of the information the observation space. IEEE Transactions on Information Theory, 45(4):1315-1311, 1999.
adaptive partitioning of
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv preprint 1605.08880 2016.
Shuyang Gao, Greg Ver Steeg, and Aram Galstyan. Efficient estimation of mutual information for
strongly dependent variables. In Artificial intelligence and statistics, pp. 277-286, 2015.
David Harwath, Antonio Torralba, and James Glass. Unsupervised learning of spoken language with visual context. In Advances Neural Information Processing Systems, pp. 1858-1866 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9729-9738, 2020.
Olivier J Hénaff, Aravind Srinivas, Jeffrey De Fauw, Ali Razavi, Carl Doersch, SM Eslami, and Aaron van den Oord. Data-efficient image recognition with contrastive predictive coding. arXiv preprint arXiv:1905.00 09272, 2019.
Devon Hjelm, Alex Fedorov, Samuel Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint /00../88000066 2018.
Karan Grewal, Phil Bachman, Adam
Alexander Kraskov, Harald Stögbauer, and Peter Grassberger. Estimating mutual information. Phys- ical review E, 69(6):066138, 2004.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs [Online]. Available: http://yann.lect http://aann..euu..oooo/// com/exdb/mnistt 2010.
David Lopez-Paz and Maxime Oquab. Revisiting classifier two-sample tests. In International Con- ference Learning Representations, 2017.
Frederik Maes, Andre Collignon, Dirk Vandermeulen, Guy Marchal, and Paul Suetens. Multimodal- ity image registration by maximization of mutual information. IEEE transactions on Medical Imaging, 16(2):187-198, 1997.
David McAllester and Karl Stratos. Formal limitations on the measurement of mutual information. In International Conference on Artificial Intelligence and Statistics, pp. 875-884, 2020.
Sudipto Mukherjee. Himanshu Asnani, and Sreeram Kannan. Cemi: Classifier based conditional mutual information estimation. In Uncertainty Artificial Intelligence, pp. 1083-1093. PMLR, 2020.
XuanLong Nguyen, Martin J Wainwright, and Michael Jordan. Estimating divergence functionals and the likelihood ratio by convex risk minimization. IEEE Transactions on Information Theory, (6(11)55877-881 2010.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint arXiv: 807.03788 2018.

---

Under review as a conference paper at ICLR 2021
Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds mutual information. In International Conference on Machine Learning, pp. 5171- 5180, 2019.
Rajat Sen, Ananda Theertha Suresh, Karthikeyan Shanmugam, Alexandros G Dimakis, and Sanjay Shakkottai. Model-powered conditional independence test. In Advances in neural information processing systems, pp. 2951-2961, 2017.
Martin Simonovsky, Benjamín Gutiérrez-Becker Diana Mateus, Nassir Navab, and Nikos Ko- modakis. A deep metric for multimodal registration. In International conference medical image computing and computer-assisted intervention, pp. 10-18. Springer, 2016.
Jiaming Song and Stefano Ermon. Understanding the limitations of variational mutual information estimators. In International Conference Learning Representations, 2019.
Taiji Suzuki, Masashi Sugiyama, Jun Sese, and Takafumi Kanamori. Approximating mutual infor- mation by maximum likelihood density ratio estimation. In New challenges for feature selection in data mining and knowledge discovery, pp- 20, 2008
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. arXiv preprint arXiv:1906.05844 05880, 2019.
Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In 2015 IEEE Information Theory Workshop (ITW), pp. 1-5. IEEF, 2015.
Greg Ver Steeg and Aram Galstyan. Discovering structure in high-dimensional data through corre- lation explanation. In Advances Neural Information Processing Systems, pp. 577-585, 2014.
William M Wells III, Paul Viola, Hideki Atsumi, Shin Nakajima, and Ron Kikinis. Multi-modal volume registration by maximization of mutual information. Medical image analysis, 1(1):35- 51, 1996.
Shengjia Zhao, Jiaming Song, and Stefano Ermon. The information autoencoding family: A la- grangian perspective latent variable generative models. arXiv preprint arXiv:18066 1806.06514, 2018.
10

---

Under review as a conference paper at ICLR 2021
A SELF-CONSISTENCY TESTS
We assess and compare the MI estimators using the self-consistency tests proposed in (Song & Er- mon 2019). We perform the tests on MNIST images (LeCun et al. 2010). The self-consistency tests examine some important properties that a "useful" MI estimator should have, because optimizing MI is more important for many downstream machine learning applications than estimating the exact value of MI.
The self-consistency tests examine: 1) capability detecting independence, 2) monotonicity with data processing, 3) and additivity. We thus perform the following experiments, where the MNIST image set induces a data distribution and each MNIST image random variable that follows this data distribution:
MI estimation between one MNIST image and one row-masked image. Given an MNIST image X, we mask out the bottom rows and leave the top f rows of the image, which creates Y t). The estimated MI f(X, Y) should be equal or very close to zero, and Y are independent. In this context, I(X, Y) should close to when is small and be non-decreasing with t. We normalize this measurement to the final value at == 28 (the last row), which should be the maximum information.
MI estimation between two identical MNIST images and two row-masked images. Given an MNIST image X, we create two row-masked images: Y, = h(X; (1) and Y2 h(X; 12), where 4 12 + 3. Since additional data processing should not increase mutual information, f[[x, X]. [Y, fxx)/f(x, Y1) should be close to
MI estimation between two MNIST images and two row-masked images. We randomly select two MNIST images and concatenate them: [X1,X2] X2], and mask the same number of rows them: (k, (Xx;;()) [Yi, Y2]. 7([X,, X2), [Y,, Ya))/((X,, ()) should be close to 2.
We have 60k MNIST images or concatnated images for training and test set of 10k images. We train each MI estimator for 100 epochs and set the mini-batch size to 64. For all methods, concatentate inputs, then convolve with a x kernel with stride 2 and 64 output channels, then apply fully connected layer with 1024 hidden units, which then maps single output. ReLU is applied after all but the last layer.
Results. In Figure we plot the results of the three self-consistency metrics for each method. In general most methods perform well for the first measurement (monotonicity) with the exception of SMILE (r = oo), which exhibits charateristicly high variance. Other settings of SMILE and DEMI both are relatively well behaved, though overall InfoNCE performs best. For the second metric ("data processing"), all methods perform well, again aside from SMILE (r = oo) variance, In the third metric, SMILE 5 also exhibits large bump in the center (where optimal should be constant 2 overall), but both InfoNCE and DEMI converge to overall, and no method performs optimally. In general InfoNCE performs best across between the first two measures, but DEMI and SMILE (T = 1) also do well in two of three.

---

Under review as a conference paper at ICLR 2021
Self-consistency Tests #1: i(x, Y//XX, X)
DEMI
InfoNCE(CPC)
SMILE (t-1)
SMILE (T-#) (MINE)
Ralio
0.2
6
12
15
16
21
24
27
Rows used
Self-consistency Tests #2: [(X. (VI, [V1, Y2)) V1)
ideal
DEMI
InfaNCE(CPC)
SMILE (t=5)
SMILE (t=*) (MINE)
12
15
18
21
24
27
Rows used
Self-consistency Tests #3: (((1,, X2), (YL, Y2])/((11, Y1)
Ideal
DEMI
InfoNCEICPC)
(t=11
SMILE (t=5)
Ralio
SMILE (t=#) (MINE)
12
15
18
21
24
27
Rows used
Figure 2: Results of the three self-consistency tests for SMILLE (r 1,5, oo), InfoNCE, and DEMI. "Monotonicity" is top, "data processing" is middle, and "additivity" is bottom.
12

---

Under review as a conference paper at ICLR 2021
B LONG-RUN TRAINING BEHAVIOR OF SMILE
As shown in Section 3. SMILE somewhat overestimates the MI for the 20 dimensional Gaussian
case in high MI regimes (~ 30 Nats or more). This did not occur at lower MI conditions or in higher dimensions.
Further investigation showed this problem to increase training went on; illustrate this, set up new experiment on the 20 dimensional Gaussian case. We ran each setting of SMILE (T 1, 5, 2o) for 100000 training steps with batch size 64, drawing samples directly from the generating distributions. This means that the training set has effectively a very large size. We did this for three ground-truth MI values of 10,20, and 30. For comparison we also run the proposed method through the same.
This setup exactly mirrors the experiment (Song Ermon 2019) Figure in Section 6.1 of that paper, and uses their provided code and generation method, except that we replace their step-wise increasing MI schedule with a constant 10, 20, or 30 nat generator, and we run the experiment longer.
SMILE
SMILE 5.0)
SMILE
(proposed)
11.5
375
17.5-
15.0
Staa
Stpg
SMILE 1.0)
(T=
DEMI (proposed)
73.
42-
60-
60-
53
50-
22-
20
12-
Seps
Rept
(T
SMILE 5.0)
SMILE
(proposed)
250
3.0
254
330
300
200
350
2ept
Stepe
Zegt
Staps
Figure 3: Long-run behavior of SMILE and DEMI for 10 (top row), 20 (middle row), and 30 (bottom row) Nats. Analytically SMILE converges to the MINE objective for 7 - cc. Smoothed trajectories are plotted in bold, exact trajectories are the semi-translucent curve, and the actual Mu- tual information is the black constant line.
The curves for the first row of Figure behavior, particularly for that for certain distribution/dommain combinations, even though SMILE and MINE are based on lower bound of MI, they can both grossly overestimate it. This may be as (McAllester & Stratos) 2020) suggests due in part to sensitivity of the estimate of In ((x outliers. The proposed method eventually overestimates as well in both the 20 and 30 nat cases, but does not have the strongly divergent behavior exhibited by SMILE (seen partieularly strongly in oo settings).
3 show good performance with relatively stable long-term The curves the third row of Figure on the otherhand suggest
13

---

Under review as a conference paper at ICLR 2021
C RESULTS OF MI ESTIMATION ON GAUSSIAN VARIABLES
We report the results the tables below of MI estimation on Gaussian variables described Sec- tion 3.1 The estimation error (1 v) [3x, y)) and the standard deviation of the estimates are reported.
Table MI estimation between 20-d Gaussian variables trained 160K data samples. The estima- tion error (1( v) (x, y)) and the standard deviation the estimates are reported.
Actual MI
SMILE
SMILE
SMILE
InfoNCE
GM
0.1
0.3 0.0
0.0
0.2
0.1
5
0.7
0.3
0.2+0.1
1.0
0.1 = 0.0
10
0.2
0.6
-1.6
3.9
0.1 = 0.0
15
0.2
-.6
-..2
8.1
0.1 0.0
20
1.4
-.0
2..0
13.1
0.1 0.1
25
4.6
8.7
-3.33+
18.1
0.2 0.0
30
4.0
6.9
-8844
23.1
0.2 0.1
35
- 7.1
4.3
8.5
28.1
0.3 = 0.0
Actual MI
CCMI
DEMI a
0.5
DEMI
0.25
DEMI
0.75
0.1
0.0
0.0 0.0
-0.6 - 0.0
0.5 0.0
5
0.4
0.0
-0.6 0.2
0.5
10
1.1
0.6
-0.9 0.4
0.9
15
-1.7
0.9
0.0
1.7
20
5.6
1.9
0.0
2.5
25
6.8
0.8
3.9
3.92
30
10.2
+.7
1.8
4.8
35
8.0
0.4
3.6
0.2
Table 3: MI estimation between 20-d Gaussian variables with cubic transformation trained on 160K data samples. The estimation error (I(x, y) [ax, y)) and the standard deviation of the estimates are reported.
Actual MI
SMILE 7
SMILE =
SMILE = 5
InfoNCE
GM
0.1
0.2
0.0
0.0
0.1
0.4 0.4
5
1.3 0.1
0.1
1.10.11
0.9
2.7
10
0.1
1.1
1.9
3.8
3.5
15
5.6
2.6
2.7
8.2
5.7 0.4
20
9.1
5.9
5.2+1.2
13.1+0.0
8.4 0.7
25
0.6
1..
1.5
18.1
12.2 0.7
30
19.7 0.9
13.8 1.8
1..9
23.1
17.1 0.8
35
26.0
+ 0.9
20.6 1.5
18.3
1.6
28.1
22.8 = 0.9
Actual MI
CCMI
DEMI a 0.5
DEMT
0.25
DEMI a
0.75
0.1
0.0
0.1
0.06
0.5
5
0.2
1.0
0.0
1.4
10
2.4
0.7
1.5
2.9
15
2.3
4.1
3.1
4.7
20
6.4
5.1
5.5
6.1
25
3.7
7.1
8.4
9.1
30
9.8
1..7
12.5+10
13.1
35
16.7 5.2
16.9 +
17.2
14

---

Under review as a conference paper at ICLR 2021
Table MI estimation between 20-d Gaussian variables trained 80K data samples. The estima- tion error (I. y) [xx y)) and the standard deviation the estimates are reported.
Actual MI
SMILE
SMILE
SMILE
InfoNCE
GM
0.1
0.1
0.1
0.6
0.0
0.1 = 0.0
5
1.5
0.5
0.5
1.2
0.1 0.0
10
1.1
1.3
-1.1
4.0
0.2 0.1
15
1.3
1.81
3.3
8.1
0.2 0.0
20
3.3
1.7
-2.0442
13.1
0.3 0.1
25
0.2
10.3
-788
18.1
0.4 0.1
30
-5.1 11.5
7.3
-.7
23.1
0.5 0.1
35
1.6 + 12.9
6.5
4.6
28.1
0.8 = 0.1
Actual MI
CCMI
DEMI a 0.5
DEMI
0.25
DEMI
0.75
0.1
0.0
0.1+0..0
0.6 + 0.0
0.5 + 0.0
5
0.6
0.2
-0.6
0.6
10
1.1
0.4
-0.3
1.2
15
-1.7
1.2
0.0
1
20
5.6
1.7
1.2
2.1
25
6.8
2.4
1.5
1.5
30
10.2
2.5
3.4
1.4
35
-7.33
7.7
8.8
8.9 2.4
Table 5: MI estimation between 20-d Gaussian variables with cubic transformation trained on 80K data samples. The estimation error (I(x, y) [xx y)) and the standard deviation of the estimates are reported.
Actual MI
SMILE r =
SMILE
SMILE = 5
InfoNCE
GM
0.1
0.3 0.0
0.1
0.1
0.1
0.2 0.2
5
0.2
0.3
1.6
1.0
3.0
10
0.2
2.2
2.8
3.9
4.8
15
6.8
4.6
4.6
8.2
6.9 = 0.5
20
1.3
8.4
7.2
13.1+0.0
9.9 = 0.8
25
16.3 0.6
1..2
1..5
18.1
13.0 0.8
30
22.8 1.7
1.2
1..5
23.1
16.8 = 0.4
35
28.5
+ 0.9
24.3 2.6
1.3
28.1
21.4 0.6
Actual MI
CCMI
DEMI a 0.5
DEMT
0.25
DEMI a 0.75
0.1
0.1
0.1 0.0
-0.6 166000 0.0
0.5 1 0.0
5
1.4
1.1
0.2
1.7
10
1.1
2.5
1.8
0.3
15
1.9
4.8
0.5
4.9
20
2.2
0.5
6.3
0.8
25
2.4
1.72
10.3
1..8
30
15.7
1..4111
14.0
14.6
35
16.3 + 2.7
18.0
19.5
15

---

Under review as a conference paper at ICLR 2021
Table MI estimation between 20-d Gaussian variables trained 32K data samples. The estima- tion error (1, y) [xx y)) and the standard deviation the estimates are reported.
Actual MI
SMILE
SMILE 7
SMILE
InfoNCE 0.1
GM
0.1
1.1 0.3
0.3
1.2
0.1 0.0
5
0.9
0.5
0.5
0.3
1.4 0.2
10
1.2
1.05
-.60
4.0
0.8 = 0.1
15
1.3
1.3
-.9
8.1
0.8 = 0.1
20
1.5
-378
-6.
13.1
1.2
25
1.3
1.1
-39
18.1
1.6
30
2.5 1.1
0.0
3.1
23.1
2.3 0.4
35
6.9 + 0.7
12.3 0.7
8.7
28.1
3.8 = 0.8
Actual MI
CCMI
DEMI = 0.5
DEMI
0.25
DEMI
0.75
0.1
3.6
0.3
0.4
0.5 + 0.0
2.5
0.2
0.2
0.7
10
6.1
0.7
0.3
1.7
15
9.96
1.2
0.1
1.2
20
91.3
2.1
2.1
0.9
25
9..7
5.4
5.6
6.1
30
07.8.6.4
1.
9.9
11.5
35
103.2 4.6
15.8
1.75
16.5 + 2.0
Table 7: MI estimation between 20-d Gaussian variables with cubic transformation trained on 32K data samples. The estimation error (I(x, y) [xx y)) and the standard deviation of the estimates are reported.
Actual MI
SMILE 7
SMILE =
SMILE = 5
InfoNCE
GM
0.1
0.7
0.2
0.7
0.0
0.6 0.6
5
2.1
1.2
2.2
1.2
4.3
10
4.7
0.9
3.7
4.10.1
8.1 0.4
15
8.4
5.8
5.6
8.3
11.6 = 0.7
20
1.07
9.0
8.8
13.1+0000
15.9 = 0.9
25
1.3
1..0
1.12
18.1
20.9 = 0.9
30
26.8 + 1.5
2..7
1..7
23.1
25.2 1.3
35
32.3
1.3
28.0
2.88
28.1
28.6 1.3
Actual MI
CCMI
DEMI a 0.5
DEMT
0.25
DEMI a
0.75
0.1
0.0
0.1 0.0
-0.5
0.5 + 0.0
5
0.1
1.2
0.1
2.0
10
0.4
3.2
2.3
3.5
15
7.3
5.5
4.5
5.2
20
8.8
0.9
7.7
8.3
25
2.3
1.70
11.0
13.0
30
2.9
1..1
15.8
0.9
35
21.3
2..0
20.2 0.7
21.4
16

---

Under review as a conference paper at ICLR 2021
Table 8: MI estimation between 50-d Gaussian variables trained on 160K data samples. The estima- tion enror (I v) [xx y)) and the standard deviation of the estimates are reported.
Actual MI
SMILE 7 =
SMILE =
SMILE 7 5
InfoNCE
GM
0.1
0.7 +7 0.2
0.6 + 0.1
0.6 16+0.11 0.1
0.0
0.1 0.0
5
1.1
0.2
0.1
1.7
3.400.3
10
0.2
0.1
0.6
4.1
1.0
15
0.6
1.5
0.9
8.2
0.5
20
0.9
2.0
-0.9
13.1
0.5 0.1
25
1.1
4.3
0.7
18.1
0.6 0.1
30
1.9
7.1
2.7
23.1
0.7 = 0.1
35
6.6
6.9
1.9
28.1
0.9 0.2
40
3.5
8.8
5.1
33.1
11
45
2.7
9.6
9.0
3.01
14
50
3.5 7.0
5.2
3.1
43.1
1.6 = 0.3
55
10.2 4.5
19.2 4.7
15.3 + 5.6
48.1 + 0.0
1.8 = 0.3
Actual MI
COMI
DEMI a 0.5
DEMI a
0.25
DEMI OV = 0.75
0.1
0.2 0.0
0.1
-0.5
0.5
5
0.8
0.1
-0.6 0.1
0.7
10
0.5
0.5
0.5
1.3
15
1.8
1.7
0.8
1.4
20
2.0
3.142.0
2.0
2.1
25
2..
18818
2.3
1.7
30
6.3
3.1
3.0
1.1
35
8.1
-.1
2.0
0.4
40
7.8
6.7
2.7
4.5
45
7.7
-.1
3.5
-.043.
50
-19.6
3.3
4.2
5.3
55
-18.6 12.5
4.6
6.4
7.7
17

---

Under review as a conference paper at ICLR 2021
Table MI estimation between 50-d Gaussian variables with cubic transformation trained on 160K data samples. The estimation error (T(z, v) [ax v)) and the standard deviation of the estimates are reported.
Actual MI
SMILE T xc
SMILE
SMILE
InfoNCE
GM
0.1
0.5
0.1
0.5 0.1
0.5
0.1
0.4 = 0.4
5
3.1 0.3
2.5 0.1
3.0
1.3
4.9 0.5
10
7.0 0.5
4.6
5.2
4.2
8.7 0.7
15
9.4
0.5
6.7
8.0
=
20
0.6
1..
9.2
13.1
25
0.6
1..2
12.6
18.1
17.4
30
0.9
1..
1..6
23.1
20.1
35
2
2.7
2..5
28.1
21.2 = 1.5
40
29.6
2.1
2..1
33.1
24.5 2.1
45
32.8 + 1.5
2.1
31.6+1.3
3..1
26.8 1.4
50
36.6 1.1
2.3
2.1
4.0
30.1 = 2.1
55
41.2 + 1.9
17.0
42.3 1.8
48.1 0.0
32.9 = 2.6
Actual MI
CCMI
DEMI
0.5
DEMI
0.25
DEMI
0.75
0.1
0.1
0.1
-0.6 0.0
0.5 - 0.0
0.1
1.1
1.0
2.3 0.2
10
4.2
3.3
3.1
4.1 0.2
15
6.6
0.5
5.6
6.3
20
0.8
0.6
7.4
7.7
25
9.2
9.4100
10.0
9.8
30
2.9
1..4
12.3
12.0
35
5.2
1..7
1.84
13.0
40
15.0 6.5
1..7
18.7
17.8 2.0
45
16.2+3.0
1.7
21.1
20.6 1.9
50
3.2
2.1
2.22
25.4
55
22.8 3.4
1.6
29.0 1.3
29.9 1.4
18

---

Under review as a conference paper at ICLR 2021
Table 10: MI estimation between 50-d Gaussian variables trained on 80K data samples. The estima- tion enror (I v) [xx y)) and the standard deviation of the estimates are reported.
Actual MI
SMILE
SMILE
SMILE 7 5
InfoNCE
GM
0.1
1.4 0.3
1.3 0.2
1.5 0.3
0.5
0.1 = 0.0
5
1.1
0.4
1.0
0.2
4.6 0.2
10
0.3
0.6
0.5
4.1
5.5 0.6
15
0.5
0.9
1.3
8.3
4.5 0.3
20
0.9
2.8
0.5
13.1
3.5 0.5
25
6.2
4.2
3.6
18.1
2.
30
8.6
7.8
4.6
23.1
3.2 = 0.4
35
5.9
1..4
4.5
28.1
3.4 0.5
40
6.1
5.7
4.2772
33.1
3.8 0.6
45
7.0
1..6
5.3
3..
4.2 0.9
50
14.4 + 11.4 20.8
21.5
1.64
4.0
5.3 1.0
55
10.2
32.3 7.4
23.5
6.7
48.1 + 0.0
6.7 1.0
Actual MI
CCMI
DEMI a 0.5
DEMI a = 0.25
DEMI O = 0.75
0.1
0.2 0.0
0.1
-0.2
0.6 0.0
5
1.1 0.8
0.2
-0.5
1.1
10
0.5
1.140.4
0.4
1.5
15
1.8
1.7
0.9
1.2
20
2.0
3.9
2.8
4.1
25
2..9
5.6
4.8
1.7
30
6.3
5.
1.3
2.0
35
8.1
9.6
9.2
10.1
40
7.8
4.1
11.6 3.5
11.3 4.3
45
7.7
10.6
1..1
13.9 +
50
-19.6
4.99
15.0
18.1 436
55
-18.6 12.5
2.8
20.2 3.2
24.1 3.8
19

---

Under review as a conference paper at ICLR 2021
Table MI estimation between 50-d Gaussian variables with cubic transformation trained on 80K data samples. The estimation error (T(z, v) [(x v)) and the standard deviation of the estimates are reported.
Actual MI
SMILE
SMILE
SMILE
InfoNCE
GM
0.1
1.1 0.2
1.2 + 0.2
1.1
0.2
0.9 0.4
5
0.1
2.8 0.2
3.0+
1.6++.11
5.5 0.6
10
6.4
4.4
5.0
4.0
10.0 = 0.5
15
0.2
0.7
0.3
8.0
13.9 0.4
20
1..7
1.
9.8
13.1
18.1 = 0.4
25
1.7
1..
1..9
18.1
22.0 = 0.4
30
2.7
0.67
18.
23.1
25.8 = 0.8
35
1.0
2..4
2..4
28.1
29.9 0.8
40
+.7
3.11
2..
33.1
33.9 0.8
45
32.3 1.0
2.3
2..2
3..1
37.4 = 1.0
50
37.7
448
1.2
4.0
41.7 = 1.5
55
41.7 + 1.5
2.2
46.8 + 1.8
48.1 0.0
45.3 0.8
Actual MI
CCMI
DEMI
0.5
DEMI
0.25
DEMI
0.75
0.1
0,2 0.0
0.2
-0.5 0.0
0.5 - 0.0
0.1
2.1
1.1
2.5 0.3
10
4.8
4.1
3.2
4.4 0.3
15
0.9
6.0
5.7
6.4
20
1.6
7.7
7.4
8.5
25
2.8
10.1
10.
10.6
30
2.6
1..0
1..4
14.2
35
2.9
1..6
1..0
18.3 1.4
40
3.3
2.2
2.17
21.2 1.6
45
3.9
2..0
25.4 1
24.7
50
26.1
1.5
29.8
1.2
55
3.5
3..1
34.1
84.22..3
20

---

Under review as a conference paper at ICLR 2021
Table 12: MI estimation between 50-d Gaussian variables trained on 32K data samples. The estima- tion enror (I v) [xx y)) and the standard deviation of the estimates are reported.
Actual MI
SMILE =
SMILE
SMILE 7 5
InfoNCE
GM
0.1
2.2 + 0.8
2.2 0.9
1.9 + 0.8
1.2
0.1 = 0.0
5
1.7
0.4
1.2
5.7
5.0 0.0
10
0.2
0.8
0.5
4.1
9.6 0.1
15
3.7
1.5
0.2
8.4
13.2 0.4
20
1.1
1.3
0.7
13.1
15.60.1
25
5.9
5.7
0.5
18.1
0.8
30
3.7
5.6
1.6
2..1
17.2 1.8
35
1.7
9.3
5.7
28.1
18.0 1.1
40
10.3
16.7
1.
33.1
18.6 = 1.3
45
15.0 1.5
1.5
26.
38.1
20.7 1.6
50
18.8 + 1.2
26.9
2.3
4.0
21.6 = 1.2
55
24.4 1.1
31.6 1.1
28.4
48.1 0.0
23.8 2.1
Actual MI
CCMI
DEMI a
0.5
DEMI a
0.25
DEMI
& 0.75
1..
0.2
0.8
0.9 + 0.1
5
74.3
0.7
0.1
1.2
4.2
1.0
0.1
1.6
15
3.3
0.0
1.1
2.7
20
7.3
4.6
3.3
5.0
25
6.2
6.8
6.5
7.6
30
7..
10.3
10
10.91.5
35
5.5
1..3
1.6
15.5 + 2.1
40
3.7
16.4
1..9
19.3 + 1.8
45
10.
1.9
2.14
24.9 1.4
50
8.5
2.2
26.3
28.2 2.2
55
105.1
3..4
33.4
14.21.7
21

---

Under review as a conference paper at ICLR 2021
Table MI estimation between 50-d Gaussian variables with cubie transformation trained on 32K data samples. The estimation error (7(z,)) y) [(x v)) and the standard deviation of the estimates are reported.
Actual MI
SMILE
SMILE T
SMILE
InfoNCE
GM
0.1
2.0 + 0.7
2.2 + 0.5
2.2 + 0.5
0.5
1.4 = 0.6
5
2.7
2.7
1.7
4.5 0.9
10
5.1
6.9
4.1
4.3
9.3 = 1.1
15
0.2
1.4
0.3
8.4
13.0 = 1.1
20
1.3
1.7
11.9
13.2
17.9 = 1.3
25
0.5
2.70
1.92
18.1
23.0
30
0.6
2.1
2..
23.1
26.4=.22
35
0.9
0.77
2.9
28.1
31.2 = 0.8
40
2.5
3.0
3.19
33.1
36.8 1.1
45
31.3 1.0
4.8
3.8
38.1
40.6 = 1.0
50
35.7 0.9
47.8
4.3
43.1
46.0 = 1.1
55
41.0 + 1.3
53.2 + 1.3
50.0
0.8
48.1
51.0 1.3
Actual MI
CCMI
DEMI
0.5
DEMI
0.25
DEMI
0.75
0.1
0.1
0.5 1 0.1
-0.1 0.1
0.6 0.0
2.7
2.1
1.7 0.1
2.6 0.3
10
4.9
4.0
3.7
4.1
15
1.6
0.5
5.8
5.9
20
1.5
8.6
8.6
0.8
25
1..6
11.9
1..1
12.0
30
1.0
16.6
1.
16.2
2.8
2.7
2.9
20.8
40
23.3 2.0
2.7
24.8
25.3
45
28.3
3.0
0.6
30.1
50
33.3 3.2
1.0
3.
35.1
55
nan + nan
38.9 1.6
38.4
19.5122
22

---

SMILE Under review as a conference paper at ICLR 2021
Table 14: MI estimation between 100-d Gaussian variables trained 160K data samples. The estimation error (1( y) (xx v)) and the standard deviation of the estimates reported.
Actual MI
SMILE
SMILE
InfoNCE
GM
0.1
1,2 + 0.2
1.3
1.1 0.2
0.1
0.1 = 0.0
5
1.8 0.2
0.2
0.1
2.3
5.0 0.0
10
0.1
1.5
0.9
4.1
9.7
15
0.4
2.1
0.4
8.4
13.6
20
0.6
3.7
1.3
13.2+
15.3 0.8
25
9.0
8.2
1.9
18.1
15.2 = 0.8
30
1.6
1..8
8.6
23.1
15.9 0.8
35
1.5
14.6
1.
2..1
16.3 = 0.9
40
1..7
17.5
1.73
33.1
16.1 1.0
45
2.2
3.9
18.6
38.1
16.1 = 0.9
50
2.6
2..2
2..9
43.1
16.9 1.3
55
4.5
2.93
2.0
4.0
17.7 1.4
60
29.3
28.7
29.0+
53.1
17.9 1.8
65
6.1
3..
33.2
58.1
18.7 1.0
70
3.5
4..7
30.8
6.0
20.2 1.3
75
35.7 8.6
9.1
9.3
6.0
21.3 = 2.0
80
34.1
13.6
4
40.2 + 6.9
73.1 0.0
21.8 1.1
Actual MI
CCMI
DEMI a 0.5
DEMI e
0.25
DEMI a
0.75
0.1
0.1
0.7
0.2
0.6
5
1.3
1.1
0.5
1.6 0.3
10
1.1
1.7
1.3
2.5
15
1.
0.9
2.7
3.1
20
2.1
4.6
4.2
5.2
25
3.7
1.6
8.4
2.1
30
1.7
6.9
11.3
1.5
35
0.2
7.8
1..7
11.4
40
2.1
9.5
16.6
1.90
45
6.2
10.
2..5
1..1
50
2.1
10.6
2..1
1..
55
8.8
16.6
2..0
21.5
60
3.7
16.
2..0
23.0
65
12.0
6.0
5.2
27.0
70
9.3
6.3
3..2
28.2 08.2660 6.0
75
12.5 7.6
2.77
3.92
32.4 2.84228
80
12.4 12.3
2.4
4.72
39.9 1 5.4
23

---

Under review as a conference paper at ICLR 2021
Table 15: MI estimation between 100-d Gaussian variables with cubic transformation trained on 160K data samples. The estimation error (1 y) (z, v)) and the standard deviation of the estimates are reported.
Actual MI
SMILE = 00
SMILE T
SMILE T =
InfoNCE
GM
0.1
1.1
1.2
1.1
0.1 0.0
1.2 = 0.8
5
0.1
0.2
3.0
2.1
5.7 3.7115 1.5
0.1
7.7
5.5
4.4
10.3 1.5
15
0.3
11.2
8.5
8.4
15.6 = 1.8
0.2
1..6
1.2
13.2
19.8=11 1.1
25
1.3
2.1
1..8
18.1
23.8 = 0.9
30
17.9
2.5
2.7
23.1
29.3 = 1.2
35
0.8
3.8
2.6
2..
33.2 = 0.9
40
0.8
0.3
0.8
0.0
37.8 = 1.2
45
1.2
4.9
0.5
3..
42.0 1.1
50
1.1
4.5
0.6
4.0
45.8 = 1.2
55
38.7
5.5
4.0
4..1
50.5 = 1.1
60
4.4
5.5
0.7
5.0
54.4 = 1.4
4..
+.8
5.6
58.1
58.2 0.8
70
53.4 1.2
67.0
0.8
6.0
62.8 1.5
58.6 + 1.7
72.2 1.0
68.1
6.0
67.0 = 1.3
80
63.4 + 1.5
77.2 = 1.0
72.8 = 0.8
0.0
Actual MI
COMI
DEMI
0.5
DEMI
0.25
DEMI
0.75
0.1
0.3 E 0.0
0.1
-0.3 = 0.0
0.5 - 0.0
0.1
0.1
1.8
2.9
10
5.0
4.3
4.2
5.2 0.3
15
0.8
7.2
6.3
0.8
20
9.1
9.9
8.2
9.7
25
11.7
11.7
11.2
1.19
30
2.9
1.22
14.2
1.97
35
2.5
1.43
1..
1.24
40
4.1
2.7
2..8
45
1.9
1..0
2.1
27.0
50
1.6
1.0
3..0
1.1
55
4.8
1.9
1.2
1.9
60
3..
2.6
39.7
3.74
65
2.3
4..4
2..0
2.0
70
3.0
4..4
48.1
48.1
75
46.1 1.8
5.0
52.7
5.95
80
52.6 + 1.7
58.8 1.5
57.4 2.9
58.5
24

---

SMILE Under review as a conference paper at ICLR 2021
Table 16: MI estimation between 100-d Gaussian variables trained 80K data samples. The estimation error (a, y) (xx v)) and the standard deviation of the estimates reported.
Actual MI
SMILE
SMILE
InfoNCE
GM
0.1
2.1 + 0.6
2.3 + 0.7
2.3 0.3
0.1
0.1 = 0.0
5
2.0
1.0
1.2
4.2
5.0 0.0
10
3.1
0.5
1.
4.1
100
15
4.2
3.2
2.4
8.6
14.7=0.1
20
6.1
5.3
4.9
13.3
19.0 = 0.2
25
1.2
7.9
6.9
18.2+000
22.8 = 0.7
30
1.8
12.3
9.1
23.1
24.6 = 1.0
35
2.3
16.8
1..5
2..1
26.2 = 1.4
40
18.9
2.34
18.3
33.1
27.5 1.1
45
2.6
23.9
1.9
38.1
28.2 0.9
50
2.9
2..0
1.7
43.1
29.9 1.1
55
6.7
2.04
2..0
4..
31.1 1.5
60
+.4
6.6
2..
53.1
31.7 1.3
65
2.1
10.6
30.8
58.1
33.5 = 1.6
70
30.5 8.5
2.1
3..0
6.0
35.3 1.2
75
40.0 + 13.0
40.0
6.7
6.0
36.3 0.9
80
41.1 9.2
48.8 + 8.8
44.2 5.6
73.1 0.0
38.2 1.4
Actual MI
COMI
DEMI a 0.5
DEMI e
0.25
DEMI a
0.75
0.1
0.1
0.2
1.2
5
1.3
0.2
1.2
1.8 0.3
10
1.1
2.1
1.4
2.9
15
1.
0.9
2.4
0.8
20
2.1
1.0
4.2112
1.5
25
3.7
8.9
8.4
1.2
30
1.7
1.8
1.2
13.
35
0.2
1..
14.8
1..0
40
2.1
1..0
18.1
20.0 2.2
45
6.2
3.5
2..0
2.7
50
2.1
22.
2..0
2..6
55
8.8
23.2
2.71
30.
60
3.7
2..
2.5
3..4
65
12.0
3.85
36.2 4.0
3.5
70
9.3
35.1
42.1
2.3
75
12.5
37.8
45.5
45.5
80
12.4 12.3
4..1
48.7 3.5
51.0
25

---

Under review as a conference paper at ICLR 2021
Table 17: MI estimation between 100-d Gaussian variables with cubic transformation trained on 80K data samples. The estimation error (1 y) (xx u)) and the standard deviation of the estimates are reported.
Actual MI
SMILE = 0C
SMILE T
SMILE
InfoNCE
GM
0.1
2.3 0.4
0.4
0.5
0.3
5.5 2.0
5
3.1
0.3
3.4
2.2
9
10
0.1
8.3
0.2
4.0
13.6=24 2.2
15
8.6
1..9
9.4
8.5
18.6
20
11.7
1.3
13.6
13.2
22.8 1.8
25
1.3
2..3
1.3
18.1
27.9 1.7
30
1.4
2..1
2.4
23.1
32.2 = 1.5
35
0.5
0.5
0.8
28.1
36.5 = 0.9
40
0.4
0.4
0.7
33.1
41.3 = 1.2
45
0.8
4.8
0.5
38.1
44.7 =.7=2.1 2.1
50
0.7
4.7
4.7
43.
50.0 = 1.4
55
4.1
5.8
4.82
48.1
55.1 1.1
60
46.6 +
0.75
5.82
5..1
59.8 1.5
65
1.0
0.84
5.83
58.1
65.0 0.9
70
56.5 +.5
0.7
0.94
6.1
68.7 = 1.4
75
61.7
0.6
0.4
0.0
72.9 1.3
80
67.3 #
79.3 0.6
74.9
0.5
0.1
78.4 1.3
Actual MI
CCMI
DEMI
0.5
DEMI
0.25
DEMI
0.75
0.1
0.1
0.8
0.2
0.6 - 0.0
3.2
2.9
2.3
2.9 0.4
10
5.1
4.8
4.2
4.8 1.1
15
1.2
6.6
6.3
1.1
20
1.5
9.6
8.7
0.9
25
1..1
1..7
11.9
1..7
30
2.1
16.2
15.4
16.1
35
2874
20.4+100
1.7
20.6
40
3.9
2..7+
2.8
2
45
2..6+
2..6+
+.1
1.1
50
2.6
1.
3.8
34.0
55
3.5
1.4
36.9
3.5
60
30.
4.9
4.60
42.6
65
4.0
1.6
4.0
47.8
70
2.3
52.1 1.3
51.6
52.5
75
51.
57.1
5..3
57.9 1 1.8
80
2.5
62.1 0.9
6..4
62.4 + 1.5
26

---

Under review as a conference paper at ICLR 2021
MI estimation between 100-d Gaussian variables trained 32K data samples. The estimation error (a, y) (xx v)) and the standard deviation of the estimates reported.
Table 18:
Actual MI
SMILE
SMILE
SMILE
InfoNCE
GM
0.1
3.6 1.2
3.5 1.2
3.7
2.2
0.1 = 0.0
5
3.0
1.2
2.2
6.2
5.0 = 0.0
10
4.2
2.1
2.1
5.2
=
15
0.4
4.0
2.9
8.1
15.0 = 0.0
20
9.5
7.0
5.7+1.1
13.4
20.0 0.0
25
1.2
1..4
7.8
18.2+
25.0 = 0.0
30
1.6
1..2
10.6
23.1
29.8 0.1
35
2.0
16.1
15.7
2..1
34.4 0.3
40
17.9
18.4
14.9
33.1
38.9 0.5
45
4.2
2.70
17.7
38.1
43.4 0.3
50
2.8
2..9
23.1
43.1
47.4= 0.6
55
25.9 2.2
3.8
2.9
4..
51.3 = 0.7
60
30.1 2.1
2.9
3.2
53.1
54.9 1.5
65
1.0
3.2
36.3
58.1
58.2
70
1.7
1.6
6.0
61.7
75
44.6 + 0.8
0.4
4.9
6.0
65.2 = 2.2
80
49.1 1.8
57.6 1.2
54.1 + 0.6
73.1 + 0.0
68.9 1.8
Actual MI
CCMI
DEMI a 0.5
DEMI 0
0.25
DEMI
& 0.75
19.6
0.1
0.3
1.6 + 0.1
5
63.3 3.7
0.3
1.4
2.0 0.3
3.6
2.1
1.5004
1.9
15
5.3
0.9
3.7
4.5
20
3.4
6.9
6.1
7.0
25
3.67
1..1
0.8
1..2
30
3.5
12.9
12.6
15.1
35
4.4
17.6
16.7
19.1
40
4.8
2.84
2.0
+3.0
45
2.8
2.52
2.15
27.4
50
4.3
3.28
1..4
3.7
55
6.9
34.5
33.8
3..0
60
4.6
2.2
3.2
4
65
6.0
4.5
4..7
4.64
70
5.0
1.6
1.3
51.6
75
103.9
2..6
53.9
54.8
80
1.0
59.2
1.2
27

---

Under review as a conference paper at ICLR 2021
Table 19: MI estimation between 100-d Gaussian variables with cubic transformation trained on 32K
data samples. The estimation error (1 y) (2x, u)) and the standard deviation of the estimates are reported.
Actual MI
SMILE = 0C
SMILE =
SMILE
InfoNCE
GM
0.1
0.6
2.7
0.5
0.8
2.9 2.7
5
0.2
0.5
0.3
2.4
6.1
10
6.2
0.3
0.2
4.6
9.8 1.6
15
9.7
13.7
10.5
8.6
13.4
20
1.2
18.6
14.9
13.4
17.3 1.7
25
+.3
2.6
1..5
18.2
21.7 = 1.2
30
2.2
0.7
2..4+04
2.12
26.1 = 2.5
35
2.4
0.5
2.5
28.2
30.7 2.1
40
0.5
0.4
0.5
3..1
36.4 1.9
45
3..7
0.7
39.3
38.1
40.4 = 1.9
50
0.4
0.5
4..
4.1
46.0 = 1.2
55
4.4
0.8
5..
48.1
49.1 1.7
60
1.0
59.1
5.75
5..2
54.6 1.2
65
+.5
0.8
6.60
58.2
58.4 1.5
70
58.5 + 0.7
0.5
6.81
6..1
64.2 = 0.9
75
63.4 #
0.8
0.5
0.1
67.7 - 2.0
80
68.9 + 0.8
79.4 1.3
75.7
0.7
74.1 1.6
Actual MI
CCMI
DEMI
0.5
DEMI
0.25
DEMI
0.75
0.1
2.0 0.3
2.0
0.6
0.8 - 0.1
0.1
3.1
2.3
2.7 0.4
10
6.7
4.8
4.2
4.0 0.5
15
2.7
7.0
0.5
6.6
20
2.3
1..5
1..3
10.3
25
1.2
14.5
14.2
13.2
30
2.2
1.8
17.6
1.2
35
2.1
2.7
22
2.75
40
2..7
2..6
26.7
0.7
45
1.9
1.1
+
3.9
50
2.7
3..4
3.9
3.27
55
1.5
4.84
1.1
1.2
60
1.3
4..2
4..6
4..5
65
46.1 2.9
51.1
101
5..0
70
2.6
1.47
5..9
55.5
75
56.3
6.2
6..3
60.1
80
3.2
6.5
64.9
6..0
28


Reviews:
[{"review_id": "vYw8U3TKzHn", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Summary:\n\nThis paper proposes DEMI, a discriminative approach to estimate mutual information (MI). The main idea is that, instead of learning (generative) distributions of joint and marginals, learning a single likelihood ratio that is discriminative and hence more tractable: a posterior $p(z | x, y)$ trying to distinguish between the joint distribution $p(x, y)$ and the product distribution $p(x)p(y)$. Once the posterior is learned, it can be used to estimate the MI.\n\n## Strength:\n\n- This paper studies a very important problem for the representation learning community -- mutual information has been a very powerful, principled technique for deep representation learning and many applications, but there are many challenges in scalable and accurate, low-variance estimation. Therefore developing an accurate MI estimator is of high importance and significance.\n- I find the idea of “lifting” the distribution and converting the MI estimation problem into a discriminative setting interesting, and looks novel. The method makes sense, and the training procedure is very simple, achieving better estimation than the baselines.\n- This paper is well-placed and contains comprehensive discussion of recent works about the limitations and research challenges on mutual information estimation. The mathematical connection to existing methods (MINE, InfoNCE, SMILE, etc.) provide an interesting insight. \n\n## Weakness:\n\n- The method only discusses estimation of mutual information, not maximization of MI for representation learning.\n- The biggest weakness of this paper would be: experiments. The training data used in the experiments is either low-dimensional or synthetic, so there remains a question about how well this method will scale to a high-dimensional, and challenging deep learning setting. As in (Song & Eromn 2019), empirical analysis on the CIFAR-10 dataset would be needed --- if provided, my rating would increase.\n- Bias and tradeoff analysis (similar to Song & Ermon 2019) is missing.\n\n## Question:\n\n- The hyperparameter $\\alpha$ is said to be set to 1.0 (Section 3), which does not seem feasible based on the Equation (5) (7). Was it meant to be 0.5? Can the authors clarify on this? Also, I am curious how sensitive DEMI is on the choice of the prior hyperparameter $\\alpha$. This would be a good analysis to have for the completeness of the paper.\n\n## Additional comments:\n\n- Section organization: I suggest having an introduction as a separate section, with methods being the following section. Section 3 (experiments) and 4 (results) can be combined. For section 2 (Related work), a different name could be considered because the main content here additionally includes a theoretical connection to existing approaches, which is in fact an important contribution of the paper.\n- The plots in figure 2 are not properly scaled, with too many lines overlapping one another. I suggest the authors improve the plot for better readability.\n- Typo in section 4.2: overalll. \n- Please place a whitespace after the column (DEMI:) in the title.", "extra_scores": {}, "extra_text": {}}, {"review_id": "FeamT628qnp", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Seems like the most direct way to estimate mutual information using a classifier. I like this work because it is much more straight-forward than the prior work such as MINE. It shows sufficient performance on the experiments shown.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Fcn5tDhLYov", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed a discriminative estimator for mutual information, to alleviate the shortcomings of the existing estimators such as MINE and SMILE. A classifier was built to decide whether the sample is drawn from the joint distribution or the independent one (product of marginals). Theoretical justification and experimental results were provided to support the proposed estimator. The paper was written with clarity and easy to follow.\n\nHere are some detailed comments on the technical contribution of this paper:\n1) There is a closely related piece of work in the literature (see below). They also proposed a discriminative estimator for KL divergence, with mutual information as a special case. It would be nice if the authors could relate to this existing work, and provide experimental comparison to their estimator.\nMukherjee, Sudipto, Himanshu Asnani, and Sreeram Kannan. \"CCMI: Classifier based conditional mutual information estimation.\" In Uncertainty in Artificial Intelligence, pp. 1083-1093. PMLR, 2020. (https://arxiv.org/pdf/1906.01824.pdf)\n\n2) From Figure 1 right column, we see that all estimators (including the one proposed) underestimate the mutual information when it is high. Could the authors give more analysis and explanation on this phenomenon?\n\n3) It would be nice if the authors could provide experimental results on more realistic datasets, and show the advantage of the proposed estimator when it is used for other downstream tasks. Often, estimating the mutual information is not the end goal, but an intermediate step to achieve other goals (see the MINE paper for examples).\n\n4) A minor point: In equation (10) the last part, it should be (1-z) * log( 1 - q(...) ) instead of (1-z) * ( 1 - log(q(...)) ).\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "FT8eTeRCuF", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work suggests a new discriminative mutual information estimator that relies on a classifier to directly estimate the log density ratio of p(x,y)/p(x)p(y), without variational lower bound. In general, the idea is easy to follow and simple simulations are done to demonstrate its effectiveness. However, I still have some concerns:\n1. A classifier based MI estimator reminds me of a closely related problems: the independence test. For the latter, there are also a few recent proposals based on a classifier to distinguish p(x,y) from p(x)p(y). I understand the methodologies are different, but I still feel some motivations are similar. It would be better if authors can clarify this point.\n[1] Lopez-Paz, David, and Maxime Oquab. \"Revisiting classifier two-sample tests.\" ICLR 2017.\n[2] Sen, Rajat, Ananda Theertha Suresh, Karthikeyan Shanmugam, Alexandros G. Dimakis, and Sanjay Shakkottai. \"Model-powered conditional independence test.\" NeurIPS 2017.\n\n2. Authors discussed the theoretical optimum of their estimator when the number of samples approching to infinity. In the simulations, it seems that the number of training samples is also very large (e.g., 160k). What will happen in case of moderate or small number of samples?\n\n3. For me, the simulation on the self-consistency tests does not demonstrate a big advantage of DEMI, especially considering that a few competitors are not included (e.g., GM mentioned in [Song and Ermon, 2019]). On the other hand, lots of work on variational MI (including this one) claim the great potential on representation learning with either mutual information maximization or information bottleneck. However, validations are totally missing. In this sense, it would be much better if authors can provide a simple representation learning demo, just like [Hjelm et al., 2018]. What will happen if we replce MINE with DEMI.\n\n4. It seems from Fig. 1, the advantage of the estimator becomes more obvious with the increase of dimension. Can authors provide some explanation or theoretical analysis?\n\n5. It seems to me the work is prepared in a quick time. There are a few typos (e.g., the 6 line in the second paragraph of page 3, $\\hat{p}(x,y;\\hat{D})$ should be $\\hat{p}(y;\\hat{D})$). The clarity and location of figures can be improved. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: In the paper, the authors propose a new method for estimating the mutual information based on a neural network classification that is fairly straight forward. The proposed method compares relatively well with known methods for estimating mutual information with a very large number of samples. The main issue of this classifier (a neural network) is that it requires that a classifier that discriminates between x, y pairs coming from p(x,y) and x, y pair coming from p(x)p(y) (this is done via reshuffling). The reviewers point out that the procedure is interesting, but it does not perform significantly better than the other proposed methods.

Also, I want to add that the proposed method is trained by using a given NN trained with 20 epochs and a mini-batch of 64. This is a significant issue because if we train the NN to reduce the validation error the posterior probability estimates are typically overconfident a significant work is being done to calibrate them. Why 20? How do we select this number if we cannot use a validation set? With less training example does 20 also work? This is very relevant because in the areas in which p(x,y)/p(x)p(y) is low for very high MI values getting these estimates correctly is critical. The classifier does not need to perform accurately in classification, but an estimation of the posterior probability and NNs will tend to be overconfident here and provide a biased estimate for these values. It will also provide an overestimate probability in the area that both p(x,y) and p(x)p(y) are high. 

Finally, the authors reference the paper by Nguyen, Wainwright, and Jordan, but they do not acknowledge how that paper actually estimates log(p(x,y)/p(x)p(y)) similarly. That paper is very general and theoretical, and this paper can only be understood as a particular implementation of their solution. I think the authors missed that point in their paper. Also, I think the authors should acknowledge the papers that have come before using nearest neighbor or histograms for entropy estimation. 
Decision: Reject