Improved Mutual Information Estimation
Authors: Youssef Mroueh*
Igor Melnyk*
Pierre Dognin*
Jerret Ross*
Tom Sercu*
Venue:  (2020)
Keywords: mutual information
variational bound
kernel methods
Neural estimators
mutual information maximization
self-supervised learning

Abstract: We propose a new variational lower bound on the KL divergence and show that the Mutual Information (MI) can be estimated by maximizing this bound using a witness function on a hypothesis function class and an auxiliary scalar variable. If the function class is in a Reproducing Kernel Hilbert Space (RKHS), this leads to a jointly convex problem. We analyze the bound by deriving its dual formulation and show its connection to a likelihood ratio estimation problem. We show that the auxiliary variable introduced in our variational form plays the role of a Lagrange multiplier that enforces a normalization constraint on the likelihood ratio. By extending the function space to neural networks, we propose an efficient neural MI estimator, and validate its performance on synthetic examples, showing advantage over the existing baselines. We then demonstrate the strength of our estimator in large-scale self-supervised representation learning through MI maximization.

Under review as a conference paper at ICLR 2020
IMPROVED MUTUAL INFORMATION ESTIMATION
Anonymous authors Paper under double-blind review
ABSTRACT
We propose variational lower bound on the KL divergence and show that the Mutual Information (MI) can be estimated by maximizing this bound using a witness function hypothesis function class and an auxiliary scalar variable. If the function class is in a Reproducing Kernel Hilbert Space (RKHS), this leads to jointly convex problem. We analyze the bound by deriving its dual formulation and show its connection to a likelihood ratio estimatiin problem. We show that the auxiliary variable introduced in our variational form plays the role of a Lagrange multiplier that enforces a normalization constraint on the likelihood ratio. By extending the function space to neural networks, we propose an efficient neural MI estimator, and validate its performance on synthetic examples, showing advantage over the existing baselines. We then demonstrate the strength of our estimator large-scale self-supervised representation learning through MI maximization.
INTRODUCTION
Mutual information (MI) is an ubiquitous measure of dependency between a pair of random variables, and is one of the corner stones of information theory. In machine learning, the information maxi- mization principle for learning representation from unlabeled data through self-supervision (Bell & Sejnowski, 1995) motivated the development of many MI estimators and applications (Hjelm et al., 2019; Noroozi & Favaro, 2016; Kolesnikov al., 2019; Doersch et al., 2015; van den Oord et al., 2018b; Hu et al., 2017). The information bottleneck (Tishby et al., 1999; Kolchinsky et al., 2017) is another principle that triggered recent interest in mutual information estimation. MI is also used to understand the information flow in neural networks, in learning clusters ( (Krause et al., 2010) and in regularizing the training of Generative Adversarial Networks (GANs) (Chen et al., 2016).
In many of those machine learning applications and other scientific fields, one has to estimate MI given samples from the joint distribution of high dimensional random variables. This is a challenging problem and many methods have been devised to address it. Since MI is defined as the Kullback- Leibler (KL) divergence between the joint distribution and the product of marginals, one can leverage non parametric estimators of f-divergences (Nguyen et al., 2008; Nowozin et al., 2016; Sriperumbudur et al., 2009). Specifically of interest to us is the Donsker- Varadhan (DV) representation of the KL divergence (Donsker & Varadhan, 1976) that was used recently with neural networks estimators (Belghazi et al., 2018; 8; Poole et al., 2019). Other approaches to estimating the MI are through finding lower bounds using variational Bayesian methods (Alemi et al., 2016; 2017: Barber & Agakov, 2003; Blei et al., 2017), as well as through geometric methods like binning (Kraskov et al., 2004).
In this paper we propose new estimator of MI that can be used in direct MI maximization or as a regularizer, thanks to its unbiased gradients. Our starting point is the DV lower bound of the KL divergence that we represent equivalently via joint optimization that we call D-DV on a witness function of and an auxiliary variable in Section 2. In Section 3. we show that when the witness function f is learned in Reproducing Kernel Hilbert Space (RKHS) the DDV problem is jointly convex in both and 7). The dual of this problem sheds the light on this estimator as a constrained ratio estimation where 7) plays the role of Lagrange multiplier that ensures proper normalization of the likelihood ratio. We also show how the witness function can be estimated as a neural network akin to Belghazi et al. (2018). We specify our estimator for MI in Section 4, and show how it compares to alternatives in the literature (Nguyen et al., 2008; Belghazi al., 2018: Poole et al., 2019). The experiments are presented in Section 5. On synthetic data, we validate our estimators by estimating MI on Gaussian variables and by regularizing GAN training as in Chen al. (2016). On real data,

---

Under review as a conference paper at ICLR 2020
Lower Bounds on KTAP.Q)
f-div Fep [1] - Roe! +
DV
D/-D
[ours]
sup ICN
sup sek
Er |s] log (Eq [<])
stp FC JeRACCR
(E- 1s1 - Eg [e-] - 4 +
f-MINE
*MINE Ex,, (f(e, Exury (c -] -
[ours]
sup Bp,, (f(r)))) v))
By [eltass) +1]
sup
Estimators of ((XY)
DV-MINE sup Bp.,, N[l log (Bpus, (c
o-MINE
sup Fpn. [f(z,)) Ey, (Ee, [eltnns)- m(tl)+1 For
Figure 1: Overview of the paper. Top row shows several KL divergence lower bounds for two probability distributions I and Q. By substituting = Pzy. Q PaPy, and defining neural network, obtain corresponding MI estimators, 1-DV and 1-MINE are the proposed bound and its derived estimator. Further details are provided in the text.
we explore our estimator in deep MI maximization for learning representation from unlabeled data. Figure shows an overview of all the bounds and related MI estimators discussed in this paper.
LOWER BOUNDS ON KL DIVERGENCES AND MUTUAL INFORMATION
Consider two probability distributions and Q, where absolutely continuous w.r.t. Q. Let p and q be their respective densities defined on X Rd. Their KL divergence is defined
p(x) ((x))
p(x) ((x)
(L(P. Q) = Ez~p log
p(xx) log
dx.
We interested in the MI between two random variables X, Y where X is defined on X' Rd,, and Y on y C Rd Let Pay be their joint densities and Pz. Py the marginals of X and Y respectively, The MI is defined follows:
T(X;Y) = (I.(Pay: PxPy),
(1)
which is the KI. divergence between the joint density and the product of marginals. Non-parametric estimation of MI from samples is important problem in science and machine learning. In what follows, we review variational lower bounds on KL to enable such estimation.
Variational Characterization of KL divergence. Let H be any function space mapping X' I. The first variational characterization of the KL divergence goes back to Donsker & Varadhan (1976):
(L Q) 2 D (P.
Sup(Eu-Fff log(E log(Ee-gee(t)) (()): f EH),
(2)
where the equality holds if and only if (iif) f log(p/ E H. We refer to this bound as the DV bound.
The second variational repressentatin was introduced Nguyen al. (2008); Nowozin et al. (2016)
and derived through convex duality to be finally stated as follows:
KL(PP. Q) > D (P. Q 1 (f En-ce(()) f HY,
with equality ii f log(p/)) E H. We call this bound the f-div bound (as in f-divergence).
From Eq. 2 and Eq. 3 we see that the variational bounds are attempting to estimate the log-likelihood ratio f'= log(p/g). and the tightness of the bound depends on the representation power of H.
In order to compare these two lower bounds, observe that log(t) S t - 1, > 0. Therefore log (Ei) < (x)) which means that for any function space H we have:
KL KL Q) 2 Dew Q D (P. Q).
from which we conclude that the DV bound is tighter than the f-div bound.
Now, given samples {x, 1......,, N,,; ~ P). fy..i=1....
~ Q) estimating the KL diver- gence can be done by computing the variational bound from Monte-Carlo simulation. Specifically,
2

---

Under review as a conference paper at ICLR 2020
for the DV bound we have the following estimator:
N
N Σ 1=1
(P,, (P, Q = sup
Σ
(ax) - log
edlo))
:s E H
(5)
N
N
Note that the Mutual Information Neural Estimator (MINE) (Belghazi et al., 2018) considered the above formulation with the hypothesis class HH being a neural network. For the f-div bound we have similarly the following estimator:
N 1 N
N (l
).
Fft (P. Q) = 1 sup
1 N
E
f E }
H
(6)
for which Nguyen et al. (2008) introduced and studied a convex estimator with H being an RKHS.
While DV bound is tighter than the f-div bound, in order to learn the function f using stochastic gradient optimization, f-div is better fit because the cost function is linear in the expectation, whereas in the DV bound we have a log non- non-linearity being applied to the expectation. This non- linearity introduces biases in the mini-batch estimation of the cost function noted in Belghazi et al. (2018). In the following, we show how to alleviate this problem and remove the non-linearity at the price of an additional auxiliary variable that will enable better optimization for the DV bound.
An n-trick for the DV Bound. We start with the following elementary Lemma, that gives variational characterization of the log, All proofs are given in the Appendix A.
Lemma 1. Let > 0, we have: log(z) = min, e "x + 1.
Using Lemma we can now linearize the log in the DV bound. Lemma 2 (17 D--Dsseer-aardhaan Varadhan). Let HP be any function space mapping X' to R: (L((P, Q) 2 D (P, Q) = = inf : f H E R} T(S,n) = e T (x) Eg-rs (xx)
(7)
(8)
We refer to this bound as DV-V bound.
Note that for 1-0 0 we recover the f-div bound. Using Lemma 2, we can now rewrite the estimator for the DV bound in Eq. as follows:
N
D (P. Q = inf (f,, m), where i(f,n) = = 1 f.n
Σ
eflva
f(x)) + 1, (9)
N
1=1
enabling unbiased stochastic gradient optimization of the function f. We note that similar variational
tricks of non-linearities have been devised for g(n) = / in Argyriou et al. (2008); Bach et al. (2011).
DVV BOUND, CONVEXITY, MMD RATIO ESTIMATES AND LAGRANGIANS
We established that the DV bound is tighter than the f-div and derived DDDV as an alternative to DV, enabling unbiased stochastic optimization. However, the role of 1) in improving the lower bound remains unclear. To highlight its role, we consider the n-DV bound from Eq. 7, and restrict the hypothesis function class to an RKHS, similarly to Nguyen et al. (2008). Working in the dual, we can compare n-DV estimator to the f-div estimator and further understand this hierarchy of lower bounds. See Figure 2 for an illustration.
For simplicity, and without loss of generality, we let RKHS be a finite dimensional feature map, i.e. H = [f) = (w, ((x)) Φb X - Rum E I=). (.m)..N Now for f E H. the loss given in Eq. 8 for 7 DV can be rewritten as follows:
T(.. I(w, m) "Eznge (w, (x)) (w, Ex-r (x)) +
(10)
Following Nguyen et al. (2008), we consider the following regularized loss: L(u, 17) L(wi, n)) ((w), and the corresponding sample- sample-based formulation L(ww, 7) = L(w, 77) + 2(w), where N(w)
is a convex regularizer, e.g., ?(w)= 2े w|l [ The D-DV primal problem can now be defined as:
n-DV P: min L(w, n) 1(w).
(11)
w.9)
3

---

Under review as a conference paper at ICLR 2020
f-div
1-DV
Primal
sup rew
EE
Ege! +
sup yEN
{ [f] - BB [er] - - 11
Dual in RKHS
min [Bg (r log(r)) + Ea[r] 1 AMMD"(rg,P))
min rsa, Ka(r]-1
{ Ir log(r)) + AMMDD(rq.p)
Figure 2: Comparison of dual formations of 7--V and f-diy in RKHS. Here is estimator for density ratio p/q and MMD (P, Q) [[Ez=>((<z) (y))l As can be seen, both duals have relative entropy term, but the f f-div bound does not impose the normalization constraint on the ratio, which biases the estimate, while D-DV uses as the Lagrangian multiplier to impose the constraint Bo [r] = and ensure density normalization.
In the following, we show that Lww, n) is jointly convex in (w, m7 and derive its dual, which will shed light on the nature of the likelihood ratio estimate as well as the role of 7/.
Convex Estimate in RKHS. In Lemma we first establish the convexity of the n--V loss function. Lemma 3. L and L are jointly convex in w and 7).
Lo(u, +(( -_ Our KL. estimate is therefore given as DØ (P. -i("", ,""), where (w" .11)) arg (i.)).)) Le,
7-DV Dual is Constrained Likelihood Ratio Estimation. Another angle for estimating the KL divergence through likelihood ratio estimation (Mohamed & Lakshminarayanan, 2016). Given ratio estimate f of p/g, the KI. divergence be computed as log(),,), which be easily estimated using samples from Q-
In Theorem 1, proven in Appendix A, we show that the dual problem (denoted D). corresponding to the primal minimization problem (denoted P) of n-DV, reduces to a constrained likelihood ratio estimation problem (denoted C).
Let (.) (.) be the Fenchel conjugate of (i.). The n-DV bound restricted to an RKHS, amounts the following regularized convex minimization: P (min L(w, + (l(W)), that has the following dual form:
Theorem 1.
(L r(wal)d 0 (A(r)), where ((r) = Sx 20 ((2)p(z)dz Sx (())
D min max / r(y) rly))og log (r(w)) gly)dy + (1- - r>0
Noticing that n plays the role of a Lagrangian multiplier, this is equivalent to the following likelihood ratio estimation problem:
C min / r(y) log (r(y)) dlydd + S." r>0
( @()))()))d J r(v)a(y)(())dd)
subject to: I r(v)d())dy=1 1 X
Therefore, we have P= C. Let (w* .n" an optimizer of Let be an optimizer of D, then the KL estimate is given by: D Q fx (1) log (v)( ( L(w", ,n)").
The regularizer ((w) = à w!| can now be given the following interpretation based on the results of Theorem 1. Recall the definition of the MMD (Gretton et al., 2012) between distributions using an RKHS: MMD (TP @)
expression in Theorem 1, ( E (v))l. ||- Replacing the Fenchel conjugate ! (.) by its
(-)(.) that DV is equivalent to the following dual (n-DV-D)
--DV-D min max / r(y) log (r(w)) (v))dy + 1n- ( ((w))())dy-11 2X
MMD (rg.p)
x
(12)
which can be written constrained ratio estimation problem (V-DV-C
/ r(y) log (r(u)) (())dy + MMD subject to: / r(v)g())dy=1. r20
1
-DDVCC min
2X
(13)
4

---

Under review as a conference paper at ICLR 2020
Hence, it is clear now that the r7DV optimization problem is equivalent the constrained likelihood ratio estimation problem r given in Eq. 13, where the ratio is estimated using the MMD distance in the RKHS between rq and p. It is also easy
that p/q is feasible point and for the universal feature p/g, therefore, for universal kernel, p/q is optimal and we recover p/q. When comparing 1--V-D D from Eq. 12 and n--VV from Eq. 13,
map MMD (rg, p) = 0 iif
the KI. divergence for see that 7-1 plays the role of a Lagrangian that ensures that rq is indeed a normalized distribution. In practice, we solve the primal problem (P), while the dual problem (D) and its equivalent constrained form (C) explain why this formulation estimates the KL divergence and the role of as a Lagrangian multiplier that enforces a normalization constraint. Let r" be the solution, then we have:
D Dow (P.
r" r"(y) log (r" (v) du)dy = -L("",1 :n")-
(14)
For comparison, the f-div bound, restricted to an RKHS, is equivalent to the following ratio esti- mation (follows from the proof of Theorem by eliminating max on and setting 1) = 0), and is consistent with the results of Nguyen et al. (2008) (see Eq. 51 therein):
f-div min / r(y) log (r(u)) dy)dy + r(y)g())dy + 1 MMD (rq.p). (15) r>0
1
2X
Let r" be the optimum, then the KL divergence can be estimated as follows Nguyen et al. (2008):
(P. = / r'(() (1) log (r"())) + (1-/ (w)aa()))
(16)
Comparing DV (P, Q) and Dtd (P, Q) we see that they both have a relative entropy term but the f-div bound does not impose the normalization constraint on the ratio, which biases the estimate. We end with the following remark: In Theorem I, if we replace the loss by its empirical counterpart i, from Eq. 9, the equivalent Dual takes the following form:
N
N
N
min T:D
Σ
rs log (r:) + +
4 ((r))
Σ rr0())) subject to: ==1
N r3
Σ 1=1
=
N
N
1=1
and the KL estimate is given by: (P. EN
r log (rt) = i(w" (n")))).
From RKHS to Neural Estimation. One shortcoming of the RKHS approach is that the method depends on the choice of feature map 4. We propose learn as deep neural network as in MINE (Belghazi et al., 2018 (). Given samples x; from IP, Di from Q the KL estimation problem becomes:
N
NN (P.
min min e -1
(MO) - Σ (w, ((:))) + +7
(17)
+ 7,
N
1=1
which can solved using BCD on (w, 17. 4). Note that if ((-) is known and fixed, then optimization
problem in Eq. 17 becomes convex in and w. We refer to this bound as --DDVcoovvx.
What Do Neural Estimators of KL or MI Learn? We conclude with the following observations:
1. Ratio estimation via Feature Matching. KL divergence estimation with variational bounds boils down ratio estimation using form of MMD) matching.
2. Choice of Teature/Archiiecture. The choice of the feature space or the architecture of the neural network introduces bias in the ratio estimation; also observed in Tschannen al. (2019).
3. Ratio Normalization. n-DV bound introduces a normalization constraint on the ratio, that ensures a better consistent estimate.
4. Impact of Regularizers. The choice of the regularizer dictates the choice of the metric. For instance, Ozair et al. (2019) used gradient penalties as a regularizer, and this corresponds to a ratio matching in the Sobolev discrepancy sense. Please see Appendix A. for more details.
4 DV-V MUTUAL INFORMATION ESTIMATION
We now specify the KL estimators given in Section 3 for the MI estimation problem. Given function space H defined on X' ),
T(XY)) 2 bV (X;Y) = sup E y) - log (Ep. Eng ellww) 2 (X:)), (18)
JEM
5

---

Under review as a conference paper at ICLR 2020
where (X: fp..f(a..) - Ep, Bp, ef
Equivalently, with the 17-trick we have:
bD (X:Y)
inf sem
e "B (2,,) - f f(r, + 17-1 1.
(19)
Now, given iid samples from marginals x; Ps and yj ~ Py: 1... N, and samples from the joint (zi, 34) ~ Pay, i = ===:... 1...N, N,, we can estimate the MI as follows:
N f(x, + - 1, (20) Σ N
ID (x, Y) = Dpv PaPyy)
inf e SEM
1 N
Σ
1
1=1
when H is an RKHS, 1) can be seen as a Lagrangian ensuring that the ratio estimation r(z; y) of is normalized when integrated P>Py, i.e., 1) Lagrangian associated with the constraint PaPy
xxxy Pa (z) ( drdy 1. In Table we review other variational bounds for MI based on f-div, DV, and n-DV bounds (a-MINE from Poole et al. (2019) is discussed next).
MI Estimator
Loss to minimize f.
Constraints
(bound)
DV-MINE
log N EN (ltind)
N Eim (a),
f is DNN
(DV)
Unbiased
1 IN (8))
f is DNN, m running avg. of LDN (((L)))))
N Em (Ir,
DV-MINE
F-MINE
f ii in RKHS (convex)
1/2 Emm (arr,8i)
(f-div)
or f is DNN
G-MINE
Σm
(eeiii a(Ri)
+ +log(a(ū))) Em f(0,
is DNN, a>> f is DNN
(DV)
InfoNCE
log 1 Σe= (((18)))) - -f(I..
is DNN
MINE
EN
efl EN f(a, v.) + EN (w, (as, vi) + -
is RKHS
(oursa--DV)
or is DNN; 17 E R
MIME convex
N (w))
2(-) fixed
u Rdm(()) T
Table 1: Given iid samples from marginals ~r and ÿú Py and samples from the joint (x,, vı) Pay. we list some MI estimators, corresponding variational bounds, associated losses, and constraints on the function space. MI estimators include biased and unbiased DV-MINE from DV (Belghazi al., 2018), f-MINE from f-div (Nguyen et al., 2008; Nowozin al., 2016), InfoNCE (van den Oord al., 2018a), a-MINE from (Poole et al., 2019) and yy MINE, P-MINE-ccove convex from (oDrs)
Improved Donsker- Varadhan for Mutual Information and a-DV Bound Discussion. In the following Lemma we show that for the particular case of MI estimation, the DV bound can be made tighter:
Lemma 4. For any function g that maps X' x y to R+ we have:
KL(Pz.g, Ep.. log(Ep, g(x, u)) 2 Epr.y log(g(x,)) v))
Therefore: T(X;Y) = se Ep,, f(r y) - Ep, log(Ep, cf where ((X x y) is the entire space of functions defined on x y. For any function space H map X y to R we have:
1(x;)) 3 (X:Y) = sup Up...f(s 3) Ep, log(LE, (w..))
SEN
Using Jensen's 's inequality,
easily derive the following hierarchy of lower bounds:
(X; T(X: Y) 2 (x;)( 2 (X; Y) 2 //-div (X;
(21)
In Theorem 2, apply the 1--trick of Lemma to T=bb((x; (X; Y), where replace each
log eEppe ((x.y))) using the variational form of log which leads to 77y). The interchangeability between minimization and expectation and going from scalar 77 for each y to function r7()) needs some care and we prove it the Appendix B. appealing to the interchangeability principle (Lemma in Dai al. (2016)).

---

Under review as a conference paper at ICLR 2020
Theorem 2 (17-trick for Improved DV Bound). Let H be a space of ftactiioo defined functions
X x y we have: T(X;Y) 2 - -inf (EX(.NE(()) Pp..P eI(2..)-1(3) + -.p,, v(y) P. (S (x, u)), where ()) is the entire space of functions defined on V. Let Hn be function space subset of (V)) we obtain:
Y(X;Y) 2 -x (X:Y)
f y) - Ep.., e (o ml)) Ep, + 1. (22)
sup SEX mn..e.neeee
Hence, we recover the lower bound given in Poole al. (2019) (given in Table 1), that was derived through other variational principles. In the following, we make a few remarks about Dv Y):
1. The 1e (X; Y) bound, as given in Poole et al. (2019), can be also obtained by setting nyy) log(a(y)), a(y) > O. While this seems equivalent, the restriction of a to R+ (17 is in I)) changes the landscape of the optimization problem and the interchangeabiiity principle does not apply in straightforward way.
Poole et al. (2019) refers to a(y) as a "critic" (as in the reinforcement learning literature). As shown in our analysis in Section 3, when we restrict H to an RKHS, nyy plays role of Lagrangian multiplier or a dual potential in a ratio estimation problem. In this case is enforcing for each y. instead of imposing single constraint Jx Ju r(,, ())(())(())ddd))) = in the T Dv (X; Y) bound.
multiple constraints fx (xr((,)))(e v))(x)dx
3. No Free Lunch: while [x; Y) is tighter than our (X; bound, it comes at the price of estimating function 7y)) together with the witness function f. This triggers higher computational cost and larger estimation errors.
4. We show in Appendix C how The (xbv((;Y) can be extended to conditional MI estimation.
5. Generality: 17-V bound is applicable for general probability distributions P and Q: (L(P,, Q) 2 DD (P, Q), while T (o,v(X; (X; Y) only works for Pry and Q PP=Py-
Finally, in Algorithm we outline the steps of 7-MINE for MI estimation.
Algorithm 1 1- 17-MINE (Stochastic BCD)
Inputs: X, Y dataset X E Nxd, Y IR Ig,,,, such that (x; X 91 Yi..) Pay Hyperparameters: any ag (learning rates), ne (number of critic updates) Initialize 17, 0 parameter of the neural network fo
for i = 1......... ..... Maxiter do for j 1....e.. Ree
Fetch minibatch of size N (Ii, Vi) Pay
Fetch a minibatch size N (Ii, Tii ~ PaPy {ūe obtained by permuting rows Y}
Evaluate L(fo, = e 1
N E (fo( Em fo(2s, vi) + n -
Stochastic Gradient step on
0 8 ff. (We use ADAM)
end for Update vy:
ai(fa.n)
+ On On
end for
Output: fe. n, jot (x, + fo( - - EN efu(zz)0)) - +7
EXPERIMENTS
In this Section, conduct set of experiments using synthetic and real data number of applications to compare the proposed MI-M MI estimator to the existing baselines.
MI estimation. We compared different MI estimators three synthetically generated Gaussian datasets [5K training and 1K testing samples]. Each evaluated MI estimator was run 10 times and its average performance (together with the standard deviation) is shown in Fig. 3.a. As can be seen,
7

---

Under review as a conference paper at ICLR 2020
2-d Goussian
10-d Gaussian
20-d Gaussian
0.5
2.0
0.421
1.688
3.546
0.4
1.5
0.3
0.1
0.5
1-
0.0
0.0
2000
Iteration
X
Uncons GAN
Corel GAM DV- MING
LOKO
Cand GON M
Figure 3: Performance of different MI estimators on synthetic Gaussian datasets. Top: MI estimation. Data was sampled from 2-, 10- and 20-dimensional Gaussian distributions with randomly generated means and covariance matrices. As we increase the data complexity, the difference between estimators decreases, although we observed that MIMINE (or convex extension) on average performed better than the baseline methods, converging to the true MI [red line] faster. Bottom: MI for GAN regularization. Top row: unconditional GAN baseline fails at capturing all 25 modes in the Gaussian mixture. Middle row: MI-regularized conditional GAN using DV-MINE (Belghazi et al., 201 8) converges after 140K steps of the generator. We found this estimator to be sensitive to the hyper-pareee parameters and unstable during training. Bottom row: MI-regularized conditional GAN using -MINE the model converges in 50K steps of the generator.
MI estimation in high dimensions is a challenging task, where the estimation error for all methods increases as the data dimensionality grows [red line shows true value for MIJ. Nevertheless, the proposed 1I-MNNE is able to achieve on average more accurate results compared to the existing baselines. We also see that the convex formulation of 11-MIEE has overall a better performance and fast convergence rate. This estimator has a linear witness function that is defined as f'(-) = (w, ((-)) using pre-trained fixed feature map ( from the regular MINE, In the experiments that follow compare the proposed estimator MINE to DV-MINE, as a main baseline approach.
MI-regularized GAN. We investigate GAN training improvements with MI, and especially dimin- ishing mode collapse, as addressed in Belghazi et al. (2018) in Section 5.1. Belghazi et al. (2018) uses a 25-Gaussian dataset to show improvements on GAN Clustering by using MI objective for regularization. As in InfoGAN (Chen et al., 2016), the conditional generator G' is supplied with random codes c along with noise 2, we maximize the mutual information between I I(G(z, c). e) using 7-MNNE estimators. In Fig. 3.b, we repeat this task and establish that MI-INEE can recover all the modes within fewer steps than DV-MINE (50K vs. 140K) and with a stable training.
Self-supervised: Deep Info (Hjelm et al., 2018). In unsupervised or self-supervised learning, the objective is to build a model without relying on labeled data but using an auxiliary task to learn informative features that can be useful for various downstream tasks. In this experiment we evaluate the effectiveness of the proposed MINE estimator for unsupervised feature learning using the recently proposed Deep InfoMax method from Hjelm et al. (2018). For feature representation used an encoder similar to DCGAN (Radford et al., 2015), shown in Fig. 4.a and evaluated results on CIFAR10 10 and STL.10 datasets (STL.1 images were scaled down to match CIFAR 10 resolution).
8

---

Under review as a conference paper at ICLR 2020
53.1%
48.15
74.5%
44.5%
57.1%;
77.5%
MINE NN
label
E
MI loss
E
MINE NN
tained
3x
64 16
12 2y6 3024
64
(a)
(c)
Figure 4: (a) Encoder architecture and classification accuracy (top1) CIFAR- 10 dataset for different pre- trained encoders. Each number represents test accuracy of the system trained by maximizing MI between features from layers pointed by the corresponding arrows. Interestingly, the highest accuracy was obtained by pre-training encoder composed of just the first two convolutional layers (see (b) and (c) for details of this process). (b) Model pre-training by maximizing MI between features from different layers [additionally transformed by neural net, aimed at constructing witness function f(-)). (c) After pre- training. we fix encoder and attach trainable linear layer to perform traditional supervised training on the same or different datasets.
The encoder is trained by maximizing MI I [a', y') between features from one of the shal- low layers I (x' = (z)) and a deeper layer (y' Ex (v)). We examined different layer combinations and found that the encoder com- posed of only the first two convolutional layers the downstream
Table 2: Classification accuracy (topl) results CI- FARIO (C10) and STL10 (S10) for unsupervised pre- training task with DV-MINE and 17-MNE using encoder in Fig. 4.b. For reference we also list results for su- pervised (CF) training using full encoder in Fig. 4.a. ry M-ME--baaee -baeed pre-training achieves overall better re- sults when evaluated on same or different data, even outperforming the supervised model on S10.
give the best performance tasks. As shown in Fig. 4.b the encoder features passed through additional trainable neural network layers, whose job is to build a classi- fier fx', y²), discriminating cases when x' are y' are coming from the same image and cases when x' are y' are unrelated. Finally, we at- tach a linear layer to the pre- pre-trained and now fixed encoder (see Fig. 4.c) to perform super- vised training. In Tab. 2 we present the results for two MI estimators: M-MINE and DV-MINE, whose loss functions are listed in Tab. 1. As can be seen, 17- --N--badded pre-training performs competitively with DV-MINE, achieving overall better results on both datasets, showing practical benefits of the proposed approach.
Test
C10
S10
Train
DV
1)
Sup.
DV
1
Sup.
C1O
77.5
74.8
84.2
55.1
56.4
S10
67.5
68.3
61.8
63.3
61.3
Self-Supervised: Jigsaws with MI. The self-supervision Jigsaw pretext task (Noroozi & Favaro, 2016; Kolesnikov et al., 2019) aims at solving an image jigsaw puzzle by predicting the scrambling permutation From image X with = (x1 2 3x3 jigsaw patches, and permutation y= n; [1,9] 9). scrambled patches Tw, = =
(0) generate the puzzle. Each patch is passed through encoder F., which needs to learn meaningful representations such that classifier CJ can solve the puzzle by predicting the scrambling order n; (Noroozi & Favaro, 2016) (see Fig. 5.a). While this standard formulation relies on a CE-based classification of the permutation, we propose here to use MI Jigsaw, where we train the encoder E' to maximize I (EE(x,)); m.)) by using MI estimators DV- and MIIN,, as seen in Fig. 5.b. A patch preprocessing similar Kolesnikov al. (2019) avoids shortcuts based on color aberration, patch edge matching, etc. (Noroozi & Favaro, 2016); for details of our implementation, see Appendix D. All our models are built on a 10% subset of ImageNet (128K train., 5K val., IK classes) as proposed by Kolesnikov et al. (2019). This is larger set than Tiny ImageNet (200 classes) used many publications. F. a ResNet50 for all our experiments. In our ImageNet target classification task, E from CE and MI Jigsaw trainings are frozen (at 200 epochs) and followed by linear classifier C' (Fig. 5.c); an adequate setup for comparing encoders as argued by Kolesnikov et al. (2019).
All models are compared on target task in Tab. 3. Best accuracy numbers are reported for C's trained for exactly 200 epochs. For all results, E is trained from Jigsaw task (CE or MI) and frozen, with only trained as in Kolesnikov et al. (2019). DV- and 17 MIMNE share the same f architecture. MINN gives better accuracy performance compared DV-MINE. CE model trained from a Jigsaw-

---

Under review as a conference paper at ICLR 2020
supervised encoder E provides upper-bound for supervised performance for E from the Jigsaw task. Despite CE being better than 77MINE and DV-MINE, 17-INE does a respectable job at learning useful representation space for the target task, better than DV-MINE, More details about the Jigsaw experiments setup can be found in Appendix D.
Table 3: ImageNet (10% subset) classification ac- curacy results (in %). DV-MINE and 17MINE are using fixed Encoder from MI training. CE is for using CE Jigsaw Encoder. Numbers reported are average and standard deviations over models from different initialization seeds.
Encocler
Atr Classifier
Loss (a)
Payser permutation Idbel
Encoder
f(E(ae).
MI Loss
(b)
Jigio permutation label
DV-MINE
1-MINE
CE
Encoder
Rixt
Classifier
(c)
top1
8.5 + 1.3
11.0 1.1
12.9 + 0.3
top5
20.0 2.5
24.1 2:2
28.1 + 0.6
Figure 5: (a) Jigsaw CE training. (b) Jigsaw MI train-
top10
27.8
3.1
32.7 =
37.7 0.6
ing. (c) ImageNet Classification CE training.
CONCLUSION
In this paper we introduced a new lower bound on the KL divergence and showed how it can be used to improve the estimation of mutual information using neural networks. Theoretically the dual of our n-DV formulation reduces to a constrained likelihood ratio estimation. In practice, the stability of 1-MINE is due its unbiased gradient. We tested estimator 17-MIN synthetic data and applied it on various real-world tasks where MI can be used as regularizer, or as an objective in self-supervised learning. We used 1/-MNNE in unsupervised learning of representations through MI maximization and by solving Jigsaw puzzles. We leave for a future work the analysis and various possible applications of conditional mutual information neural estimators.
proved that
REFERENCES
Alexander A. Alemi, lan Fischer, Joshua V. Dillon, and Kevin Murphy, Deep variational information bottleneck, 2016.
Alexander A. Alemi, Ben Poole, Ian Fischer, Joshua V. Dillon, Rif A. Saurous, and Kevin Murphy, Fixing broken elbo, 2017.
Andreas Argyriou, Theodoros Evgeniou, and Massimiliano Pontil. Convex multi-task feature learning. Mach. Learn., 2008
Francis Bach, Rodolphe Jenatton, and Julien Mairal. Optinization with Sparsity-Inducing Penalties (Foundations and Trends(R) R) in Machine Learning). Now Publishers Inc., Hanover, MA, USA, 2011. ISBN 160198510X, 9781601985101.
David Barber and Felix V. Agakov. The im algorithm: A variational approach to information maximization. In NIPS, 2003.
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeswar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and R Devon Hjelm. Mine: Mutual information neural estimation, 2018.
Anthony J. Bell and Terrence J. Sejnowski. An information-maximizziio maximization approach to blind separation and blind deconvolution. Neural Comput., 1995.
David M. Blei, Alp Kucukelbir, and Jon D. McAuliffe. Variational inference: A review for statisticians. Journal of the American Statistical Association, abs/ hbbs/1601.00060 2017.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In NIPS, 2016.
10

---

Under review as a conference paper at ICLR 2020
Bo Dai, Niao He, Yunpeng Pan, Byron Boots, and Le Song. Learning from conditional distributions via dual kernel embeddings. CoRR, abs/1607.04579, 2016.
Carl Doersch, Abhinav Gupta, and Alexei A. Efros. Unsupervised visual representation learning by context prediction. 2015 IEEE International Conference on Computer Vision (ICCV), Dec 2015. doi: 10. 2011000/ccv.201..166 URL http://tt//// dx doi. org/10.0............. 2011999CCC)
M. D. Donsker and Srinivasa Varadhan. Asymptotic evaluation of certain markov process expectations for large time, Communications on Pure and Applied Mathematics, 1976.
I. Ekeland and T. Turnbull. infiniteddimension Optimization and Convexity. The University of Chicago Press, 1983.
Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. JMLR, 2012.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Improved training of wasserstein gans. arXiv:1704.00028, 2017.
Zaid Harchaoui, Francis Bach, and Eric Moulines. Testing for homogeneity with kernel fisher discriminant analysis. In NIPS, 2008.
R Devon Hjelm, Alex Fedorov, Samuel Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In ICLR, 2018.
Karan Grewal, Adam Trischler, and
Devon Hjelm, Alex Fedorov, Samuel Lavoie Marohildar,, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. International Conference on Learning Representations, 2019. URL https:
sropenreview.noe www..nee/oooummmeeeeeeeoo
Forum?id=Bke cCKK.
Weihua Hu, Takeru Miyato, Seiya Tokui, Eicchi Matsumoto, and Masashi Sugiyama. Learning discrete representations via information maximizing self-augmented training. In Proceedings of the 34th International Conference on Machine Learning. 2017.
Artemy Kolchinsky, Brendan D Tracey, and David Wolpert. Nonlinear information bottleneck. arXiv preprint 17X5v:1705.02436 2017.
Alexander Kolesnikov, Xiaohua Zhai, and Lucas Beyer. Revisiting self-supervised visual representa- tion learning, 2019.
Alexander Kraskov, Harald Stögbauer, and Peter Grassberger. Estimating mutual information. Physical review. E, Statistical, nonlinear, and soft matter physics, 2004.
Andreas Krause, Pietro Perona, and Ryan G. Gomes. Discriminative clustering by regularized information maximization. In Advances in Neural Information Processing Systems 23. 2010.
Shakir Mohamed and Balaji Lakshminarayanan.
Learning in implicit generative models.
arXiv:1610.088 03483,03333.1 2016.
Youssef Mroueh and Tom Sereu. Fisher gan. arXiv:1705.06675 NIPS, 2017.
Youssef Mroueh, Chun-Liang Li, Tom Sercu, Anant Raj, and Yu Cheng. Sobolev gan. ICLR, 2018.
Youssef Mroueh, Tom Sercu, and Anant Raj. Sobolev descent. In AISTATS, 2019.
XuanLong Nguyen, Martin J Wainwright, and Michael Jordan. Estimating divergence functionals and the likelihood ratio by penalized convex risk minimization. NIPS. 2008.
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. Lecture Notes in Computer Science, pp. 69,84, 2016.
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-gan: Training generative neural samplers using variational divergence minimization. In NIPS, 2016.
11

---

Under review as a conference paper at ICLR 2020
Sherjil Ozair, Corey Lynch, Yoshua Bengio, Aaron van den Oord, Sengey Levine, and Pierre Sermanet. Wasserstein dependency measure for representation learning. CoRR, 2019.
Ben Poole, Alexander A Alemi, Jascha Sohl-Dickstein, and Anelia Angelova. Improved generator objectives for gans, arXiv preprint arXiv:1612.02788) 2016.
Ben Poole, Sherjil Ozair, Aaron van den Oord, Alexander A. Alemi, and George Tucker. On variational bounds of mutual information, 2019.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv:1571.06434, 2015.
Bharath K. Sriperumbudur, Kenji Fukumizu, Arthur Gretton, Bernhard Scholkopf, and Gert R. G.
Lanckriet. On integral probability metries, (-divergences and binary classification. 2009.
Naftali Tishby, Fernando Pereira, and William Bialek. The information bottleneck method. pp. 368-377, 377, 1999.
Michael Tschannen, Josip Djolonga, Paul K. Rubenstein, Sylvain Gelly, and Mario Lucic. On mutual information maximization for representation learning, 2019.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. ArXiv, 2018a.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, 2018b.
12

---

Under review as a conference paper at ICLR 2020
Supplementary Material for Improved Mutual Information Estimation
The supplementary material is organized as follows:
In Appendix A we layout proofs of Lemmas 1, 2 and 3. Those lemmas provide the 1y-DV formulation. We also give in this Appendix the proof of the main Theorem This theorem gives the dual formulation of this variational bound in RKHS with Tikhonov Regularization and sheds the light on the ratio estimation interpretation of ny-DV.
We give in Appendix A.I the impact of other regularizers on the estimation of KL divergence in RKHS.
In Appendix B, we show that further improved bounds can be obtained for the particular case of mutual information leading to an improved Donsker- Varkah--a We show that by using the interchangeability principle, we can obtain an equivalent functional r-DVV bound that is related to a-MINE introduced in Poole et al. (2016).
In Appendix C, we show how those improved variational bounds can be extended to conditional mutual information.
In Appendix D, we give more details on the experimental setup of self-supervised learning with the jigsaw task.
A TECHNICAL PROOFS
Proof of Lemma:l. Let g()) = e e + 1. We have g((n) = - + and g" = = > 0. Hence g is convex and admits unique minimum. First order optimality gives us: g((n")
- 11 x + 1 = 0, hence e-n" x = and 1/" log(x)) Finally, we get
g(nn) e log(()) x log(x) - = c 1) x + log(z) - 1 log(x) - 1 = log(x))
Proof of Lemma 2. By Donsker- Donsker-Varadhan Varadhan representation have:
Dov (P, Q sup Fx) log (Range((x)
feat
inf log (Benge((t) - Bxar((x)
FEA
= inf (inf e "E (w) + 11 1] Exwzf(x) fear
inf {e EEngee(() - Eeszf(a) 1}.
Jew
Proof of Lemma 3. L is sum of linear function in w. 17 and ((w), that are both convex. We need prove that Jo(w 1) e (w.,b)) is jointly convex, since L can be written a sum of and other convex functions:
V VwJl(w, = - (w.bl) b.
v Volow 1)) = - (1000,)) 8 b PSD (Positive Semi-Definite)
2 fo e (m b.
02 fo
= (
dwen
Onp
The Hessian of fo(w, 11), H fo. has the following form:
8 Brassdru
Me (ul & -e (t
H fo m) =
L
8
" e ee
Iyow
Let us prove that for all (w, (w: (w',n')) Ho () (w',n/) n7/) 2 0, (. ,7'). We have:
(w',n/) H Holw,, (w(,))))
e ee ((w', . 2 - (w', b) + n/ (w,b)
e (e (((w', ;b)) 2n' (w', b) + 12)
(w. ((wl n²) > 0.
13

---

Under review as a conference paper at ICLR 2020
Hence, fo is jointly convex in 7 and w, and so is L. Now adding the perturbation - to the loss, get
"
n'nn') H f6w, n)(w n/) = (m ((w', b) - r/²)² + ee 0, for finite 77 and hence we have strict convexity.
Proof of Theorem 1. Let's consider:
((x) (w, (())) and (p() e'.
Recall that the Fenchel conjugate of f (*(p) sup, (p,x) - f(x)). follows that (²((1)
t tlog(t) - ift > 0, 0 for t O, and oo otherwise.
mine e "Ey-0 (ww, ((y))) Exwe (w, f(x)) -
min 4 J 4 (1w, p(v))(())))))))))))))))) d())dy w. /w.// ((z)p(x)dr + - 1(w))
w.y
Sfx (=(1))
w, (()))(o)de)) - s.t. ((y))
min e w,3,3
Introducing Lagrangian's have
min maxe T J @ (z(y)) (y)dy w,ge
w. / (<))p(=)dz
+1 (7-1+(((w) / a(y)((u p(v)) z(y))
e J ( (z(n)) (v)dy
+1 / aly)((w, p(v)) =(y))dy
max min mias
w. ((z)p(o)dr
using strong duality (convex function and lower somi-continuous) to swap min, max (Ekeland & Turnbull, 1983)
(e(((elw)) a(v)=(vi)dy)
max min
max
9
w. x (())p(x)dr + / a(y))(()dy)) =((w) 17 n+11 1)
max
( g(0)e max ((z) a(y)
o (((1))(o)dd++ / a(v)) +
max min
((0)e
( ((0)e 0p0( a(v)
Ω (/, { ((=))(Z)d / a(y))(y)dy) +
max min a<00 "
x
((y))-1)
/x
( (ly)e-mpp
a(y)
-5 ( (())p(=)dr / (ly) +n
max min a>0 7
((()e-1))
x
x
( q(y)e
a(v)
( (G)))()dd aay))()))l))
min max as0
(()e--))
+ Ω2
Hence, we have for a > 0, and finite ry:
/ ((y)e-"'> 7
( dy = [ Lowee--
aay))
o(y)
log( aly)
aly)
dp
auye-- ((y)e-1)
((y)e-1)
[ g(v)e
a(v)
log( a(v)
aly)
dy
q(y)e-1
((y)e-1)
a(y)e-1
[ (aly) log
aly)
(()e--1) - a()))dy
[ aly) log
(a(y) g(y) dy + (1 - 1) / a(y))dd.
Define
r(y) = a(y)
> 0
g(v)
14

---

Under review as a conference paper at ICLR 2020
then have
(A) / a(v))="pp'
(
(_a(s)
a(y) (()))=
dy / r(v) log (rll)) + 1) J, r(u)a(u)dy
and
((v)) = =0- /
(e (()oooo)))a ((v)a()))())d))
(B) 12"
Hence, by replacing (A) and (B) by their expressions, get:
min L(w, n) w..)
(Aolv)e "0" - 22 (/, ((z))(o)dd / a(v))(v)dy) -
-imin max as0 "
(od (/x r(o)a(o)dy-11
r(y) log (r(v)) (n-1) min max r()30 "
(/, (())p(x)d ((0)
Now consider again the D-DV Bound:
min L(w, 1))
/ r(v) log (((l)))()dd + (n (n- (1-11111/ r(v)a())dy-
min max r20 7
+ 5 (fz (G)))ooo) - - r(u))(w)(())dd)
min / r(v) ((())) + 12 (/ (()))(z)dd / ((v)a()))())d))
s.t. S ((0)a())dy=1 =1
We see that DV-V amounts to likelihood ratio estimation, where match with the dual norm S2" the difference of kernel mean embedding of p. and rq. And 7) plays a role of Lagrange multiplier that enforces the constraint that rg is normalized distribution. In particular, consider Q(u)) こ [|uu| 12 so that Ω (1) zt ||/2 and hence have the following equivalent problem of the regularized DD bound:
min / r(y) log (r(y)) + r>0
1 ((z)p(x)dr J ((w))()))())dd 2A
1/
MMD matching und
rly)aly)dy
Normalization Constraint
Another way coming informally to the regularized DD-DV objective is start from expression of KL using r:
/ (()) (og(((i))))) AMDD <
/ (()))(x)= =1
We see that r in the dual function in estimating the ratio, 77 is the Lagrangian of the inequality constraint, and the MMD is added the cost penality.
15

---

Under review as a conference paper at ICLR 2020
A.1 IMPACT OF REGULARIZERS ON KL ESTIMATION
As discussed before, the choice of the regularizer Ω translates to a particular type of metric used in estimating the likelihood ratio: O(.) = 11.112 translates in a ratio estimation in the MMD sense. Here, we consider other choices of regularization used in Fisher discriminant analysis (Harchaoui al., 2008; Mroueh & Sercu, 2017), specifically:
-isherr min L(f.n) + E
P40 fx)),
(23)
or gradient penalty regularizer (Gulrajani al., 2017; Mroueh al., 2018; 2019):
D-DD-Sobolev min L(S.)) + E 2~ [v](2))]²
(24)
[.
It is easy to see that when we restrict f to H. Diihheisher has the same objective as in Eq. 11 with ((w) = (w, (C(P, Q))w), where C(PI, Q) Beg ((x) @ ((x) is a covariance matrix. Similarly, LLDD--Sobller corresponds to Eq. 11 with (?(w) - 2 (w, (D(P,, 1)),,, where Q)
d -= a, (x)) (z)()) is Gramian of derivatives. The convex conjugate Q (w) is
therefore 2x (wr, (Ct ()) and 2I (ur, (DT(P, Q))w), for D--V--Fisher and 1--DV-obblee
respectively (where i denotes the pscudo-inverse).
Let = - (y)( Recall from Harchaoui et al. (2008) that the Fisher discrepancy is (P, () C(P, op,e). and from Mroueh et al. (2018; 2019) the Sobolev discrepancy (T, (6r.o: D(P, (rr.)) Now replacing Ω by its expression in Theorem we see that the choice of the regularizer dictates the discrepancy used for the ratio estimation:
min / r(y) log (r(y)) dyyddy +
1 F3 p). subject to: S r(v)g(9)dy-1) = 2X
7--D DV---iisher: r>0
D-DV-Sobolev min f r(y) log (())) + S p). subject to: / 1(0)9(8)dy=11 rS0
2X
B FURTHER BOUNDING MUTUAL INFORMATION: a- a-MINE FORMULATION FROM INTERCHANGEABILITY PRINCIPLE
We start by the classical result of Donsker and Varadhan:
Lemma (Donsker- Varadhan). For any Distribution IP and any distribution Q and any non-negative function for which Egpp log g(x)) is finite, we have:
(L( Q log(Ez.Qg(a) 2 EZ=P log(g(x)).
This gives us representation of KL divergence:
KL(P,
=sup ExpP log(g(x)) log(Bn-gg0=)) sup Ezwrs(x) log (()))
9>0
The following Lemma shows that a finer application of the Donsker-Varadhal Lemma gives an improved variational representation of MI:
Lemma 6. Let > 0
KL(pa PxPy) + Ep. log(E,,g(z, 2 Ep.., log(g(x,
Hence, we gel
I(X:Y) sup Ep., log(g(æ, 8)) Ep, log 9>0
Ep f(, v) - Ep. Log(LE,, (2.11))
sup (e
E E f v) - Ep, log(E,. (x,8))
sup (e
where ((X V) is the entire space of functions defined on X y
16

---

Under review as a conference paper at ICLR 2020
Proof.
KL(pz.y PmPy) + Ep, lo, y)) = Ep., log( Pay ) + Ep, log(Ep,
PaPy
Bpyls log( Pule) + log(Ep,
= E..
Py
= Ep. (KL(Polas Py) p + log Log(Epp (l))
Now applying Donsker-Varadhant Varadhan Lemma, we have:
(L(Pyla: Py) + log log(En, 8)) 2 Bpule log(g(æ,,)) v)).
We conclude therefore that KL(paay PxPy) Ep. log(En, I(z, ((z, v))
2 Ep, (Epgie log(g(z, u)))
Ep.. (o(()(,,)))
B. 7-TRICK AND INTERCIANGEABILITY PRINCIPLE FOR IMPROVED DV
We just showed that improved Donsker- Varadhan representation of Mutual Information be be derived as follows:
T(x;))= sup (E
p P, log(R (=(.))
We show in this section that, by using interchangeability principle, we can replace the "log" non linearity preceding the expectation with an "functional" --trickk.
We start first by stating the interchangeability principle between expectation and minimum operations: Lemma 7 (Interchangeability Principle Lemma 1 in (Dai al., 2016)). Let be random variable on and assume for any & E E. the function g). IR + (-o0, +00) is a proper and convex function. Then
lower semi-continuous convex
Ee min ((, WEX
min Beg(u(E),
(E)
Applying the interchangeability principle and the variational representation of log we obtain the following Theorem:
Theorem (Theorem restated: n-trick for improved DV Bound), Let H be a space of bounded functions defined on X x Y, we have:
I(X;)) 2 Te
inf Ep..n, (n.y) - nlu) + Ep,,(1) - Epp., (sx v)),
where (Y)) is the entire space of functions defined on y. Let He be a function space subset of (Y)), obtain:
1(x;Y) 1xt (X;()
Ep v) Ep. n, ((..)) (x..0)- w(s) -EE, m74) EE,,(yy 1.
sup Me.
Proof.
We know from Lemma that for any function space H
1(x;Y) 2 inf Ep, log(E,, (( - Epe.y
(25)
few
Applying the -trcck for each log in the expectation:
Ep, log (lp f(x.w)) Ey, min "aineeEE,. neR
"Ep, ((.,) +1
(26)
We assume here that the function space f is bounded. Before applying the interchangeabiilty Lemma let us verify that all assumptions hold for
17

---

Under review as a conference paper at ICLR 2020
g(n. v) = EB +1 1.
Convex. ((, is convex. Considering derivatives wrt to m/: g1 v) - Bpp (eflaw))) + and ("(n.v) "p, (e/(z.))) > 0.
Proper Function, g(7, -) is proper functionn since f is bounded (-M & [(x, S M):
g(0, y) = Ep, (x.2)) - 1 MM -1<0.
and
((),)) > Vn,
since
g(7,8) 2 g(n"", = log(Epp Log(Bpp_e(uuug e((ww) > -M, Vn.
Lower semi-continuity. ( is lower semi-continuous since continuous (sum of continuous functions).
Now back to Eq. 26, we are ready to apply the interchangeability principle given in Lemma 7:
F, le, (x.))) = Ep, mine "Ep, ((.,)) +7 nER
Ep, min ((1 v) NER
= min ((E)()))
Ep, Ep,.((((),.)
min By, (e ((v) e ((a.v) + n(v)) - (())E((())
Using this equivalent form, we have:
inf Ep, log(Ep, c(())) - Ep.. (xx,
FEA
inf min Ey (e ( + +n(()) 1) Exp.. (f(x,) fEx ())
inf (EG())
Ep,., Ep (a,))=)o + Ep, Bly) - Epn. (f y)-
follows that
1(X;Y)
inf fext .E5O)) E Ep..,, ((a.v) e ((..v))((()) n(v) + Bp,,(()) Ep..y (f v)
Ep f(, v) - Ep. py p (z.2)) ((2.)))-(())) Ep, (1v) + 1.
sup TEx(nED())
Consider now a function space He from y RR we have:
I(X;Y)
sup Ep.., (xx y) - Ep (m.))) n(x) - E,, 771) + 1, JEX ME
The MI lower bound is looser than regular 7 MINE,EE, since we have two lower bounds passing from all function spaces 9 (X x Y) on to H. then going from Ef ()) to Ay-
No free lunch: tighter bound but need to estimate a function nTx)) which makes bigger estimation emoos. To get good estimate of 7Tx),, to obtain its optimal value log(E,, (=x,)), log(B,,,e/<x.))) I many samples from Py (which means many epochs cycling through the same data in random order). The --irikk with only scalar variable is better computationally, since it enjoys convexity and has better optimization properties. But as it is clear from this discussion, 77 function of modality is good for conditional MI estimation which we will discuss next.
need for each
18

---

Under review as a conference paper at ICLR 2020
C CONDITIONAL MUTUAL INFORMATION
To illustrate the power of improved Donsker- Varadhan and its variational representation with the 17- trick and interchangeability principle, we show how to extend this Conditional Mutual Information estimation.
We have (X, Y) pair of random variables that may depend another variable Z. This conditional dependence is of high importance in all applications of joint embedding or in the information bottleneck principle, etc.
T(X: 2| = (Payss Palspo/-
meaning that conditioned on Z, how much X and Y are dependent.
Therefore we have:
I(xY\Z)) Ez-p, KL (llPa//pll))))))P
Now applying Donsker-Varre Varadhan Lemma, we have for all and for any function f
KL(Pz.. y -IIPa|=Ppl=> + log ExiiPPle f(z, 2) 2 Ep..ve log(f(x, U, 2))
Now marginalizing on 7, we get
Ez-p, KL(Pe.alellP4111ppp.. + Ezsp, log s
[xx M,, 2 Ezup, Ep log( (x, 21 2))
Hence,
I(X;YIZ) 2 Ezmp, B log(f(z, - Ezup, log Pa far y,
and
I(X;
sup Ezep, Eps,y. (x, Ezap. log s (x, (27)
fE .f0
Now using the n-trick for each followed by the interchangeability principle, we have the final variational form:
sup E. Ez-p, Ep Ep 81 - Esnpa (c-nes) Ep (anMa)) + +m(2)= - 1)
((X;YZZ>
JE
(28)
D JIGSAW SELF-SUPERVISION AND MUTUAL INFORMATION
Jigsaw Task Preprocessing For Jigsaw self superviiovi task, the encoder needs to learn features from images randomly cropped and tiled into 3x3 jigsaw puzzles. In order to avoid shortcuts based on matching edges of patches picking up on color aberration discussed in Noroozi & Favaro (2016), images are resized to 292x292, randomly cropped into 255 x55 and jigsawed into 3x3 or 855 per patch wherein 64x64 patch is randomly cropped, avoiding potentially easy edge matching. 255 x55 crops are randomly assigned to gray scales with probability of prior to patch extraction. Each patch RGB channel is normalized to mean, unit variance. At test time, the 255 crops are obtained from images center crops.
ImageNet 10% For the Jigsaw experiments, we decided to subsample ImageNet to 10% within each class for both training and validation sets (128K training and 5K validation images). The reason that running on all of ImageNet 1.3 Million images was not practical for reasonable training turnaround our infrastructure given what we had to train one set of encoder models from the pretext task and one set classifier models for the target task. Even though we only train classifiers for the target task (as explained later in this section), we still need to forward input features through the Encoder which can still be computationally costly despite only backpropagating through the classifier. This 10% subset of ImageNet was proposed by (Kolesnikov et al., 2019) and can be used for faster training turnaround while still being larger than the Tiny ImageNet dataset (200 classes, 500 training, 50 vaildation, 50 test images per class) often used in publications.
19

---

Under review as a conference paper at ICLR 2020
Jigsaw CE Training For self-supervision Jigsaw pretext task (Noroozi & Favaro, 2016; Kolesnikov
et al., 2019) we aim at solving an image jigsaw puzzle by predicting the scrambling permutation т.
Given image X with x= {21..... .......... 2g 3x3 jigsaw patches, and permutation y n2 : [1,9] 9] - [1, 9). scrambled patches Z=, = ( (1). (9) generates the puzzle. The Encoder F2 is based on ResNet50 and outputs 2048 features for each patch. The classifier C (or "descrambler" in this case) is a simple Neural Network with a linear projection upfront from 2048 to 512 for each patch features that are then stacked (I x())512) and linearly projected a 5 dimensional subspace, followed by batch normalization, ReLU, dropout, and another linear projection from 512 to 100 (number of our subset of permutations from Noroozi & Favaro (2016)). E and C area trained with minibatch made from 16 permutations of 10 images (160 effective minibatch size). The 10 images and 16 permutations are randomized for each minibatches. For both E and C' updates, we use ADAM with (31 = 0.9, 82 = 0.999) with a learning rate of 10-1 that stays fixed for all epochs and no weight decay. We let the model build for a total of 200 epochs with a random seed fixed (for reproducibility). We perform 4 independent builds with 4 different random seeds. We let the model build and just survey the Jigsaw solving accuracy assess the health of training only, not to select and encoder model as performance in pretext task is not good predictor of performance in the target task (as observed in Kolesnikov et al. (2019). We run for 200 epochs so to be able to turnaround results fast on our infrastructure, not for the goal of early stopping as no overfitting was observed.
Jigsaw MI training For MI Jigsaw training, we train E so to maximize MI such that max I (E(E (F)( x), which means minimizing the following losses derived from Eq. 8:
LDv(f) log E. Ev.y P.P, exxp (f v))] Ex.grr,, [f U)
(DV-MINE algo.)
L, = e Ex.g P.P, [exp (f( y))] EnyrP., (f(z, v)] + - (7-MINE algo.)
where r are scrambled images patches, and are scrambling permutations. (I, PPPy are sampled from marginals Par and Py (xr was not scrambled by permutation y) while (zc, y)-y)~~ ~ Pay are sampled from the joint distribution (u permutation scrambled ær). For MI training, we need to define the witness function fxx y). x is the set of patches feature vectors from and y is a one-hot embedding of our 100 permutations from Noroozi & Favaro (2016). I and y are first linearly projected in common dimensionality of 512. Then x goes through a shallow network made of 2 sequential stages each composed of CNN, batch normalization, ReLU, and max pooling. After these 2 stages, we have about a combined 9216 features that we project to 512 dimensional space (same as dimension used for the projection of y)- [1x, y) is then just the dot product of 512 feature vectors from x and y once normalized, i.e. a cosine distance that can give scores between -1 and 1. We train using minibatches of 10 images and 16 permutations (160 effective minibatch size) as sampling from the joint distribution Pay. allowing for 16 unused permutations for sampling from marginal distribution P.Py. Therefore expectation terms in the losses above for joint and marginal distribution samples are using the same number of samples (160) for each minibatch. Again, we use ADAM for E and faz 3) with (31 = 0.9, 32 = 0.999) with a learning rate of 10-4 that stays fixed for all epochs and no weight decay. For MINE, we use ADAM for 1/ with the same parameters as for E and f(x v) but with a learning rate of 10-2 to allow for faster updates of n, 1/ is initialized to for MINIEE We run models for 200 epochs of training. A fixed random seed is given to ensure reproducibility. We use 4 seeds to ensure variability of our model builds (same seeds as used for Jigsaw CE). Once again, we observe the training and validation loss indicators of a training's 's health and observe overfitting.
ImageNet 10% Target Task For our target classification task, Encoders E from 200 epochs of Jigsaw Task training are frozen and used to train classifier C for ImageNet 10% with 1000 output classes. C is composed only of a linear projection from 2048 to 1000 classes. For each input image, we resize the images to 232 and take 192x199 center crop before turning it into 3x3 cells grid where each individual patch (now 64x64) is forwarded through E' to get a 2048 -048-dimma vector for each of the 9 patches. Then, we take the mean of these vectors and pass this 2048-dim mean vector to the classifier C' as done in Kolesnikov et al. (2019). Only the classifier C' is trained using ADAM with (3) = 0.9, 32 = 0.999) with a learning rate of 10-4 that stays fixed for all epochs and no weight decay. We let C' build for total of 200 epochs with random seed fixed (for reproducibilityy); we used seeds for each classification build, for total of 8 final model configuration (4 seeds for CE and MI Jigsaw training and 2 seeds for Target Task task classification). The reason for 4 and 2 seeds is just so that our infrastructure can handle all the required builds (200 epochs of training each time). We then report average and standard deviation over a set of 8 accuracies (top1,
20

---

Under review as a conference paper at ICLR 2020
top5 and top10 in %) the validation set (5K) in Tab. 3. We used ADAM with these parameters for all classifier builds. Here again random seeds were fixed.
Model CE We built a "baseline" we call mode CE in Tab. 3. The CE model is an attempt at providing a meaningful upper-bound on the performance on our MI trained Encoders by using Jigsaw CE trained encoder, freezing it and just learning C. 200 epochs of training is used, 2 random seeds as well. Despite being better than 7-MINE (and DV-MINE), the difference is only 1.9% absolute for topl (for 7 - M N E). This means that our MI training, and particularly 1-MMNEE does a respectable job at learning a useful representation space for the target task.
21


Reviews:
[{"review_id": "rkg_JRR6Kr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper propose a variational bound on the Mutual Information and showed how it can be used to improve the estimation of mutual information. \n\nI am afraid I cannot judge of the quality and correctness of the method introduced by the authors. I am not familiar with the subject and will be a poor judge of the quality of the paper. Nevertheless I find that the presentation of the paper could be improved. \n\nFor instance,  while I enjoyed Fig. 3 that showed the performances of different estimators of the entropy, i add the zoom out on a big screen to be able to see anything at all! Clearly, one the most important figure of the paper will be unreadable when printed! It is also not entirely clear how this figure supports the claim of superiority of the proposed method.\n\nThe only comment I may have is that it would be interesting --since the authors want to apply their bound to the case of neural networks-- to compare with, the rigorous estimation of the entropy of NN with random weights in Gabrie et al, NeurIPS 2018, Fig. 2 . It would be a much challenging task, albeit a synthetic one, than the Gaussian dataset one presented in Fig. 3. \n\n\n\n\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I did not assess the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "rJxjxNSatH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents to use \\eta-trick for log(.) in Donsker-Varadhan representation of the KL divergence. The paper discusses its dual form in RKHS and claims it is better than the dual form of f-divergence (while I don't think it's a correct claim). Nevertheless, in experiments, we see that it outperforms the original neural estimation of mutual information.\n\n[Overall]\nPros:\n1. The idea of avoiding the log-sum-exp computation in the DV representation of KL is good. One of the main reasons is to get rid of biased estimation. This idea may not be too novel, but definitely is useful.\n\nCons:\n1. I don't agree with some claims in the paper. Nevertheless, these claims are some of the main stories supporting the paper.\n2. The presentation of the paper should be improved. Including the presentation flow between sections, and also misleading part in experiments.\n3. There are TOO MANY typos in equations. \n\n[Cons In Details]\n<The role of discussing the dual formulation.>\nThe paper spends huge paragraphs discussing the role of the dual formulation. And it also introduces \\eta-DV-Fisher and \\eta-DV-Sobolev, which can be seen as extensions of the proposed method.\nNevertheless, the author doesn't present an evaluation using the dual form. It's a pity that this part is missing. Having Section 3 makes the paper contain several sporadic arguments unrelating to the research questions. Re-organize the paragraphs/ presentation is suggested.\nAnother example is introducing perturbed loss function into the proposed loss function to make it strictly convex. This paragraph is misleading and can be moved entirely to the Appendix.\n\n<Claim on the Proposed Method is Better than f-divergence>\nThe author emphasizes (in multiple places) that the f-divergence biases the estimate. And it exemplifies from eq. (15) to eq. (16). Nevertheless, in eq. (16), when r* is optimum, there should be no second term. The author's claim is based on the comparisons between eq. (13) and eq. (15) when assuming only the MMD term reaches zero. The statement may not be fair.\n<Section 4>\nSimilar to Section 3, this section is cluttered. I don't get the reason why the author specifically come up with a new section comparing only to one mutual information estimation approach (a-MINE). Another irrelevant part of the research question is point 4 under page 7. Why discussing the extension of the a-MINE to conditional MI estimation?\nSome Typos: In equation (21) and (22), \\eta is missing. In Algorithm 1, there are too many typos such as missing \\theta under f, entirely wrong equation for the Output.\n\n<MI Estimation Experiment>\nCan the author discuss the standard deviation for various MI estimation approaches? The large standard deviation for MINE seems unusual.\n\n<Self-Supervised Experiment>\nThe author mentioned that the network considering only the first two convolutional layers, followed by a linear classifier, leads to the best performance. Is there no other layer in between? Also, in Figure 4 (a) and (c), is the purple-color layer means the final classification layer? It is a bit confusing.\n\n<Appendix>\nI understand most of the people would not read the Appendix, but I do. Missing brackets, grammatic errors,  missing integrals, wrong notations, missing punctuation marks, ill-structured presentations, etc., are the problems in the Appendix. I would greatly appreciate the author also spend some time in the Appendix.\n\n[Summary]\nI would vote for a score with 4 or 5 to this paper.\nRegarding there're only 3/6, I'm proposing a score of 6 now. But I look forward to the authors' response and then addressing the problems that I identified. I feel the paper should be a strong paper after a good amount of revision.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Sket5dr3jB", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n\nThe paper contains one main contribution, the unbiased version of MINE obtained using the eta-trick, however a lot of theory is presented and not always very clearly. While some results are interesting (like the constrained ratio estimation interpretation of the dual) some parts are unclear and of lesser relevance. The section on “What Do Neural Estimators of LK or MI Learn” is a collection of different remarks without much coherence, some of which are imprecisely stated. The comparison with the estimator from Pool et al. (2019) could also be much simplified, in particular I would review the list of remarks below theorem 2.\n\nAnother weakness of the paper is that two important aspects in assessing the quality of an estimator are overlooked:  the variance of the estimator and the performance on finite data. Bias is not the only property which matters, both the variance and the dependence on the number of samples should be assessed experimentally, especially when no discussion or theoretical results are provided.  \n\nI liked to see experiments performed on different tasks and datasets but overall that section could be significantly improved. \n\nThe experiment comparing different MI estimators on synthetic Gaussian data is interesting. The plots are however difficult to read, it would be good to make them larger or split the results in different panels. For the 2D and the 20D case, the results reported in the MINE paper are much closer to the true MI than what is reported here, could the authors explain this difference? It would also be good to see some experiments done with a higher ground truth MI, 3.5 is not a lot for higher dimensional cases.\n\nConcerning the Deep InfoMax experiment, we see some improvements when using the proposed MI estimator with Deep InfoMax, however I doubt that this task is an ideal test case for an MI estimator since it has been shown (On Mutual Information Maximization for Representation Learning, Tschannen et al.) that the performance on downstream supervised tasks often does not clearly depend on the quality of MI estimation. \n\n\nSome additional points:\n\nAssuming a finite dimensional feature map in section 3 actually is a loss of generality. \n\nThe proof of lemma 3 is  hard to follow with some notations used without being properly introduced or changed in unexpected ways (eta switches places in f_b(w, eta), what does q do here?, what does PSD mean?). \n\nThe proof of theorem 1 is again unclear. It starts with a typo in the second line (E_y~Q should be replaced by an integral). A couple of lines below a function alpha is introduced without further explanation. \n\nTypos:\n\nThere is one x missing in the proof of Lemma 1 Appendix A.\n\n\n\n\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Hkg48sR2KH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper develops variations on the Donsker-Varadhan (DV) lower bound on KL divergence which yields a lower bound on mutual information as a special case.  The paper is primarily theoretical with an emphasis on the case where the witness function f is drawn from an RKHS (a support vector machine).  They discuss, and present experimental results where the feature map of the RKHS is computed by a neural network, although the theoretical results largely do not apply to optimization of the neural network.\n\nI have two complaints.  First, the authors ignore fundamental limitations on the measurement of mutual information from finite samples.  See \"Fundamental Limitations on the Measurement of Mutual Information\" by McAllester and Stratos.  This paper makes the intuitively obvious observation that I(X,Y) <= H(X) and one cannot give meaningful lower bound on H(X) larger than about 2 log N when drawing only N samples --- the best test one can use is the birthday paradox and a non-repetitive sample only guarantees that the entropy is larger than about 2 log N.  This is obvious for discrete distributions but holds also for continuous distributions --- from a finite sample one cannot even tell if the distribution is continuous or discrete.   So meaningful lower bounds for \"high dimensional data\" are simply not possible for feasible samples.  Given this fact, the emphasis needs to be on experimental results.\n\nThe experimental results in this paper are extremely weak. They should be compared to those in \"Learning Representations by Maximizing Mutual Information Across Views\" by Philip Bachman, R Devon Hjelm, William Buchwalter\n\n\nResponse to the author response:\n\nThis was written earlier but there was a mishap when I attempted to submit it and it was not actually submitted until comments were no longer available to the authors so I am putting this in the review.\n\nBounding the ratio of the densities already bounds the mutual information.  In order for the actual mutual information to be large (hundreds of bits) the log density ratios must actually be extreme.  We do believe, presumably, that mutual information in video data can be hundreds of bits.  So I am still not convinced that lower bounds on large quantities of mutual information are meaningful.\n\nAlso, your bound, like the DV bound, involves an empirical estimator $\\frac{1}{N} \\sum_{i=1}^N\\; e^{f(x)}$ of an expectation $E_x e^{f(x)}$.  The true expectation of an exponential seems likely to be dominated by rare extremes of f(x) which contribute exponentially to the expectation.  I see no reason to believe that the empirical estimate is meaningful in a real application such as vision.\n\nRegarding the experiments, I do not have much interest in experiments on synthetic Gaussians.  The most meaningful experiments for me are the pre-training results for CIFAR.  But you seem to be comparing yourself to your own implementations of (weak?) baselines rather than performance numbers reported by, for example, Hjelm et al.  or van den Oord et al. (CPC).\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper centers on an unbiased variant of the Mutual Information Neural Estimation (procedure), using the so-called "eta trick" applied to the Donsker-Varadhan lower bound on the KL divergence. The paper's contribution is mainly theoretical though experiments are presented on synthetic Gaussian-distributed data as well as CIFAR10 and STL10 classification experiments (from learned representations).

R1's criticism of the theoretical contributions centers on fundamental limitations on finite sample estimation of the MI, contending that the bounds simply aren't meaningful in high-dimensional settings, and that the empirical work centers on synthetic data and self-generated baselines rather than comparisons to reported numbers in the literature; they were unswayed by the author response, which contended that these criticisms were based on pessimistic worst-case analysis and that "mild assumptions on the mutual information and function class" could render better finite-sample bounds. Some of R3's concerns were addressed by the author rebuttal and associated updates, but remained critical of the presentation, in particular regarding the dual function, and downgraded their score.

Because R2 disclosed that they were outside of their area of strong expertise, a 4th reviewer was sought (by this stage, the paper was the revised version). Concerns about clarity persisted, with R4 remarking that a section was "a collection of different remarks without much coherence, some of which are imprecisely stated". R4 felt variance and sample complexity should be dealt with experimentally, though this was not directly addressed in the author response. R4 also remarked that the plots were difficult to read and questioned the utility of supervised representation learning benchmarks at assessing the quality of MI estimation, given recent evidence in the literature.

The theoretical contributions of this submission are slightly outside the bounds of my own expertise, but consensus among three expert reviewers appears to be that the clarity of exposition leaves much to be desired, and I concur with their assessment that the empirical investigation is insufficiently rigorous and does not draw clear comparisons to existing work in this area. I therefore recommend rejection.
Decision: Reject