Mutual Information Gradient Estimation for  Representation Learning
Authors: Liangjian Wen
Yiji Zhou
Lirong He
Mingyuan Zhou
Zenglin Xu
Venue:  (2020)
Keywords: Mutual Information
Score Estimation
Representation Learning
Information Bottleneck

Abstract: Mutual Information (MI) plays an important role in representation learning. However, MI is unfortunately intractable in continuous and high-dimensional settings. Recent advances establish tractable and scalable MI estimators to discover useful representation. However, most of the existing methods are not capable of providing an accurate estimation of MI with low-variance when the MI is large. We argue that directly estimating the gradients of MI is more appealing for representation learning than estimating MI in itself. To this end, we propose the Mutual Information Gradient Estimator (MIGE) for representation learning based on the score estimation of implicit distributions. MIGE exhibits a tight and smooth gradient estimation of MI in the high-dimensional and large-MI settings. We expand the applications of MIGE in both unsupervised learning of deep representations based on InfoMax and the Information Bottleneck method. Experimental results have indicated significant performance improvement in learning useful representation.

Published as a conference paper at ICLR 2020
MUTUAL INFORMATION GRADIENT ESTIMATION FOR REPRESENTATION LEARNING
Liangjian Wen' Yiji Zhou', Lirong He', Mingyuan Zhou?, Zenglin Xu SMILE Lab, School of Computer Science and Engineering
University of Electronic Science and Technology of China, Chengdu, China
2 Center for Artificial Intelligence
Peng Cheng Laboratory. Shenzhen, China
3 McCombs School of Business
University of Texas at Austin, Austin, United States
4 School of Computer Science and Technology
Harbin Institute of Technology, Shenzhen, China
w1 1616 6 @egmall 1. com, zhouy i ji fotootttiiee out look com, ronghhee221@1@ com, mingyuan. zhou @mccombs utexas edu,, xuzenglin@hit. eed.i.
ABSTRACT
Mutual Information (MI) plays important role in representation learning. How- ever, MI is unfortunately intractable continuous and high-dimensional settings. Recent advances establish tractable and scalable MI estimators discover useful representation. However, most of the existing methods are not capable of pro- viding an accurate estimation of MI with low-variance when the MI is large. We argue that directly estimating the gradients of MI is more appealing for represen- tation learning than estimating MI in itself. To this end, we propose the Mutual Information Gradient Estimator (MIGE) for representation learning based on the score estimation of implicit distributions. MIGE exhibits a tight and smooth gra- dient estimation of MI in the nigh-dimensional and large-MI settings. We expand the applications of MIGE in both unsupervised learning of deep representations based on InfoMax and the Information Bottleneck method. Experimental results have indicated significant performance improvement learning useful represen- tation.
INTRODUCTION
Mutual information (MI) is an appealing metric widely used in information theory and machine learning to quantify the amount shared information between a pair of random variables. Specifi- cally, given pair of random variables x, y, the MI, denoted by I (x:v), is defined as
T(x;)) = (x(x.)) log
p(x, y) p(x)p(y)
(1)
where E' is the expectation over the given distribution. Since MI is invariant to invertible and smooth transformations, it can capture non-linear statistical dependencies between variables (Kinney & At-
wal 2014). These appealing properties make it act as fundamental measure of true dependence. Therefore, MI has found applications in wide range machine learning tasks, including feature
selection wKwak & Choil 2002 Fleuret, 2004 Peng et al.] 2005). clustering (Müller et al.) 2012
Ver Steeg & Galstyan| 2015). and causality (Butte Kohane 1999). has also been pervasively used in science, such as biomedical sciences (Macs al. 1997). computational biology (Krish-
naswamy al. 2014), and computational neuroscience (Palmer et al. 2015)).
Recently, there has been revival of methods in unsupervised representation learning based on MI. A seminal work is the InfoMax principle (Linsker 1988), where given an input instance æ, the goal of the InfoMax principle is to learn a representation (x) by maximizing the MI between the input and its representation. A growing set of recent works have demonstrated promising empirical performance in unsupervised representation learning via MI maximization (Krause al. 2010 Hu

---

Published as a conference paper at ICLR 2020
et al. 2017 Alemi et al. 801 Oord et al. 2018; Hjelm et al. 2019). Another closely related work is the Information Bottleneck method (Tishby et al. 2000; Alemi et al. 2017). where MI is used to limit the contents of representations. Specifically, the representations are learned by extracting task- related information from the original data while being constrained to discard parts that are irrelevant to the task. Several recent works have also suggested that by controlling the amount of information between learned representations and the original data, one can tune desired characteristies of trained models such as generalization error (Tishby & Zaslavsky 2015 Vera et al. 2018), robustness (Alemi et al. 2017), and detection of -utto--distribution data (Alemi et al. 2018a).
Despite playing a pivotal role across a variety of domains, MI is notoriously intractable. Exact computation is only tractable for discrete variables, or for a limited family of problems where the probability distributions are known. For more general problems, MI is challenging to analytically compute or estimate from samples. A variety of MI estimators have been developed over the years, including likelihood-ratio estimators (Suzuki et al.) 2008). binning (Fraser & Swinney) 1986, Dar- bellay& & Vajda 1999 Shwartz-Ziv & TTishyy] 2017)). k-nearest neighbors (Kozachenko & Leonenko) 1987 Kraskov et al. 2004] Pérez-Cruz] 20088] Singh & Póczos| (0166), and kernel density estima- tors Moon al. 1995| Kwak & Choi 22022; Kandasamy al. 2015). However, few of these
mutual information estimators scale well with dimension and sample size in machine learning prob-
lems (Gao et al. (0155).
In order to overcome the intractability of MI in the continuous and high-dimensional settings, Alemi eet al.] (2017) combines variational bounds of Baareer & Agakov (2003) with neural networks for the estimation. However, the tractable density for the approximate distribution is required due to variational approximation. This limits its application the general-purpose estimation, since the underlying distributions are often unknown. Alternatively, the Mutual Information Neural Estima- tion (BIle.BBelghazi et al. (2018) and the Jensen- -hanennn- MI estimator (JSD, Hjellm et al., (2019) enable differentiable and tractable estimation of MI by training a discriminator to distinguish sam- ples coming from the joint distribution or the product of the marginals. In detail, MINE employs a lower-bound to the MI based on the Donsker Varadhan representation of the KI.-divergence, and JSD follows the formulation of f-GAN KL-divergence. In general, these estimators are often noisy and can lead to unstable training due to their dependence on the discriminator used to estimate the bounds of mutual information. As pointed out by Poole et al. (2019), these unnormalized critic estimators of MI exhibit high variance and are challenging to tune for estimation. An alternative varinariance choice of MI estimator is Information Noise- Contrastive Estimation (InfoNCE, Oord ee al. (2018),, which introduces the Hoise-Contrassiig Estimation with flexible critics parameterized by neural networks as a bound to approximate MI. Nonetheless, its estimation saturates at log of the batch size and suffers from high bias. Despite their modeling power, none of the estimators are capable of providing accurate estimation of MI with low variance when the MI is large and the batch size small (Poole et al. 2019). As supported by the theoretical findings in McAllester & Statos] (2018), any distribution- free high-confidence lower bound on entropy requires a sample size exponential in the size of the bound. More discussions about the bounds of MI and their relationship can be referred to Poole et al. (2019).
In summary, existing estimators first approximate MI and then use these approximations to optimize the associated parameters. For estimating MI based on any finite number of samples, there exists an infinite number of functions, with arbitrarily diverse gradients, that can perfectly approximate the true MI at these samples. However, these approximate functions can lead to unstable training and poor performance in optimization due to gradients discrepancy between approximate estimation and true MI. Estimating gradients of MI rather than estimating MI may be better approach for MI optimization. To this end, to the best of our knowledge, we firstly propose the Mutual Information Gradient Estimator (MIGE) in representation learning. In detail, we estimate the score function of an implicit distribution, Vx log (x), to achieve general -purpose MI gradient estimation for representation learning. In particular, to deal with high-dimensional inputs, such as text, images and videos, score function estimation via Spectral Stein Gradient Estimator (SSGE) (Shi et al., 2018) computationally expensive and complex. We thus propose an efficient high-dimensional score function estimator to make SSGE scalable. To this end, we derive new reparameterization trick for the representation distribution based on the lower-variance reparameterization trick proposed by Roeder et al. (2017).
general-purpose
We summarize the contributions of this paper as follows:
2

---

Published as a conference paper at ICLR 2020
=
= 10
= 20
Mutual Information
2.5
True Mi MINE
2.0
1.5
1.0
0.0
10
True Gradient
20-
40
V. MINE
Gradient
V, MINE-f
10
20
MIGE (ours)
-0.5
p
P
P
Figure Estimation performance of MINE, MINE-f and MIGE. Each estimation approach has been taken additional 20 times and plotted with light curves. Top: True MI and corresponding estimation of MINE and MINE-f. Bottom: True gradient and corresponding estimation of MINE, MINE-f and MIGE, Our approach MIGE only appears in bottom figures since it directly gives gradient estimation. As we observe, MIGE gives more stable, smooth and accurate results.
We propose the Mutual Information Gradient Estimator (MIGE) for representation learning based on the score function estimation of implicit distributions. Compared with MINE and MINE-/. MIGE provides tighter and smoother gradient estimation of MI in a high- dimensional and large-MI setting, as shown in Figure I of Section
We propose the Scalable SSGE to alleviate the exorbitant computational cost of SSGE in high-dimensional settings.
To learn meaningful representations, we apply SSGE gradient estimators for both In- foMax and Information Bottlenck, and have achieved improved performance than their corresponding competitors.
2 SCALABLE SPECTRAL STEIN GRADIENT ESTIMATOR
Score estimation of implicit distributions has been widely explored in the past few years (Song et al.,
2019 Li & Turner 2017 Shi al. 2018). promising method of score estimation the Stein
gradient estimator Li Turnr 201 Shi et al. (018). which is proposed for implicit distributions. inspired by generalized Steins identity (Gorham & Mackey 2015 Liu & Wang) 2016) as follows.
Steins identity. Let q(x) a continuously differentiable (also called smooth) density supported on X' C Rd, and h(x) = [/ (x),11(x)), 12(x),.... (x)]² is a smooth vector function. Further, the boundary conditions on h is
(x) 0, Vx E 0X if X' is compact, or lim ((x)h(x) = 0 x = Rd.
(2)
x-000
Under this condition, the following identity can be casily checked using integration by parts, assum- ing mild zero boundary conditions on h.
Eg [(x)VVx Tx) Vxh(x)] =.

---

Published as a conference paper at ICLR 2020
Here h is called the Stein class of q(x) if Steins identity Eq. 3 holds. Monte Carlo estimation of the expectation in Eq. (3) builds the connection between Vx gogg and the samples from g(x) in Steins identity. For modeling implicit distributions, Motivated by Steins identity, [Shi al. (2018)
proposed Spectral Stein Gradient Estimator (SSGE) for implicit distributions based on Stein 's iden- tity and spectral decomposition of kernel operators where the eigenfunctions are approximated by the Nyström method. Below we briefly review SSGE. More details refer to Shi et al. (2118) Specif- ically, we denote the target gradient function
Vx log (x).. The 2th component of the gradient is (x) = Xx, log ((x). We assume 91..... .9 E ( q). {15) denotes an orthonormal basis of L (X', q). We can expand g. gi(x) into the spectral series, i.e. (x) -j (,y11,(x) (x). The value of the jth eigenfunction th at x can be approximated by the Nyström method (Xu et al. 2015). Due to the orthonormality of eigenfunctions { 12 there is a constraint under the probability measure q(): I (x) (x))(())(x) dig. where d,g 1li = j]. Based on this constraint, can obtain the following equation for { 1
estimate by g : X I g(x)
f ((x,))) = jub(();
(4)
where k(.) is a kernel function. The left side of the above equation can be approximated by the Monte Carlo estimate using i.i.d. samples x' x..,,"M from q(.) M Kg pab, where K is the
Gram Matrix and 2 ( (x*) ..... 1b [x T We can solve this eigenvalue problem by choose the J largest eigenvalues A 2 2 A for K. Uy denotes the eigenvector of the Gram matrix. The
approximation for {155) can be obtained combined with Eq. (4p following: (xy(x) R (x)
VM Σm Ujjnk xx)
X;
Furthermore, based on the orthonormality of {Wg we can easily obtain Big = -E, Vx, (x). By taking derivative both sides of Eq. (4), we can show that:
M3 Vx 45 Xx / / ((xx)) /Vta:///d/////....//..vv// /Va.As,,))55/000111
(5)
Then estimate following:
Vx, (x, x") 1h (x"
PyM
Finally, by truncating the expansion to the first terms and plugging in the Nyström upproximations of {15) 221 we can get the score estimator:
(x Σ (x),
By=
1 M
Σ m=
V 2 (x
In general, representation learning for large-scale datasets usually costly in terms of storage and computation. For instance, the dimension of images in the STL-10 dataset is 96 x 96 x 3 (i.e., the vector length is 27648). This makes it almost impossible to directly estimate the gradient of MI between the input and representation. To alleviate this problem, we introduce random projection (RP) (Bingham & Mannila 2001) to reduce the dimension of x.
We briefly review RP. More details refer to Bingham & Mannila (001)) RP projects the original d-dimensional data into k-dimensional (A: << d) subspace. Concretely, let matrix Xax denotes the original set of N d-dimensional data, the projection of the original data X kx is obtained by in- troducing a random matrix Rexd whose columns have unit length, as follows (Bingham & Mannila, 2001), X-R x BexaXaxxM After RP, the Euclidean distance between two original data vectors can be approximated by the Euclidean distance of the projective vectors in reduced spaces:
|x|
x
v d/k ||xx
Rxall,
(8)
where X₁ and x₂ denote the data vectors in the original large dimensional space.
4

---

Published as a conference paper at ICLR 2020
Based on the principle of RP, can derive Salable Spectral Stein Gradient Estimator, which is an efficient high-ding high-dimensional score function estimator. One can show that the RBF kernel satisfies Steins identity (Liu & Wang 2016). [Shi et al. (2018) also shows that it is promising choice for SSGE with a lower error bound. To reduce the computation of the kernel similarities of SSGE in high-dimensional settings, we replace the input of SSGE with projections obtained by RP accord- ing to the approximation of Eq. (8) for the computation of the RBF kernel.
MUTUAL INFORMATION GRADIENT ESTIMATOR
As gradient estimation is a straightforward and effective method in optimization, we propose gradient estimator for MI based on score estimation of implicit distributions, which is called Mutual Information Gradient estimator (MIGE). In this section, we focus on three most general cases of MI gradient estimation for representation learning, and derive the corresponding MI gradient estimator for these circumstances.
We outline the general setting of training an encoder to learn a representation. Let X' and 2 be the domain, and Eg X' - Z with parameters & denotes continuous and (almost everywhere) differentiable parametric function, which is usually neural network, namely an encoder. p(x) denotes the empirical distribution given the imput data E X. We can obtain the representation of the input data through the encoder, z = Eg (x). tv (2) is defined as the marginal distribution induced by pushing samples from p(x) through encoder (-) We also define qu (x, z) the joint distribution with x and z, which is determined by encoder (.).
Circumstance I. Given that the encoder (-) deterministic, our goal is estimate the gradient of MI between input x and encoder output z w.r.t. the encoder parameters tb. There is a close relationship between mutual information and entropy, which is following:I/ z) [() + He (z) H (x, z). Here (x) is data entropy and not relevant to tb The optimization of L (x, z) with parameters & can neglect the entry (x). We decompose the gradient of the entropy of (z) and qu (x, 2) as (see Appendix A):
(z) V gy (z) (Z)), V (x, - (x. [log qxx, z)].
Hence, we can represent the gradient of MI between input x and encoder output z w.r.t. encoder parameters as following:
T = = - Ey (z) [log [()) + V (x. ((x, 2)].
(10)
However, this equation is intractable since an expectation w.r.t (z(z) is directly not diffcrentiable w.r.t vb. Roeder et al. (2017) proposed general variant of the standard reparameterization trick for the variational evidence lower bound, which demonstrates lower-variance To address above prob- lem, we adapt this trick for MI gradient estimator in representation learning. Specifically, we can obtain the samples from the marginal distribution of z by pushing samples from the data empirical distribution p(x) through (-) for representation learning. Hence can reparameterize the repre- sentations variable z ~ To (z) using a differentiable transformation: = (x(x). with x ~ p(x), where the data empirical distribution p(x) is independent of encoder parameters tb. This reparame- terization can rewrite an expectation w.r.t gu (z) and qx z) such that the Monte Carlo estimate of the expectation is differentiable w.r.t tb.
Relying on this reparameterization trick, we can represent the gradient of MI w.r.t. encoder param- eters Eq. as follows:
V (x; = (yx(x) V2 (o
+ Eg(x) V (xxx)) log (xx, E (x) (x, Es (x))],
(1I)
where the score function V, log Le E. (x)) can be estimated based on i.idd samples from an im- plicit density go (Ep (x)) (Shi et al.) 2018 Song et al. 2019). The samples form the joint dis- tribution qo (x, z) are produced as tollowing: we sample observations from empirical distribution p(x); then the corresponding samples of is obtained through (.). Hence
also estimate V (xx.)) log qxx, E, (x)) based on i.i.d. samples from qu (x, Ex, (x)). V E.g (x) and (x E, (x)) directly computed with x.
5

---

Published as a conference paper at ICLR 2020
Circumstance II. Assume that encode the input to latent data space h Co (x) that reflects useful structure in the data. Next, we summarize this latent variable mapping into final representa- tions by the function fw, Eg (x) fu Cp (x). The gradient estimator of MI between h and represented by the data reparameterization trick as follows:
V L (h:z) V H (z) V H.
(12)
- V. log (x)) (
V Cx(xx
+ [g(x) V (h,x) log q( (x)) V (x)))
(13)
Circumstance III. Consider stochastic encoder function F. e) where is auxiliary variable with independent marginal p(E). By utilizing data reparameterization trick. we can represent the gradient of the conditional entropy H
follows (see Appendix A): v (2)xx) = -I p(x) [Bpte) (V (alx) log g( Eas (x, V Fx,
(14)
where the term V (z|x) log q( E E) x) can easily estimated by score estimation.
Based on the condition entropy gradient estimation in Eq. (14). the gradient estimator of MI between input and encoder output can be represented as following:
V V = V H (z) - V (z
(15)
p(x))(()) [og (x,
-Ept) [ [V log q( Ess (x,
(x,
(16)
In practical MI optimization, we can construct MIGE of the full dataset based on mini-batch Monte Carlo estimates. We have provided algorithm description for MIGE in Appendix B.
4 TOY EXPERIMENT
Recently, MINE and MINE-/ enable effective computation of MI in the continuous and high- dimensional settings. To compare with MINE and MINE-J, we evaluate MIGE in the correlated Gaussian problem taken from (Belghazi et al. 2018).
Experimental Settings. We consider random variables and y (x,y Rd), coming from a 2d-dimension multivariate Gaussian distribution. The component-wise correlation of x and y defined as follows: corr(x;, SipP, P (-1, 1). where dij Kronecker's delta and p is the correlation coefficient. Since MI is invariant to smooth transformations of x, we only consider standardized Gaussian for marginal distribution p(x) and p(y). The gradient of MI w.r.t p has the analytical solution: (x = - We apply MINE and MINE- f to estimate MI of x. y by sampling from the correlated Gaussian distribution and its marginal distributions, and the corresponding gradient of MI w.r.t P can be computed by backpropagation implemented in Pytorch.
Results. Fig I presents our experimental results in different dimensions d = (5, 10, 20}. In the case of low-dimensional (d 5). all the estimators give promising estimation of MI and its gradient. However, the MI estimation of MINE and MINE- are unstable due to its relying on a discriminator corresponding estimation of
produce estimation of the bound on MI. Hence, showed in Fig 1 MI and its gradient is not smooth. As the dimension d and the absolute value of correlation coeffi- cient |p| increase, MINE and MINE-f are apparently hard to reach the True MI, and their gradient estimation of MI is thus high biased. This phenomenon would be more significant in the case of high-dimensional or large MI. Contrastively, MIGE demonstrates the significant improvement over MINE and MINE-f when estimating MI gradient between y.y. In this experiment, we compare our method with two baselines on an analyzable problem and find that the gradient curve estimated by our method is far superior to other methods in terms of smoothness and tightness in a high-dimensional and large-MI setting compared with MINE and MINE-f.
random variables
5 APPLICATIONS
To demonstrate the performance in downstream tasks, we deploy MIGE to Deep InfoMax (Hjelm et al., 2019) and Information Bottleneck (Tishby al. 2000) respectively, namely replacing the original MI estimators with MIGE. We find that MIGE achieves higher and more stable classification accuracy, which indicating its good gradient estimation performance in practical applications.

---

Published as a conference paper at ICLR 2020
Table 1: CIFAR-I0 and CIFAR-100 classification accuracy (top 1) of downstream tasks compared with vanilla DIM. JSD and infoNCE are MI estimators, and PM denotes matching representations
prior distribution (Hjelm al. 2019).
CIFAR-10
C-FAR-100
Model
conv
fc(1024)
Y(64)
conv
fc(1024)
Y(64)
DIM (JSD)
55.81% %
45.73%
40.67%
28.41%
22.16%
16.50%
DIM (JSD PM)
52.2%
52.84%
43.17%
24.40%
18.22%
15.22%
DIM (infoNCE)
51.82%
42.81%
37.79%
24.60%
16.54%
12.96%
DIM (infoNCE PM)
56.77%
49.42%
42.68%
25.51%
20.15%
15.35%
MIGE
57.95
57.09%
53.75%
29.86%
27.91%
25.84%
Table 2: STL-10 classification accuracy (top 1) down- stream tasks compared with vanilla DIM. The dimension of ST images (27648) results in exorbitant computational cost. Random Projection (RP) is applied to reduce the di- mension.
46
Y(64) accurac
44
43
42
41
STL-10
16
32
128
256
512 1024
Model
conv
fo fc(1024)
Y(64)
RP dimension
DIM (JSD)
42.03%
30.28%
28.09%
DIM (infoNCE)
43.13%
35.80%
34.44%
Figure STL-10 Y(64) classification accuracy (top 1) with different RP di- mension.
MIGE
unaffordable computational cost
MIGE RP to 512d
52.00%
48.14%
44.89%
5.1
DEEP INFOMAX
Discovering useful representations from unlabeled data is cently, growing set of methods is explored to train deep neural network encoders by maximizing the mutual information between its input and output. number of methods based on tractable vari- ational lower bounds, such as JSD and infoNCE, have been proposed to improve the estimation of MI between high dimensional input/output pairs of deep neural networks (Hjelm et al.) (019).. To compare with JSD and infoNCE, expand the application of MIGE in unsupervised learning of deep representations based on the InfoMax principle.
problem for deep learning. Re-
Experimental Settings. For consistent comparison, we follow the experiments of Deep Info- Max to set the experimental setup as in Hjelm al. (2019). We test DIM on image datasets 100 CIFAR-100 and STL-10 to evaluate our MIGE. For the high-dimensional images in STL-10, directly applying SSGE almost impossible since it results exorbitant computational cost. Our proposed Scalable SSGE is applied. to reduce the dimension of images and achieve rea- sonable computational cost. As mentioned in Hjelm et al. (2019), non-linear classifier is chosen to evaluate representation, After learning representation, and train non-linear classifier using the representation as the input. The same classifiers are used for all methods. Our baseline results directly copied from Hjelm al. (2019) or by running the code of author.
freeze the parameters of the encoder
Results. As shown in Table 1 MIGE outperforms all the competitive models in DIM experiments on CIFAR-10 10 and CIFAR- 100. Besides the numerical improvements, it is notable that our model have the less accuracy decrease across layers than that of DIM(JSD) and (IM(inoNCE)) The results indicate that, compared to variational lower bound methods, MIGE gives more favorable gradient direction, and demonstrates more power controlling information flows without significant loss. With the aid of Random Projection, we could evaluate on bigger datasets, e.g., STL-10. Table P shows the result of DIM experiments on STL-10. We can observe significant improvement over the baselines when RP to 512d. Note that our proposed gradient estimator can also be extended to the multi-view setting(i.e.. with local and global features) DIM, is beyond the scope of this paper. More discussions refer to Appendix C.
Codes available https://p thub. http://:tttuu....comm/eevvvrr

---

Published as a conference paper at ICLR 2020
Ablation Study. To verify the effect of different dimensions of Random Projection classification accuracy in DIM experiments, we conduct an ablation study on STL-10 with the above experimental settings. Varying RP dimension &s { {16, 32, 64, 128, 256, 512, 1024). we measure the classification accuracy of Y(64) which is shown in Fig 2 We find that the classification accuracy increases with RP dimension from 16 to 128. After that, the approximation in Equ. ( with the further increase of the RP dimension reaches saturation, while bringing extra computational costs.
5.2 INFORMATION BOTTLENECK
Information Bottleneck (IB) has been widely applied to variety of application domains, such as classification (Tishby & Zaslavsky, 2015. Alemi et al. 2017; Chalk et al. 2016: Kolchinsky et al. 2017), clustering (Slonim & Tishby 2000), and coding theory and quantization (Zeitler | 2008 Courtade Wesel 2011). In particular, given the input variable x and the target variable y. the goal of the IB is to learn a representation of (denoted by the variable that satisfies the following characteristics:
1) z is sufficient for the target y, that is, all information about target y contained in x should also be contained in In optimization, it should be
2) z is minimal. In order not contain irrelevant information that is not related to y, is required to contain the smallest information among all sufficient representations.
The objective function of IB is written as follows:
max (); S.I. [(z; x) S
(17)
Equivalently, by introducing Lagrangian multiplier 8, the IB method can maximize the following objective function: GıB [(z:y) 3I(
[[z x). Further, is generally acknowledged that I(z; = H(y) Hy)z)) and (y) is constant. Hence can also minimize the objective function of the following form:
LIB Hyl2)) + (1;
(18)
where 3 2 0 plays a role in trading off the sufficiency and minimality. Note that the above formulas omit the parameters for simplicity.
To overcome the intractability of MI in the continuous and high-dimension setting, Alemi et al. (017)) presents variational approximation IB, which adopts deep neural network encoder to produce conditional multivariate normal distribution, called Deep Variational Bottleneck (DVB). Rencently, DVB exploited restrict the capacity of discriminators in GANs (Peng al., 2019) However, a tractable density is required for the approximate posterior in DVB due to their reliance on a variational approximation while MIGE does not.
To evaluate our method, we compare MIGE-IB with DVB and MINE-IB in IB application. We demonstrate an implementation of the IB objective on permutation invariant MNIST using MIGE.
Experiments. For consistent comparison, we adopt the same architecture and empirical settings used in Alemi et al.] (2017) except that the initial learning rate of 2e-4 is set for Adam optimizer, and exponential decay with decaying rate by factor of 0.96 was set for every 2 epochs. The implementation of DVB is available from its author?? Under these experimental settings, we use our MI Gradient Estimator to replace the MI estimator in DVB experiment. The threshold of score function's Stein gradient estimator is set as 0.94. The threshold is the hyper-parameter of Spectral Stein Gradient Estimator (SSGE), and it is used to set the kernel bandwidth of RBF kernel. Our results can be seen in Table and it manifests that our proposed MIGE-IB outperforms DVB and MINE-IB.
brrbb_eeem demo
8

---

Published as a conference paper at ICLR 2020
Table 3: Permutation-invariant MNIST misclassification rate. Datas except our model cited from Belghazi et al. (2018)
Model
Misclass rate
Baseline
1.38%
Dropout
1.34%
Confidence penalty
1.36%
Label Smoothing
1.4%
DVB
1.13%
MINE-IB
1.11%
MIGE-IB (ours)
1.05%
CONCLUSION
In this paper, we present a gradient estimator, called Mutual Information Gradient Estimator (MIGE), avoid the various problems met in direct mutual information estimation. We manifest the effectiveness of gradient estimation of MI over direct MI estimation by applying it in unsupervised or supervised representation learning. Experimental results have indicated the remarkable improve- ment over MI estimation in the Deep InfoMax method and the Information Bottleneck method.
ACCKNOWLEDGEMEET
This work was partially funded by the National Key R&D Program of China (No. 2018YFB FEFFBBSS0 1005100 & No. 2018 1005104).

---

Published as a conference paper at ICLR 2020
REFERENCES
Alexander A. Alemi, Ian Fischer, Joshua V. Dillon, and Kevin Murphy, Deep variational information bottleneck. In ICLR, 2017.
Alexander A Alemi, lan Fischer, and Joshua V Dillon. Uncertainty in the variational information bottleneck. arXiv preprint arXiv: / 807.00906, 2018a.
Alexander A. Alemi, Ben Poole, Ian Fischer, Joshua V. Dillon, Rif A. Saurous, and Kevin Murphy. Fixing broken ELBO, ICML, 2018b.
David Barber and Felix V Agakov. The im algorithm: a variational approach to information maxi- mization. In Advances in neural information processing systems, pp. None, 2003.
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeswar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and R Devon Hjelm. Mine: mutual information neural estimation. ICMI., 2018.
Ella Bingham and Heikki Mannila. Random projection dimensionality reduction: applications to image and text data. In Proceedings of the seventh ACM SIGKDD international conference on Knowledge discovery and data mining, San Francisco, CA, USA, pp. 245-250, 2001.
Atul J Butte and Isaac S Kohane. Mutual information relevance networks: functional genomic clustering using pairwise entropy measurements. In Biocomputing 2000, pp. 418-429. World Scientific, 1999.
Matthew Chalk, Olivier Marre, and Gasper Tkacik. Relevant sparse codes with variational informa- tion bottleneck. In Advances in Neural Information Processing Systems, pp. 1957-1965, 2016.
Thomas A Courtade and Richard D Wesel. Multiterminal source coding with an entropy-based distortion measure. In 2011 IEEE International Symposium on Information Theory Proceedings, pp. 2040-2044. IEEE, 2011.
Georges A Darbellay and Igor Vajda. Estimation of the information by adaptive partitioning of the observation space. IEEE Transactions on Information Theory, 45(4):1315-1321, 1999.
François Fleuret. Fast binary feature selection with conditional mutual information. Journal of Machine learning research, 5(Nov):1531-1555, 2004.
Andrew M Fraser and Harry I. Swinney. Independent coordinates for strange attractors from mutual information. Physical review A, 33(2): 134, 1986.
Shuyang Gao, Greg Ver Steeg, and Aram Galstyan. Efficient estimation of mutual information for
strongly dependent variables. In Artificial intelligence and statistics, pp. 277-286, 2015.
Jackson Gorbam and Lester Mackey, Measuring sample quality with stein's 's method. In Advances in Neural Information Processing Systems, pp. 226-234, 2015.
R. Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildonn Karan Grewal, Philip Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In ICLR, 2019.
Weihua Hu, Takeru Miyato, Seiya Tokui, Eiichi Matsumoto, and Masashi Sugiyama. Learning discrete representations via information maximizing self-augmented training. In ICML, 2017.
Kirthevasan Kandasamy, Akshay Krishnamurthy, Barnabas Poczos, Larry Wasserman, et al. Non- parametric von mises estimators for entropies, divergences and mutual informations. In Advances in Neural Information Processing Systems, pp. 397-405, 2015.
Justin B Kinney and Gurinder Atwal. Equitability, mutual information, and the maximal informa- tion coefficient, Proceedings of the National Academy of Sciences, 11109):334-3359, 2014.
Artemy Kolchinsky, Brendan D Tracey, and David H Wolpert. Nonlinear information bottleneck. arXiv preprint arXiv: 170..02436..66 2017.
10

---

Published as a conference paper at ICLR 2020
LF Kozachenko and Nikolai N Leonenko. Sample estimate of the entropy of random vector. Problemy Peredachi Informatsi, 23(2)99-16, 16, 1987.
Alexander Kraskov, Harald Stögbauer, and Peter Grassberger. Estimating mutual information. Phys- ical review E, 69(6):06615) 138, 2004.
Andreas Krause, Pietro Perona, and Ryan G Gomes. Discriminative clustering by regularized infor- mation maximization. In Advances in neural information processing systems, 2010.
Smita Krishnaswamy, Matthew H. Spitzer, Michael Mingueneau. Sean C Bendall, Oren Litvin, Erica Stone, Dana Peer, and Garry P Nolan. Conditional density-based analysis of t cell signaling in single-cell data. Science, 346(6213): 125666., 2014.
Nojun Kwak and Chong-Ho Choi. Input feature selection by mutual information based on parzen window. IEEE Transactions on Pattern Analysis & Machine Intelligence, (12):1667-1671, 2002.
Yingzhen Li and Richard E Turner. Gradient estimators for implicit models. arXiv preprint arXiv:1705.07107, 2017.
Ralph Linsker. Self-organization in a perceptual network. Computer, 21 1988.
Qiang Liu and Dilin Wang. Stein variational gradient descent: A general purpose bayesian inference algorithm. Advances in neural information processing systems, pp. 2378 22866, 2016.
Frederik Maes, Andre Collignon, Dirk Vandermeulen, Guy Marchal, and Paul Suetens. Multimodal- ity image registration by maximization of mutual information. IEEE transactions on Medical Imaging, 16(2):187-198; 1997.
David McAllester and Karl Statos. Formal limitations on the measurement of mutual information. arXiv preprint arXiv:1811.0425/1 2018
Young-Il Moon, Balaji Rajagopalan, and Upmanu Lall. Estimation of mutual information using kernel density estimators. Physical Review E, 52(3):2318, 1995.
Andreas C Müller, Sebastian Nowozin, and Christoph H Lampert. Information theoretic clustering using minimum spanning trees. In Joint DAGM (German Association for Pattern Recognition) and OAGM Symposium, pp. 205-215. Springer, 2012.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. NIPS, 2018.
Stephanie E Palmer, Olivier Marre, Michael J Berry, and William Bialck. Predictive information in a sensory population. Proceedings of the National Academy of Sciences, 1 112(22))6908-66113 2015.
Hanchuan Peng, Fuhui Long, and Chris Ding. Feature selection based mutual information: criteria of max-dependency, max-relevance, and min-redundancy. IEEE Transactions on Pattern Analysis & Machine Intelligence, (8):1226-1238. 2005.
Xue Bin Peng, Angjoo Kanazawa, Sam Toyer, Pieter Abbeel, and Sergey Levine. Variational dis- criminator bottleneck: Improving imitation learning, inverse rl, and gans by constraining infor- mation flow. In 7th International Conference on Learning Representations, ICLR 2019, New
Orleans, LA, USA, May 6-9, 2019, 2019.
Fernando Pérez-Cruz. Kullback-leible divergence estimation of continuous distributions. In 2008 IEEE international symposium on information theory, pp. 1666-1670. IEEE, 2008
Ben Poole, Sherjil Ozair, Aäron van den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, pp. 5171-5180, 2019.
Geoffrey Roeder, Yuhuai Wu, and David K Duvenaud. Sticking the landing: Simple, lower-variance gradient estimators for variational inference. In Advances in Neural Information Processing Sys- tems, pp. 6925-6934, 2017.
11

---

Published as a conference paper at ICLR 2020
Jiaxin Shi, Shengyang Sun, and Jun Zhu. A spectral approach to gradient estimation for implicit 2018.
distributions. arXiv preprint
Ravid Shwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via informa- tion. arXiv preprint arXiv: 777.0000000 2017.
Shashank Singh and Barnabás Póczos. Finite-sample analysis of fixed-k nearest neighbor density functional estimators. In Advances in neural information processing systems, pp. 1217-1225, 2016.
Noam Slonim and Naftali Tishby. Document clustering using word clusters via the information bottleneck method. In Proceedings of the 23rd annual international ACM SIGIR conference Research and development in information retrieval, pp. 208-215. ACM, 2000.
Yang Song, Sahaj Garg, Jiaxin Shi, and Stefano Ermon. Sliced score matching: A scalable approach to density and score estimation. arXiv preprint arXiv: 07008, 2019.
Taiji Suzuki, Masashi Sugiyama, Jun Sese, and Takafumi Kanamori. Approximating mutual infor- mation by maximum likelihood density ratio estimation. In New challenges for feature selection in data mining and knowledge discovery, pp- 5-20, 2008
Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In 2015 IEEE Information Theory Workshop (ITW), pp. 1-5. IEEE, 2015.
Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. arXiv preprint physics/0004057, 2000.
Michael Tschannen, Josip Djolonga, Paul K. Rubenstein, Sylvain Gelly, and Mario Lucic. On mutual information maximization for representation learning. ArXiv, abs/1907. 2019.
Greg Ver Steeg and Aram Galstyan. Maximally informative hicrarchical representations of high- dimensional data. In Artificial Intelligence and Statistics, pp. 1014-1012, 2015.
Matias Vera, Pablo Piantanida, and Leonardo Rey Vega. The role of the information bottleneck in representation learning. In IEEE International Symposium on Information Theory (ISIT), pp- 1580-1584. IEEE, 2018.
Zenglin Xu, Rong Jin, Bin Shen, and Shenghuo Zhu. Nystrom approximation for sparse kernel meth- ods: Theoretical analysis and empirical evaluation. In Blai Bonet and Sven Koenig (eds.). Pro- ceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, Junuary 25-30, 2015, Austin, Texas, USA. pp. 1115--12..AAAIII RAAAI/AAAA1/// index. htap::/phh/AAAAAAAAAA1/////p/e/p
2015. URL TRLattp::/ www. aaai. org/ ocs/ 9860
Georg Zeitler, Ralf Koetter, Gerhard Bauch, and Joerg Widmer. Design of network coding functions in multihop relay networks. In 2008 5th International Symposiun Turbo Codes and Related Topics, pp. 249-254. IEEE, 2008.
12

---

Published as a conference paper at ICLR 2020
A DERIVATION OF GRADIENT ESTIMATES FOR ENTROPY
Unconditional Entropy Given that the encoder (g)..) is deterministic, our goal is to optimize the entropy H())) -B, log 9, where is short for the distribution go (5(z) of the representation w.r..
its parameters vb. We can decompose the gradient of the entropy of qy (z) as:
(2H(2) V (.. [log q(z)]
Eg()) V log qu (z)).
(19)
The second term on the right side of the equation can be calculated:
((2) /
(z)dz Vp f go(l))da O. (20)
(g(x)) V log (z)) Ey(z) V To (z)
Therefore, the gradient of the entropy of (z))) becomes
V V (2) = ta
(21)
Conditional Entropy Consider nondeterministic encoder function Eg E) where is an auxil-
iary variable with independent marginal p(c). The distribution (x(22=) is determined by and the encoder parameters sb. The auxiliary variable e introduces randomness to the encoder. First, we decompose the gradients of Conditional Entropy as following:
V H (z/x) = V / Pe (z, x) log ps (2) (z(x)dzdr
Bppx)/V / Pa (z/xx) log lo
-pp((x (V E (2)) Lo (lop((/) + p(z|x) (2g
Bx(x) V E (2)x log + / V (vPp(z)x)dh)
-Eix) E (x)x) log V / ppolh x)dhdr)
Bx(x) V Ex [ogg(()))]]
(22)
Note that z= (x, 6). such that we can apply reparameterizzation trick to the gradient estimator of conditional entropy Eq. (22).
H (z|x) Ep(x) [Epeee (mix) V
(23)
B MIGE ALGORITHM DESCRIPTION
The algorithm description of our proposed MIGE is stated in Algorithm I
Algorithm MIGE (Circumstance I)
1. Sampling:
Draw n samples from the data distribution p(v), n denotes mini-batch size, then compute the corresponding output of the encoder
(x(1), 2())..... (x(()) 2 (m)) (x,
(1)) (n)) (z)
2. Estimate the score function:
V, log qu (z())
(S z(n))
(xx.)) log qu (x()), z ( +- (x()((x(")) (1))) z(())
3. Estimate the entropy gradient:
Hz)
R Ei [Vaz(0) V, log go (2(1)]
H z) + - D V (x(()) z(1) V (xxz) log (x(() (0))
4. Estimate the MI gradient:
(xx() + (zH(z)- (x;((
13

---

Published as a conference paper at ICLR 2020
C DISCUSSION ON DIM(L)
DIM() (Hjelm et al. 2019) is the state-of-the-art unsupervised model for representaion learning, which maximizes the average MI between the high-level representation and local patches of the image, and achieve an even higher classification accuracy than supervised learning. As shown in T we apply MIGE into DIM(L) and surprisingly find there is significant performance gap to DIM(L).
To our knowledge, the principle of DIM(L) is still unclear. Tschannen al. (2019) argues that maximizing tighter bounds in DIM(L) can lead to worse results, and the success of these methods cannot attributed the properties of MI alone, and they strongly depend on the inductive bias in both the choice of feature extractor architectures and the parameterization of the employed MI estimators. For MIGE, we are investigating the behind reasons, e.g., to investigate the distributions of the patches.
Table 4: CIFAR-10 and CIFAR-100 classification accuracy (top 1) of downstream tasks compared with vanilla DIM(L).
CIFAR-10
CIFAR-100
Model
conv
fc(1024)
Y(64)
conv
fc(1024)
Y(64)
DIM(L) (JSD)
72.16%
67.99%
66.35%
41.65%
39.60%
39.66%
DIM(L) (JSD + PM)
73.25%
73.62%
66.96%
48.13% 13%
45.92%
39.6%
DIM(L) (infoNCE)
75.05%
70.68%
69.24%
44.11%
42.97%
42.74%
DIM(L) (infoNCE PM)
75.21%
75.57%
69.13%
49.74%
47.72%
41.61%
MIGE
59.72%
56.14%
54.01%
30.0%
28.96%
27.65%
14


Reviews:
[{"review_id": "SJx1qthYqB", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes the Mutual Information Gradient Estimator (MIGE) for estimating the gradient of the mutual information (MI) instead of calculating it directly in learning representation. They are using Stein's estimator following by a random projection to build a tractable approximation to the gradient of the MI. \n The MIGE is evaluated on several of unsupervised and supervised tasks, and shown improvement over prior MI estimation approaches in maximize the MI and learning features for classification.\n\nIn general, I think that the idea of estimating the gradient of the MI instead of directly calculating it  is an exciting research direction, and this paper combines a few pieces together (As mentioned in the paper, there was a work of Li & Turner, 2017 that applied Stein's estimator for implicit models).\nHowever, the experimental part of this paper is lacking. My main concern is regarding the performance on downstream tasks. Although the experiments demonstrate wins over different models in maximizing MI for CIFAR10 and CIFAR100, the only comparison for downstream tasks is for Permutation-invariant MNIST. One more concern is regarding the random projection. It is not clear what is the effect of it on the representation, and how it impacts on the gradine's estimation.  \n\nStrengths:\n+ Interesting new model for representation learning based on an estimation of the MI gradients'.\n+ Good set experiments looking at MI maximization performance.\n+A well-written and well-organized paper.\n\nWeaknesses:\n No comparison on downstream tasks for more datasets except MNIST. In the end, a key question is a final accuracy on different datasets and how to maximize the information effect on it. \nThere is no discussion about the effect of the random projection on the representation. For example, how it affects performance? How much the algorithm sensitive to this projection? What is the performance of the MINE if it combined with random projection...\n\nMinor comments:\n\n-Typos and English mistakes - there are many typos. For example -\n    In the introduction - \"Another closely related work is the the Information…\"\n    In section 2 - “In order to overcome this disadvantages\"  \n    In section  2.20  - \"In optimization, it should be achieved by maximizing the information between z and z.\"\n- There should be more detailed explanations of the experiments. For example - what is the projected dimension (for all the experiments). \n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "BylvZbsnFB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes MIGE---a novel estimator of the mutual information (MI) gradient, based on estimating the score function of an implicit distribution. To this end, the authors employ the spectral Stein gradient estimator (SSGE) and propose its scalable version based on random projections of the original input. The theoretical advantages of the method are presented using a toy experiment with correlated Gaussian random variables, where both the mutual information and its gradient can be computed analytically. In this setting, MIGE provides gradient estimates that are less biased and smoother than baselines. The method is also evaluated on two more complicated tasks: unsupervised representation learning on Cifar-10 and CIfar-100 via DeepInfoMax (DIM) and classification on MNIST with Information Bottleneck (IB), where MIGE outperforms all baselines by a significant margin.\n\nI recommend ACCEPTing this paper. This work discusses a vital problem and proposes a novel, well-motivated, principled and very performant solution; additionally, it demonstrates the broad applicability of the introduced method. While the proposed technique consists of previously known building blocks (spectral Stein gradient estimator and random projections), it is cleverly applied in a novel context of estimating MI gradients.\n\nWhile the paper is solid, I believe that it could be improved in the following ways. Firstly, I would like to see 1) more extensive and 2) larger-scale evaluations. In the DIM experiment, 1) would correspond to trying the DIM(L) approach, which maximises patch-wise MI. In fact, I strongly recommend including this experiment as it corresponds to and could improve current state-of-the-art. If it turns out that MIGE does not work well on DIM(L), then this would correspond to a serious issue with the method. In this experiment, 1) would also include providing other metrics for learned representations. It would be much more convincing to include estimates of true mutual information (e.g. InfoNCE bound evaluated with a large number of samples [1]) and showing that MIGE can attain higher values than baselines. 2) would correspond to evaluation on bigger datasets: (tiny) ImageNet and STL-10 dataset. Also, the toy experiment would benefit from a higher-dimensional setting (e.g. d=256 to d=1024), since these are often used in practice. \nSecondly, the paper is sloppily-written, which quite a few grammar and stylistic mistakes (e.g. sentence in sec 3, paragraph 2: “we assume obtain to…”, which starts with a lower-case letter and doesn’t make sense). Finally, the paper would benefit from the following clarifications: 1) explain what is the Nystr\\:om method, 2) provide either a proof or a citation for eq (19); also the error bound for SSGE should be provided for the paper to be self-contained, 3) explain the difference between q_\\psi and p_\\psi, which seem to be used interchangeably.\n\nAdditional remarks:\nSec 2.2, 2), “streamlining” is unclear\nCircumstances 2 and 3 can be quite easily derived from circumstance 1; also they are not evaluated empirically; it would be nice to have experiments for them, and they can be moved to the appendix in case of lack of space\nEq (19) while nice, seem to bear no significance for the proposed method and the rest of the paper; consider removing it\nSection 4.2 paragraph 2: “shrinking” for different layers wasn’t mentioned before, and is not immediately clear what it means; the reader needs to be intimately familiar with the DIM paper to understand.\nSection 4.3 mentions “threshold” for stein gradient estimator, which was not mentioned before. Please explain what it is.\nEquations (8-10) are just simple derivations and are not necessary; it would be enough to provide Eq (10).\nThe authors talk about MINE, which optimizes the InfoNCE bound [1], which is also used in DIM and CPC [2]. I strongly encourage the authors to cite [1] and [2] and mention them in the related works. Additionally, it would be clear if Figure 1 and related references and description used “InfoNCE” instead of “MINE” as the name of the method since InfoNCE is an estimator and MINE is just a particular implementation of the method.\n \n[1] Poole et. al., “On variational bounds of mutual information”, ICML 2019.\n[2] van den Oord et. al, “Representation Learning with Contrastive Predictive Coding“, arXiv 2018.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "BklSOaCu9S", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper argues that directly estimating the intractable mutual information (MI) for representation learning is challenging in high dimensions. Instead the authors propose to estimate the needed MI gradients directly using a score function based approach. Using some identities of MI the authors arrive at an expression for the gradient of the mutual information between input and latent representation (eq 10) and proposes to use a generalization of the reparameterization trick and spectral stein gradient descent to approximate this gradient. In Toy experiment and MNIST/CIFAR10 experiments the authors demonstrate that their method produces latent representations that are more informative than competing MI methods for downstream classification tasks. I found the approach and content of the paper interesting and the results seems encouraging.  My main concern is that I did not find the that I did not find the method and experimental section to be fully comprehensive and further lacking many details which makes it hard to compare the results with prior work. \n\n\nPros:\n1) I find the approach taken by the authors interesting and different from current MI estimation approaches. The paper convincingly motivates their approach by describing the deficiencies of current MI estimators and why targeting the gradients directly might have merits.\n2) The authors propose to use SSGD and 'generalized' reparameterization in a (well motivated) new setting.\n3) The cifar10 experiments in table 1 are encouraging and the toy experiment in 2D is illustrates nicely the deficiencies of the current MI estimators \n\nCons\n1) The experimental section is lacking many details to fully understand how and what experiments were performed and how comparable they are to prior work\n 2) The paper would benefit greatly from a thorough editing to clarify the presentation - there are many missing concepts and definitions that makes it hard to follow without intimate knowledge of related literature. \n \n \nFurther suggestions / questions \n\n1. In section 3. Please define q(z)_psi(z),  q(x,z)_psi and describe how they relate to  E_psi.\n\n2) What exactly are the contributions by the authors wrt to spectral stein gradient descent (sec 2.1) e.g. is it the scalable approach based on random projections described in sec 3 ? Further i would like some discussion on the quality of this approximation?\n\n3) Please provide some more details on the DeepInfoMax and Information bottleneck experiments e.g. How exactly did you estimate the MI gradients in these settings? how is the downstream task setup and is it identical to prior work?\n\n\n4) About writing style:\nI think it would benefit the paper if you let the reader decide for them self what adjectives should be used to describe a result. A few concrete suggestions:\n - Use remarkable/y about your own findings a bit more sparingly (used 4x). \n - Consider deleting “much” and “vast” in a sentence like: “our approach MIGE gives much more favorable gradient direction, and demonstrates more power in controlling information flows without vast loss”.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "B1gYH_Xuqr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper works out estimators for the gradient of Mutual Information (MI). The focus is on its recent popular use for representation learning. The insight the authors provide is to see encoding the representation as a ‘reparametrization’ of the data. This insight enables mathematical tools from the literature on ‘pathwise derivatives’. With gradients on the MI, one can estimate models that aim to maximize this quantity. For example in unsupervised learning one can learn representations for downstream tasks. This is shown in Table 1. Another application in supervised learning is the Information Bottleneck. This shown in Table 2.\n\nThree points for review:\n1)\nThe estimator for the gradient is shown at first on a toy task. Take a correlated Gaussian distribution and estimate gradients of the MI. The correlated Gaussian has an analytical form of MI, which makes this a useful experiment. The paper claims that this estimator ‘provides a tighter and smoother’ gradient estimate. I don’t see how this experiment and this claim tie together. Could the tightness or smoothness be quantified? It seems the MIGE has a lower variance, could empirical results or bounds on the variance be obtained? \n\nMoreover, this plot concerns random quantities, whereas we see only one realization. Both the MIGE and the MINE hold under expectation of samples from the data distribution. This is a toy example where we can sample infinitely from the data distribution. That means we can either a) plot more samples or b) obtain (empirical) error bounds on the gradient under these sampling distributions.\n\n2)\nThe major experimental result in the paper shows advantage of the gradient estimator in Transfer learning. Specifically, the authors compare against the recent DIM of Hjelm et al 2019. The authors train a quote ‘small fully connected neural network classifier’. However, the work of Hjelm trains a linear SVM on the representations. It is not clear where the increase in performance originates. Is it the improved representations (as obtained by using MIGE) or is it the change classifier? \n\n3)\nOne contribution of the paper is to make the gradient estimator work in high dimensions. To this end, the authors propose Random Projections. It is not clear how this approximation influences the results. An experiment regarding this topic would make the point clearer. Is RP used in the current experiments? Then how does this influence the results? Is the RP used for computational purposes? Then can we quantify the gain in computation?\n\nMinor comments: \n  *’In practice, we do not care about MI estimation’. Please explain further or refer to previous work.\n  *’In optimization, it should be achieved by maximizing the information between z and z.’ (Section 2). Two points\n    1. The information ‘between z and z’ is probably a typo?\n    2. How does sufficiency relate to an optimization problem? Doesn’t sufficiency mean in this context I(X;Y)=I(Z;Y)?\n  * Equation 12: In the part $\\nabla_psi (x, E_\\psi(x))$. Why do we take gradient w.r.t. x? It seems to me that the reparametrization is a function of x only via $E_\\psi(x)$. If not, then please explain what this tuple means.\n  * Table 1 has no units. How to interpret the numbers in this table?\n  * Section 4.3, authors note their experiment is ‘a little bit different’ from other related research. How and what exactly is different?\n\nTypographic comments\n  *Just below eqn17, ‘minibatche’ => ‘mini-batch’ or ‘mini batch’\n  *Section 4.2 ‘images classification’ => ‘image classification’\n  *’However A tractable density is’ => ‘However, a tractable density is’\n  *’Estimating gradients of MI than’ -> ‘Estimating gradients of MI rather than’\n  *Section 3, circumstance 1 ‘representation’ => ‘represent’\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes the Mutual Information Gradient Estimator (MIGE) for estimating the gradient of the mutual information (MI), instead of calculating it directly. To build a tractable approximation to the gradient of MI, the authors make use of Stein's estimator followed by a random projection. The authors empirically evaluate the performance on representation learning tasks and show benefits over prior MI estimation methods.
The reviewers agree that the problem is important and challenging, and that the proposed approach is novel and principled. While there were some concerns about the empirical evaluation, most of the issues were addressed during the discussion phase. I will hence recommend acceptance of this paper. We ask the authors to update the manuscript as discussed.
Decision: Accept (Poster)