Regularized Mutual Information Neural Estimation
Authors: Kwanghee Choi
Siyeong Lee
Venue:  (2021)
Keywords: Information Theory
Regularization

Abstract: With the variational lower bound of mutual information (MI), the estimation of MI can be understood as an optimization task via stochastic gradient descent. In this work, we start by showing how Mutual Information Neural Estimator (MINE) searches for the optimal function $T$ that maximizes the Donsker-Varadhan representation. With our synthetic dataset, we directly observe the neural network outputs during the optimization to investigate why MINE succeeds or fails: We discover the drifting phenomenon, where the constant term of $T$ is shifting through the optimization process, and analyze the instability caused by the interaction between the $logsumexp$ and the insufficient batch size. Next, through theoretical and experimental evidence, we propose a novel lower bound that effectively regularizes the neural network to alleviate the problems of MINE. We also introduce an averaging strategy that produces an unbiased estimate by utilizing multiple batches to mitigate the batch size limitation. Finally, we show that $L^2$ regularization achieves significant improvements in both discrete and continuous settings.

Under review as a conference paper at ICLR 2021
REGULARIZED MUTUAL INFORMATION NEURAL Es- TIMATION
Anonymous authors Paper under double-blind review
ABSTRACT
With the variational lower bound of mutual information (MI), the estimation of MI can be understood as an optimization task via stochastic gradient descent. In this work, we start by showing how Mutual Information Neural Estimator (MINE) searches for the optimal function 7 that maximizes the Donsker-Varadhan rep- resentation. With our synthetic dataset, we directly observe the neural network outputs during the optimization to investigate why MINE succeeds or fails: We discover the drifting phenomenon, where the constant term of shifting through the optimization process, and analyze the instability caused by the interaction be- tween the togsumezp and the insufficient batch size. Next, through theoretical and experimental evidence, we propose novel lower bound that effectively reg- ularizes the neural network to alleviate the problems of MINE. We also introduce averaging strategy that produces unbiased estimate by utilizing multiple batches to mitigate the batch size limitation. Finally, we show that L regulariza- tion achieves significant improvements in both discrete and continuous settings.
INTRODUCTION
Identifying relationship between two variables of interest is one of the great linchpins in mathe- matics, statistics, and machine learning (Goodfellow et al.) 2014: Ren et al.) 2015 ( [He et al., 2016]
Vaswani et al. 2017). Not surprisingly, this problem is closely tied to measuring the relationship between two variables. One of the fundamental approaches is information theory-based measure- ment, namely the estimation of mutual information (MI). Recently.Belghazzie Belghazi al. (2018) proposed neural network- based MI estimator, which is called Mutual Information Neural Estimator (MINE). Due to its differentiability and applicability. it motivated several researches such various loss functions bridging the gap between latent variables and representations (Chen et al.) 2016 Belghazi et al. 2018 Oord al. (018)) Hijelm et al. 2019), and methodologies identifying the re relationship be- tween input, output and hidden variables (Tisbyy & Zaslavsky 2015) Shwartz-Ziv & Tishby| [001]] Saxe et al. 2019). Although many have shown the computational tractability and its usefulness, many intriguing questions about the MI estimator itself remain unanswered.
How does the neural network inside MINE behave when estimating MI?
Why does MINE loss diverge in some cases? Where does the instability originate from?
Can we make a more stable estimate on small batch size settings?
Why does the value of each term in MINE loss are shifting even after the estimated MI converges? Are there any side effects of this phenomenon?
This study attempts to answer these questions by designing a synthetic dataset to interpret network outputs. Through keen observation, we dissect the Donsker- Varadhan representation (DV) one by one and conclude that the instability and the drifting are caused by the interrelationship between the stochastic gradient descent based optimization and the theoretical properties of DV. Based on these insights, we extend DV to draw out a novel lower bound for MI estimation, which mitigates the aforementioned problems, and circumvents the batch size limitation by maintaining the history of network outputs. We furthermore look into the L regularizer form of our bound in detail and analyze how various syper--praametrs impact the estimation of MI and its dynamics during the optimization process. Finally, we demonstrate that our method, called ReMINE, performs favorably against other existing estimators in multiple settings.

---

Under review as a conference paper at ICLR 2021
2
RELATED WORKS
Definition of Mutual Information Y is defined
The mutual information between two random variables and
dF XF
(X; = Dxz (Px @ = Exxy llog dIP x
(I)
where Pxy and Px @ Py are the joint and the marginal distribution, respectively. DKL is the Kullback- Leibler (KL) divergence. Without loss of generality, we consider IF, Fx and Fx Py as being distributions on a compact domain Q C IRd.
Variational Mutual Information Estimation Recent works on MI estimation focus on training a neural network to represent tight variational MI lower bound, where there are several types of representations. Although these methods are known to have statistical limitations (McAllester & Stratos 2018) 8), their versatility is widely employed nonetheless (Hjelm al. 2019 Veličković et al., 2018 Polykovskiy al. 2018 Ravanelli Bengio 2018 François-Lavel et al. 2019).
One of the most commonly used is the Donsker- Varadhan representation, which is first used in Belghazi et al. (2018) to estimate MI through neural networks.
Lemma 1. (Donsker- Varadhan representation (DV))
I(X;Y) sup Ex [T] log(ExxwFy (e")).
T:2-2R
where both the expectations Exxv [T] and Epx spy [e"'] are finite.
However, as the second term in Eq. leads to biased gradient estimates with limited number of samples, MINE uses exponential moving averages of miiii-batcees to alleviate this problem. To further improve the sampling efficiency of MINE, Lin et al. (2019) proposes DEMINE that partitions the samples into train and test sets.
Other representations based on f-measures are also proposed by Nguyen et al. (2010): Nowozin et al.) (016)) which produce unbiased estimates and hence eliminating the need for additional techniques. Lemma 2. (Nguyen, Wainwright, and Jordan representation (NWJ))
I(X;Y) sup Ex v [T] x [e"--11.
(3)
T:Q-R
where the bound is tight when T (dg(dP/dQ) + 1.
Nevertheless, MI is too big, estimators exhibit large bias or variation (McAllester Stratos 2018 Song & Ermon 2020). To balance in between, Poole al. (2019) design new estimator la that interpolates Contrastive Predictive Coding (Oord et al. 2018) and NWJ.
Yet, these methods concentrate various stabilization techniques rather than revealing the dynam- ics inside the black box. In this paper, we focus on the DV representation and provide intuitive understandings of the inner mechanisms of neural network-based estimators. Based on the analy- sis, we introduce a new regularization term for MINE, which can effectively remedy its weaknesses theoretically and practically.
3 How DOES MINE ESTIMATE?
Before going any further, we first observe the statistics network output in MINE during the optimiza- tion process using our novel synthetic dataset, and identify and analyze the following phenomena:
Drifting phenomenon (Fig. lab. where estimates of EFxy [Z] and log(EE je"')) drifts in parallel even after the MI estimate converges.
Exploding network outputs (Fig. Td), where smaller batch sizes cause the network outputs explode, but training with larger batch size reduces the variance of MI estimates (Fig 2a)
Bimodal distribution of the outputs (Fig. (2b). where the network not only classifies input samples but also clusters the network outputs as the MI estimate converges.
Based on these observations, we analyze the inner workings of MINE, and understand how batch size affects MI estimation.
2

---

Under review as a conference paper at ICLR 2021
10
J4lt surices
10
Janr manaas samples
10"
-22
ja sumples
1a mard sample
10.0
-22
1883
1000
2002
2000
3000
50% 1006 130h 200, 200
593
5000
3593
2000
2000
3000
36
58
154
275
24
(a)
(b)
(c)
(d)
Figure 1: Training T for 3000 iterations with batch size 100 [(a), (b)] and 10 [(c), (d)]. For (a)
and (c), colored lines represent the estimation for Y(); Erxy [T]. and log (Eg(Eex@ [e"]] of each batch. We also added the ideal MI for (a) and exponential moving average (EMA) of ((; with span 10 (decay rate 2/11) for (c). We can see that MINE fails to generate a meaningful estimation on batch size 10. Outputs of the neural network for each sample are shown in (b) and (d). As there are 100 samples per iteration for (b) and 10 for (d), total number of outputs 300000 and 30000, respectively. Note that the base-10 logarithmic scale is used on the y-axis for (c) and (d).
3.1 EXPERIMENT SETTINGS
Dataset. We designed one-hot discrete dataset with uniform distribution UI1 to estimate X(X:X) = log N with MINE, while casily discerning samples of joint distribution X, X from marginal distribution X @ X. Additionally, dimension, resulting in more network weights to train. In this paper, we used N 16.
one-hot representation increase the input
Network settings. We designed simple statistics network T with concatenated vector of di- mension N 32 as input. We pass the input through two fully connected layers with ReLU activation by widths: 32 - 256 1. The last layer outputs single scalar with no bias and activation. We used stochastic gradient descent (SGD) with learning rate 0.1 to optimize the statistics network.
3.2 OBSERVATIONS
We can observe the drifting phenomenon in Fig. 1a where the statistics of the network output are adrift even after the convergence of MINE loss. The analysis for this phenomenon will be covered in more detail with theoretical results in Section 4 This section will focus extensively the relationship between batch size and logsumerp, and the classifying nature of MINE loss.
Batch size limitation. MINE estimates in batch-wise manner, i.c., MINE uses samples inside single batch when estimating Erxy [T] and log x @ry (e" Consider the empirical DV
i(x;Y)= sup E(n) [Tel - log (m) To:l- Pxy
[e To
(4)
Px SF
where (e(")) is an empirical average associated with batch size n. Therefore, the variance of f(X;; Y) E increases as the batch size decreases. The observation in Fig. 2a is consistent with the batch size limitation problem (McAllester & Stratos] 2018: Song & Ermon 2020). which shows that MINE must have a batch size proportional to the exponential of true MI to control the variance of the estimation.
Exploding network outputs. We can understand the output explosion problem detail by com- paring Fig. 1b and Fig. [d] During optimization, network outputs of joint samples get increased by the first term of Eq. (4) where the inverse of the batch size is multiplied to the gradient of each network output. On the other hand, the output of marginal samples get decreased by the second term of Eq. 4 which concentrates the gradient to the maximum output. Note that the second term is dominated by the maximum network output due logsumezp. which is smooth approximation of max. As a single batch is sampled from the true underlying distribution, joint case samples may may not exist. If it exists, then the joint sample output dominates the term, and its output gets de-
3

---

Under review as a conference paper at ICLR 2021
100. 0-..00000000
1-10
200-209
400-409
1530-1509
2991-3000
200,4=279,0-022
400: 0-0.0
800,p=275.0=00.
BX. 2)777
2
5
j (joint)
i /tNon-joint)
(a)
(b)
Figure 2: (a) Training I' with different batch sizes 100, 200, 400, and 800. We show the histogram of Y(; of the final 1000 batches. (b) Histogram of network outputs of marginal samples at different iterations. The probability of the joint case to occur is N. hence the proportionally small area of joint cases. black vertical line drawn at 0 to assist visually.
creased accordingly, while other non-joint sample outputs also get slightly decreased. In summary, the second term acts as an occasional restriction for the increase of joint sample network outputs
The second term imposes a problem when the batch size not large enough. With reduced sample size, joint samples that dominate the second term are getting rare. For the case where joint sample does not exist, marginal sample network outputs decrease much faster compared to the opposite case, and joint sample network outputs are more rarely restricted; thus network outputs diverge in both directions (Fig. [d]. and the second term vibrates between two extreme values depending on whether the joint case occurred (Fig. Ic). This obviously leads to numerical instability and estimation failure.
Bimodal distribution of the outputs. We furthermore observed network outputs directly, as both averaging terms of DV can inhibit the observation of how the statistics network acts on each sample. From the neural network viewpoint, whether each sample is realized from the joint or the marginal distribution is not distinguishable for the joint cases in marginal samples. Therefore, the statistics network has no means but to return the same output value, as it can be seen in Fig. [1b] indicating that the network can only separate joint and non-joint cases. This approach provides a clue that the network is solving classification task, isolating joint samples from marginal samples, although the statistics network is only provided with samples from the joint and marginal distribution.
We observed the distribution of network outputs in detail, on the case where only the marginal samples are fed to the statistics network in Fig. 2bb It stands to reason that the network outputs follow the network output estimates a log- lilkikihhooo ratio between joint and
particular distribution, marginal distribution with an added constant (Lemma 33). Through this, we can view the estimated MI as sample average; hence Fig. resembles the Gaussian noise by Central Limit Theorem (CLT).
Let us continue by concentrating each network output. There is distinction between the log- likelihood ratio of the samples in the same class for the one-hot discrete dataset: j for the joint case and J for the non-joint case. This explains the classifying nature of the statistics network. and there have to be exactly two clusters in Fig. 2b. Also, as becomes -00, exp()) nears O, and exp(j) is few magnitudes bigger than exp(j) (see Fig. 2b). As mentioned above, few joint cases dominate the second term, so the second term becomes inherently noisier than the first term. Note that the effectiveness of conventional methods, such as applying exponential moving average to the second term (Belghazi et al. 2018) clipping the network output values to restrict the magnitude of network outputs (Song & Ermon 2020) can also be understood with the analysis above.
In addition, cannot interpret the network outputs directly the log- likelihodd ratio due to un- regularized outputs, or the drifting problem. We will look into this fundamental limitation of MINE more detail in the next section.
Loosely speaking, the first term slowly increases lot of joint samples network outputs, in contrary to the second term which quickly decreases few joint sample network outputs.
4

---

Under review as a conference paper at ICLR 2021
4 THE PROPOSED METHOD: REMINE
We look into the drifting problem of Fig. la in detail to introduce a novel loss with regularizer that constrains the magnitude of network outputs and enables the use of network outputs from multiple batches. All proofs are provided in Appendix.
First, let concentrate the optimal function T", which can be directly drawn from the DV representation. We start with the results of Belghazi et al. (2018).
Lemma 3. (Optimal Function T*). Let IP and Q be distributions S2 C Rd. For some constant d3 + +C* such that DKL(PIIQ) = Ep[Ta]
C E R, there exists an optimal function T = log di log log(EElee"))
Note that Mukherjee et al. (2020) directly utilize this fact to model the statistics network. Let us extend the result above, and show that C* can be any real number and 7 still be optimal.
dP Lemma 4. (Family of Optimal Functions). For any constant C E IR, T = log बमं + C satisfies DRL. (KI.(P|) Ep[T] loooooEg log(Eele")),
This explains the drifting phenomenon we encountered Fig. la C* is drifting as there are no penalties different C's. Drifting be stopped by freezing the network weights (Lin et al.) 2019). but is there a direct way to regularize C' so that the neural network can concentrate on finding single solution, rather than family of solutions?
Theorem 5, (ReMINE Loss Function) Let d be a distance function on R. For any constant C' E R and function T : : R,
DKL (PI = sup Ep log(BElle") d(log(Bale")), C"), T:2-
Note that for the optimal T", Ex (T") T(X;Y) + C' and Rog(Pxxer le (e = C². Based on
Theorem 5 we propose a novel loss function by adding a new term d(log [Eg @ [eT C*) that
regularizes the drifting of C** The details of the ReMINE algorithm is follows.
Algorithm 1: ReMINE
0 < Initialize network parameters, K < Moving average window size, i +-
repeat
Draw samples from the joint distribution: (x(", y .y(") (x)
... (1) ~ Pxy
Draw M samples from the marginal distribution: (x) N5 ).... (x) (r) {((M) ))
Px &y
Evaluate the lower-bound:
Bax + Σ (x( (Bx)BB + log( 1 ΣM Bota(m)) s(m)) s(mn))
v(0) + Ex Exxery diR BFy, C)
Update the statistics network parameters:
0 Vov(() Estimate MI based on the last window W [max((0, (0, i K + 1), min K, i)] of size K: (x x()) log( M.| Ewen ΣM 7e y(m))
i(X;Y)= = .. EWEW Σ (x
Next iteration: + it1
until convergence;
ReMINE differs from other MINE-like methods that rely on single batch estimates, as ReMINE can utilize all the previous network outputs after the convergence of i(X; Y) to establish final esti- mate. To demonstrate why these MINE-like methods cannot use the network outputs from multiple batches, first describe two basic averaging strategies and then show that both methods produce biased estimate when the statistics network T' is drifting.
Theorem 6. (Éstimation Bias caused by Drifting) The two averaging strategies below produce biased MI estimate when the drifting problem occurs.
1. Macro-averaginn (similar to that of Poole al. (2019)): Establish single estimate through the average of estimated MI from each batch.
2. Micro- aveeraiiv (our method): Calculate DV representation using the average of the each individual network outpuls.
5

---

Under review as a conference paper at ICLR 2021
RedliiE 10pt mene 10000-10
ReMIRE 20c.AAA. Window-10
Ret/INE 10ptmmetr-asee W0500-10000
HIRE 2op.mmrr--s5 Wintow
Sunf
(d)
Figure 3: (a) Training T with the same settings Fig. 1d with ReMINE. We can see that ReMINE successfully avoids both drifting and network output explosion problem. (b) Same setting as (a), but used the optimizer from Section 5.2 We suspect Adam shows better performance it accumulates previous gradients. (c) Same setting as (b) except we used micro-averaging strategy with sliding window of size 1000. (d) Training T with the same settings as Fig. Id but used the same optimizer from (b), Outputs are more stabilized compared to Fig. [d. but still fails to make stable estimate.
EXPERIMENTAL RESULTS OF L REGULARIZATION
Many choices exist for the distance function when implementing ReMINE. Here, we choose d(æz, C) X(z - C)² and explain the rationale behind it. We also explore different choices of byperparameters C and X. We show that ReMINE blocks the drifting effect by restricting log( EE s [e=)) estimates to be C. and visualize the loss surface of ReMINE for different choices of As. Finally, we show that ReMINE achieves better performance than state-of-the-ar methods in the continuous domain, and shows comparable performance on self-consistency tests
5.1 EFFECTIVENESS OF L REGULARIZATION
For the sake of brevity, let batch size B, step size 7/ f = E!") pxy [T). = log E (x)
[eT and
Py &Py
0. Then, vi(X;Y) Vf Vg V Vf Vg(1+ 2\g). As previously discussed, f increases each B joint sample outputs with (V/B. On the other hand, g is the approximation =
of the maximum marginal sample output, the gradient becomes close to >01 2Xg) VT, for the maximum output. We can break down the dynamics of into two parts, depending on whether joint samples exist in g. If not, maximum marginal sample output gets reduced with step size (1++2g)), which is, unlike MINE, adaptively adjusted by the size of g. Hence, ReMINE regularizes the maximum output be centered around - 2tt preventing the network outputs from diverging to -c.. If any joint sample exists, its network output will be big enough to dominate g. Step size is also 711 + 2Ag). which regularizes the joint sample network outputs more strongly as it increases. This, too, helps to avoid the output explosion in the +o side, as shown in Fig. 3
Impact of Fig. 4a shows the impact of changing C on MI estimation, with the same settings as Fig. la We observed that the newly added regularizer penalizes the log(Epx (eT]) term to converge towards C' as expected, without losing the ability to estimate MI.
Impact of A. As we observed in Section 3.2 the network outputs of joint and non-joint cases converge to j and J. respectively. Using this, we visualize the effect of the proposed method by drawing the loss surface of MINE and ReMINE for the one-hot discrete dataset in Fig 5
LMINE = Exxy [T] log log(Exx oFy [eT]) = j - log(ped + - -))e))
(5)
LRGMINE Er [T] log(Er, &fy [e=]) d(log ( @ry (e?T) C)
(6)
= - log(pe) + - p))) - A(lo((pe) + - C)
(7)
We again observe the drifting phenomena, as the loss surface has a plateau of equally palatable solutions. Regularization term successfully warps the loss surface, so that it has a single solution. As À increases, the loss surface becomes steeper, resulting in sporadic spikes for each gradient step.
We release our code in

---

Under review as a conference paper at ICLR 2021
Co.2
C=0
C-2
10
(a)
-5
(EA,
-10
1-10
191-200
391
1491-1500
[(jiit)
(b)
(Noo--ooint)
Ideal MI
5 0
5
0
Figure 4: (a) Applying ReMINE for 1500 iterations, with different Cs and X 0.1. (b) Histogram of network outputs for the marginal samples at different iterations. We set C = O and À = 0.1. exp(i) and exp(j) converges to N and 0, respectively. These are the likelihood ratios between joint and marginal distribution for joint and non-joint cases. We can now directly interpret the network outputs, and E() directly converges to ideal MI log N, thanks the regularization of C.
0.01)
IMINE/A
XX. Y)
ttertion
3.00
3000
2500
2000
1.13
1500
1000
-.26
-10
Figure 5: Comparing the loss surface for varying lambda. Per each batch, we averaged the network outputs of joint samples to estimate j, and non-joint cases of marginal samples to estimate j.
5.2 REMINE IN THE CONTINUOUS DOMAIN
20-D Gaussian dataset. We sampled (x, from dddimennsioal correlated Gaussian dataset where X ~ (O,, 1a) and Y N(pX, (1 p²) 12) given the correlation parameter 0 S < 1, which is taken from Belghazi et al. (2018) to test ReMINE on continuous random variables. The true MI for the dataset is I(X,
log(1 p²). In these experiments, set batch size to 64.
Network settings. We consider joint architecture, which concatenates the inputs (x, y). and then passes through three fully connected layers with ReLU activation (excluding the output layer) by widths 40 256 256 1, same the network used in Poole et al. (2019). We used Adam
optimizer with learning rate 5 x 10 31 = 0.9 and 3g = 0.999.
Comparison state-of-the-arls. As mentioned in Fig, 4b can remove the log(Ex @ry [e" term by choosing C = O. As discussed in Section 3.2 the second term is inherently noisy. Hence, using all the terms in ReMINE only in optimization and removing the second term in estimation can effectively reduce noise. We call this trick ReMINE-J.
To verify the quality of lower bounds, we compare ReMINE and ReMINE-J to InfoNCE (Oord al. 2018), JS (Hjelm et al.) 2019; Poole et al. 2019), MINE (Belghazi et al. 2018). NWJ (Nguyen et al., 2010 SMILE (Song & Ermon 2020). SMILE+JS (which estimates with SMILE, and optimizes with JS), TUBA (Barber & Agakov| 2004 Poole et al. 2019) and I, (Poole et al. 2019). To make a fair comparison, ReMINE also uses the macro-averaging strategy. the same as the other methods. better estimation performance with less variance than others, as shown in Fig. 6 Exact values for bias, variance, and mean square error to the true MI for each estimator shown in Appendix.
Our methods show comparable

---

Under review as a conference paper at ICLR 2021
94IF cr
20
JUDK
2888
2,000
SPILEA
INHRLE Jc-0,
TAMRR 1c-0 A-2)
1000 Z:000
2000E
30000 13002 20000
1000
1000
22.00
20302
L000 1,500 20000
30000 2354 73000
Figure 6: Estimation performance on 20-D Gaussian. Similar to Poole et al. (2019), we increase every 4000 iterations. The estimated MI (light) and smoothed estimation with exponential moving average (dark) are plotted for each methods, and theoretical bounds are plotted by dotted lines.
Independence (Type 31
Data Processing IType 2)
Addizivity (Type
SMILE 17 121
uned
Figure 7: Self-consistency tests CIFAR-10 Krizhevsky et al. 2009 We report the average result of 10 repeated runs. Dotted lines indicate theoretical bounds for type 1, and ideal ratio for type 2, 3.
5.3 REMINE THE IMAGE DOMAIN
Experiment settings. Song & Ermon 2020 introduced the self-consistency tests image datasets to verify whether the estimates follow the basic properties of MI. For our experiments, used the same network and optimizer, types of tests, and the number of epochs of & Ermon 2020) with batch size 16. We did not use our micro-averaging strategy for a fair comparison.
Comparison self-consistency tests. To compare the stability of DV-based estimators, we ducted self-consistency tests on ReMINE, MINE, SMILE, and SMILE+JS. For type 1, every esti- mator successfully returns values between the theoretical bound with an increasing trend. For type 2, only ReMINE and SMILE+JS estimates are close to the ideal value. For type 3, none of the esti- mators worked well. However, ReMINE shows smaller variance compared to MINE and has similar stability to SMILE+JS.
6 CONCLUSION
In this paper, we studied how the neural network inside MINE handles the MI estimation problem. We delved into the drifting problem. where two terms of DV continue to fluctuate together even after the MI estimate converges, and the explosion problem, where the network outputs become unstable due to properties of the second term in DV when batch size is small. Based on the analysis, we penalized the objective function for obtaining unique solution by using I2² regularization. De- spite the simplicity, the proposed loss and the micro-averaging strategy mitigate drifting, exploding, and batch size limitation problems. Further, ReMINE enables us to directly interpret the network output values as the log- likelihodoo ratio of joint and marginal distribution probability and performs favorably against sfae-of-ttee-al the-hhh-r art methods. However, further investigation needs to be done on the impact optimizers on the batch size limitation, and why DV-based estimators fail in some of the self consistency tests.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
David Barber and Felix V Agakov. Information maximization in noisy channels: A variational approach. In Advances Neural Information Processing Systems, pp. 201-208, 2004.
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mutual information neural estimation. In International Conference on Machine Learning, pp. 531-540, 2018.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in neural information processing systems, pp- 2172-2180, 2016.
Vincent François- Yoshua Bengio, Doina Precup, and Joelle Pineau. Combined reinforce- ment learning via abstract representations. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pp. 3582-3589, 2019.
Weihao Gao, Sreeram Kanman, Sewoong Oh, and Pramod Viswanath. Estimating mutual informa- tion for discrete-comtinuuus mixtures. Advances in neural information processing systems, pp- 5986-5997, 2017.
Ian Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, pp. 2672-2680, 2014.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Devon Hjelm, Alex Fedorov, Samuel Marchildn,, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estima- tion and maximization. In International Conference on Learning Representations, 2019, URL https://opennevieet net Leg.eenreeeeee...reeooooooooooaa
Jiantao Jiao, Weihao Gao, and Yanjun Han. The nearest neighbor information estimator is adaptively near minimax rate-optimal. In Advances in neural information processing systems, pp. 3156- 3167, 2018.
Alexander Kraskov, Harald Stögbauer, and Peter Grassberger. Estimating mutual information. Phys- ical review E, 69(6):066138 2004.
Alex Krizhevsky et Learning multiple layers of features from tiny images. 2009.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEFE, 86(11) (6(11):2277-2222 1998.
Xiao Lin, Indranil Sur, Samuel A Nastase, Ajay Divakaran, Uri Hasson, and Mohamed R Amer. Data-efficient mutual information neural estimator. arXiv preprint 19X.:01905.03319, 2019.
David McAllester and Karl Stratos. Formal limitations on the measurement of mutual information. arXiv preprint arXiv: 1.04211, 2018.
Sudipto Mukherjee, Himanshu Asnani, and Sreeram Kannan. Cemi: Classifier based conditional mutual information estimation. In Uncertainty in Artificial Intelligence, pp. 1083-1093. PMLR, 2020.
XuanLong Nguyen, Martin J Wainwright, and Michael Jordan. Estimating divergence functionals and the likelihood ratio by convex risk minimization. IEEE Transactions on Information Theory, 56(11):5847-5811 2010.
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-gan: Training generative neural samplers using variational divergence minimization. In Advances in neural information processing systems, pp. 271-279, 2016.

---

Under review as a conference paper at ICLR 2021
Aaron den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint arXiv:1807.037444 2018.
Daniil Polykovskiy, Alexander Zhebrak, Dmitry Vetrov, Yan Ivanenkov, Vladimir Aladinskiy, Polina Mamoshina, Marine Bozdaganyan, Alexander Aliper, Alex Zhavoronkov, and Artur Kadurin. En- tangled conditional adversarial autoencoder for de novo drug discovery. Molecular pharmaceu- tics, 15(10):439 4405, 2018.
Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. In International Conference on Machine Learning, pp. 5171- 5180, 2019.
Mirco Ravanelli and Yoshua Bengio. Learning speaker representations with mutual information. arXiv preprint arXiv: 10Xix:8112.0027/1 2018.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnm: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems, pp. 91-99, 2015.
Andrew Saxe, Yamini Bansal, Joel Dapello, Madhu Advani, Artemy Kolchinsky, Brendan D Tracey, and David D Cox. On the information bottleneck theory of deep learning. Journal of Statistical Mechanics: Theory and Experiment, 2019(12):124220, 2019.
Ravid Shwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via informa- tion. arXiv preprint arXiv: 7707000000000 2017.
Jiaming Song and Stefano Ermon. Understanding the limitations of variational mutual information estimators. In International Conference on Learning Representations, 2020. URL https:// openreview n forum?id=Bi X62TNEE
Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In 2015 IEEE Information Theory Workshop (ITW), pp. 1-5. IEEE, 2015.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Petar Veličković, William Fedus, William Hamilton, Pietro Liò, Yoshua Bengio, and R Devon Hjelm. Deep graph infomax. arXiv preprint /809 10341, 2018.
10

---

Under review as a conference paper at ICLR 2021
7 APPENDIX: PROOFS
7.1 PROOF OF FAMILY OF OPTIMAL FUNCTIONS
Theorem. For any constant C' E R. T
log
de C satisfies D (IP ( Ep Ep log (Bg (Ba[e")) [e")).
Proof. Suppose that T log dP in the manuscript. dq Therefore,
C. We can write the function T -T" + C by Lemma
Ep[T] = Ep[T" CC
0*
and
log(EElle") = log (Bale" -oge -0+0))
log((e--0' Bale"")
CC' + log(EEll"
Since Ep[T] log log(Ecle")) = Ep[7"* -
the function T also optimal.
7.2 PROOF OF REMINE LOSS FUNCTION
Theorem. Let d be a distance function on R. For any constant C" E R and function T Ω IR,
QPI sup Ep log(Eele")) C((
T:2-
Proof. i) For any T,
Ee [2] log(BBlee)) d(oog(oole))) C²) Bp[T] log(Lell"").
Therefore, supr:o-- Ep [T] (og(Eel d(log EB [e 1), C²) < (KI((1(()
+ C such that ii) By the theorem above, there exists T = loog dF dQ
(kI( = Ep [T log (Egle"")
and
log(Eale?")) log (Bgle dP
d 1) = log( / o dIP
dQ2) C².
dQ
Therefore,
sup Eg[T] (og(Ealee)) d(log(Egle))) 1), 2 Ep|T" - log(Eqle" d(log(Eglee") 1), C²)
7:2+3
(Pl((P
Combining and ii) finishes the proof.
11

---

Under review as a conference paper at ICLR 2021
7.3 PROOF OF ESTIMATION BIAS CAUSED BY DRIFTING
Theorem. (Estimation Bias caused by Drifting) The two averaging strategies below produce biased MI estimate when the drifting problem occurs.
1. Macro-averaging (similar to that of Poole et al. (2019)): Establish single estimate through the average of estimated MI from each batch.
2. Micro-averaging (our method): Calculate DV representation using the average of the each individual network outpuls.
Proof. Let the outputs of ith batch, jth sample inside the batch as T.g) T( joint and marginal case respectively, and the output without drifting as Tyj. and drifting constant for each batch C.. Then, Tij T + C..
When the number of batch is B and each batch size is N,
1. Macro averaging:
1 3 E (J) log( N ; exp |
(8)
B
1j
1
(J+)
+ CC) log( 3S exp(T C())
(9)
B
1
El
Ej (1)) (1 + C.) log(
N exp CiE, exp T. (
(10)
B
1
2 p( - bog(exp(-CC: N exp وننب exp 1 [Ma))]]
(11)
B
N
(((+)) loog( 1
Dg exp I. ((tl
E.l
(12)
B
N
N
1
Eillog( En exp T. (
(13)
NB
B
Mi (+)) log(
1
2ij exp (
(14)
NB
NB
2. Micro averaging:
1
(()) T())(()) ij
log( Fnj exp (oai)
Eig
(15)
NB
NB
1
Eig (J+)
(T((TT Ct) logg NB
Eij exp (T(M + +c)))
(16)
NB
1
Eig (((++) -log(( Eij exp (T( + c:)
(17)
NB
NB
1
Eig T(3o) logg(
Eij exp ij )1(
(18)
NB
NB
12

---

Under review as a conference paper at ICLR 2021
8 APPENDIX: ADDITIONAL EXPLANATIONS
As the N-dimensional one-hot discrete dataset is uniform, easily calculate the likelihood ratio of joint and non-joint case samples. For all the possible samples, Fx @ Y = 1/N2, as they are total of N²2 Also, for joint case samples, Pxy = 1/N, and IP Xx 0 for non-joint case samples. Hence, the likelihood ratio for the joint cases is N, and non-joint cases is O. These are consistent with the experimental results, where converges to log N, and j keeps decreasing. Nonetheless, as exp(j) gets closer to zero, the second term of ReMINE loss has lesser influence; hence the decreasing speed of gets slowed down to a halt it reaches 2A 5.
Additional explanations for Fig. 4b
We can explain the same result from the perspective of j and j. As we observed in Section 3.2 the network output values of joint and non-joint cases converge to j and respectively. Since the dataset is uniform, the probability p of joint cases appearing from the marginal samples is N. Therefore, can analyze the value of and j after convergence as follows: iteration i - oc.
Ex (T()) = j - (x;;)+++++ C' log N + C
(19)
N
(e ())) = log(pe) + (1 - -))pp)))-- = log(
(20)
where T(i) is the statistics network at iteration We combine Eq.
19 and Eq. 20
1 N
log N+C
N - J +
and
(21)
N
0.
(22)
In summary, j will converge to log N + C, and el to 0, as shown in Fig. 4b Note that j and j serves as a -ack-of-the tof-theeeeeeopee calculation for us to estimate network outputs easily on discrete settings.
What happens if the batch size is small? When the batch size is and C' 0, the loss function of ReMINE changes its characteristics as follows.
Joint case occurs. As the samples are indistinguishable.
L j - loogee e -d]j, 0) =>j²
(23)
which is maximized when j = 0. Non-joint case occurs.
C L=j log 2 + d(log e. e,0) = - The latter quadratic term of j is maximized when j
(24)
If the statistics network succeeds to converge on both cases for our one-hot discrete dataset,
f(x;Y)
F (n)
Pxy [To] log E (n)
[o 0 log(pe (1 p)e 22) - log (25)
when À + +0. As = N. I(xY)) + log
Intuitively, on smaller batch sizes, joint cases cannot occur in marginal samples, mentioned in Section Hence, Erxv [7] and log(EEs [e=]) behave differently compared to the larger hatch size. The regularizer term penalizes both terms in different ways. Joint cases in marginal samples can contribute only with Eq. (23). so Epxy [T] + O. Moreover, as X gets smaller, log o (e"l) gets regularized less so that it can converge to iix:; ((XYY) Y). In contrast, since MINE has no regular- ization term, namely À O, there is no way for the joint case in marginal samples to influence T, hence failing to estimate MI as shown in Fig. 8b)
Impact of À with batch size. We inspect the relationship between batch size and À in detail. Fig. shows that imposing regularization reduces noise large batch size domain. However, small batch size domain, log(Ep s [e" cannot have nonzero value, hence failing to estimate MI value. The effect of the ReMINE loss in two different domains gets mixed in between.
13

---

Under review as a conference paper at ICLR 2021
10a
joint samples Non ieint cases in marginal samplcs
joint somples Non joint cases marginal samples
3.0
10:
10
10
20
100
-109
10
-10
10
-10-
-10
-10
101 10 10-2
1000
2030
3000
4000
5000
0
1000
2000
3000 4000
5000
10a
(b)
(c)
Figure 8: (a) Comparing the joint and non-joint case outputs of ReMINE with X = 0.01 when batch size for 5000 iterations. We can see j and joint and non-joint case outputs of MINE for 5000 iterations. As the statistics network struggles diverge into two different values, becomes numerically unstable, hence failing in the middle of the training. (c) Comparing the estimation performance of different Às on varying batch sizes 1,2, The dotted line represents the true MI.
전 50. (b) Comparing the
Visualizing network outputs on 1-D Gaussian. The dataset forbids to label joint and non-joint samples explicitly, so we visualized the network outputs on 2-D plane. We used the same experiment settings Section 3.I only changing the input dimension outputs of the overlapping region remain near O, which indicates that the likelihood is equal between joint and marginal distribution. Other regions are separated by the sign of their outputs. Positive network output means that joint distribution is more probable than marginal distribution to sample that data point, and vice versa.
We can in Fig. thtt the network
-2-2
-2
-2
4-4
4
Figure 9: Visualizing network outputs on the 1-D correlated Gaussian dataset. Each axis represents a paired value x and y of each sample, and the color represents network outputs for (a) marginal and (b) joint samples.
14

---

Under review as a conference paper at ICLR 2021
APPENDIX: ADDITIONAL EXPERIMENTAL DETAILS
Additional experiments outputs on 20-D Gaussian. For quantitative comparison with other ap- proaches on the 20-D correlated Gaussian dataset, show the bias, variance and mean squared ror (MSE) of the neural network-baa and nearest neighbor-based MI estimators in Fig. [0] Fig. []
and Table T We omitted values which are more than 100 in Table IT, We additionally show results from KL (Jiao et al. 20118) Mixed KSG (Gao et al. 2017). CCMI (Mukherjee et al. [2020), TNCE
(Oord et a.. 2018 Poole
2019) and ReMINE- LI (our method with LI regularization).
Both ReMINE and ReMINE-J shows comparable or better performance compared to other methods. Note that 1.1 regularizer also suffers from the explosion problem, as the gradient is not adaptively adjusted by the magnitude of network outputs, as discussed in Section 5.1
Bias
Variance
MSE
2
-2-
1-
1-
3
InfoNCE
MINE
SMILE (T 10)
TUBA
ReMINE (C=0, A
NWJ
SMILE+JS (T= 10)
1g [a 0.01)
ReMINE-J (C=0, =
Figure 10: Bias, variance, MSE of estimators 20-D correlated Gaussian dataset
Mixad XS0
SeMINF-LL A=:
Figure 11: Estimation performance on 20-D correlated Gaussian dataset for additional estimators
15

---

Under review as a conference paper at ICLR 2021
MI
6
8
CCMI
-0.65
-1224
-2.55
-4.18
-6.00
InfoNCE
-0.30
-1.00
-2.31
-4.00
-5.89
JS
-0.57
-0.72
-0.94
-1.21
-161
KL
-4.35
-5.25
-6.46
-7.52
-8.83
MINE
-0.15
-0.24
0.32
33.81
Bias
Mixed KSG
-1.49
-2.99
-4.51
-6.08
-7.64
NWJ
-0.20
-0.36
-0.74
-1.47
-2.70
SMILE (T = 10)
-0.15
-0.26
SMILE+JS 10)
-0.19
-0.26
-0.34
-0.51
-0.74
TNCE
-0.19
-0.93
-2.26
-3.98
-5.88
TUBA
-0.17
-0.33
-0.74
-1.42
-285
Io (a = 0.01)
-0.16
-0.28
-0.53
-1.01
-1.91
ReMINE (C=0, 1)
-0.16
-0.27
-0.49
-0.62
-0.51
ReMINE-LI (C=0, X = 1)
-0.25
-0.27
-0.33
0.78
40.44
ReMINE-J (C=0, À 1)
-0.16
-0.27
-0.49
-0.65
-0.56
CCMI
0.24
0.03
0.02
0.01
0.00
InfoNCE
0.09
0.03
0.02
0.01
0.00
IS
0.12
0.10
0.15
0.22
0.57
KL
0.00
0.00
0.00
0.00
0.00
MINE
0.12
0.27
7.16
Variance
Mixed KSG
0.00
0.00
0.00
0.00
0.00
NWJ
0.15
0.26
1.54
6.85
23.42
SMILE (T = 10)
0.12
0.26
SMILE+JS (T = 10)
0.11
0.12
0.27
0.53
1.19
TNCE
0.04
0.03
0,02
0.01
0.00
TUBA
0.12
0.21
2.06
21.97
35.54
Ia (a 0.01)
0.12
0.16
0.31
0.38
0.38
ReMINE (C=0, x= 1)
0.13
0.15
0.34
0.69
1.39
ReMINE-LI (C=0, 1)
0.20
0.14
0.48
4.44
ReMINE-J (C=0,
0.12
0.13
0.21
0.29
0.59
CCMI
0.67
1.57
6.54
17.51
35.95
InfoNCE
0.17
1.03
5.33
16.02
34.68
IS
0.45
0.62
1.03
1.67
3.17
KL
18.92
27.54
41.71
56.56
77.94
MINE
0.14
0.33
7.27
MSE
Mixed KSG
2.22
8.93
20.34
36.98
58.37
NWJ
0.19
0.39
2.09
9.01
30.68
SMILL (T 10)
0.14
0.33
SMILE+JS (7 = 10)
0.15
0.19
0.39
0.79
1.74
TNCE
0.08
0.89
5.14
15.82
34.55
TUBA
0.14
0.32
2.60
23.97
43.67
La (a 0.01)
0.14
0.23
0.60
1.39
4.03
ReMINE (C=0, 1)
0.15
0.22
0.58
1.08
1.65
ReMINE-LI (C=0, A= 1)
0.27
0.21
0.59
5.06
ReMINE-J (C=0, x=
0.15
0.20
0.45
0.72
0.90
Table 1: Bias, variance and MSE of estimators on 20-D correlated Gaussian dataset
16

---

Under review as a conference paper at ICLR 2021
Additional experiments the self consistency test. We report the performance of other varia- tional bound methods (JS, La. InfoNCE) in Fig. [2] La and JS often result in unstable MI estimates, shown in the type 2 experiment. On the other hand, InfoNCE estimates MI quite reliably but also fails for type 3.
Independence (Type 1)
Data Pracessing (lype 2)
Additivity Clype 3]
IntoNCE
1.0
2.
kind
0.5
-0.5
0.5
-1.9
0.0
Figure 12: Self-consistency tests of JS, Ia. and InfoNCE on CIFAR-10. We use the average result of 10 repeated runs. Dotted lines indicate theoretical bounds for type 1, and ideal ratio for type 2, 3.
To observe on a different dataset setting, we also used MNIST (eeCun et al. 1998). As shown in tests yielded similar results.
Fig. 13
independence (Type
Data (Type
Additivity (Type
HINE WHINE
15-
1.5
GHILT
0.5
Rade
idcal
0.0
0.0
-1.0
10
20
-1.0
10
20
30
20
Ro usid
Plwws used
Piera saed
Incependence 1Type 1
Data Processing (7ype
Add tivity 1"ype 31
1.5
1.5
Nenortago
&
atoha
ideal
0.9
0.0
-0.5
-0.5
10
20
30
20
30
nown used
Rown
Figure 13: Self-consistency tests on MNIST.
To observe on different statistics network, we used modified ResNet18 (He al. 2016) that outputs single scalar. As shown in Fig. [4, SMILE has become more unstable, but there are significant differences in other variational bounds. This experiment shows that the network size has small impact the validity of this test on CIFAR-10.
Independence (Type
Data Processing (Type
Add tivity IType 3)
MINE: REMIKE JC+C, A-0.111
1.5
2.5
2.0
Patro
STALE E1 101 ideal
0.0
10
0.5
10
20
30
20
20
20 Rows used
Pows used
Rows used
Figure 14: Self-consistency tests on CIFAR-10 with ResNet18.
17

---

Under review as a conference paper at ICLR 2021
Effectiveness on the Conditional Mutual Information Estimation Task We compare the per- formance of various estimators on the conditional MI (CMI) estimation task. To set the bascline, we chose CCMI (Mukherjee et al. 2020), MINE (Belghazi al. 2018). and KSG estimator (Kraskov et al. 2004). The Experiment is performed under the Model setting in Mukherjee et al. (2020). We refer the supplementary of Mukherjee al. (2020) for
settings such network structures and optimizer parameters. We only changed the objective funcctin of MINE to test our method. As shown Fig. [5] ReMINE can reach comparable performance without changing the form to classification loss. Also, ReMINE produces stable estimates compared to MINE.
2.0
26
2.0
05
03
60
60
Figure 15: CMI estimation results on Model (n = 20000). The dotted line represents the ideal CMI value. We rerun the same experiment 10 times, and plot mean (dark) and standard deviation (light).
18


Reviews:
[{"review_id": "qdhcOU5Y9C", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The work studies a neural-network based estimator, referred to as MINE, for approximating the mutual information between two variables.  By designing a synthetic dataset, the work studies properties of MINE and, based on these findings, proposes a new method incorporating regularization, called ReMINE, that they empirically demonstrate has nice performance. \n\nStrengths:\n-- The proposed ReMINE algorithm combats drifting of the two terms in MINE's approximation of the DV representation through regularization that forces the network to find a single solution (as opposed to a family of solutions).\n-- The proposed method is novel and performs well compared to state-of-the-art.\n\nWeaknesses:\n-- I found the writing to be difficult to understand and follow. For example, the paper is not self-contained: the authors use terms like \"statistical network\" without providing definitions, and begin simulations without introducing MINE or giving a high-level discussion of how it estimates the DV representation, so it would be very difficult to read this paper without having first read the original MINE paper. \n-- The paper lacks intuition. For example, there is little to no discussion of why the synthetic dataset proposed is a good choice for studying the underlying properties of MINE.\n\nSome additional typos/comments:\n\nPg 1: \"...the value of each term in MINE loss IS shifting even...\"\nPg 2: I believe equation (3) should be a lower bound not an equality.\nPg 2: \"...where estimates of  __ and __ DRIFT in parallel...\"\nPg 4: mu and sigma in Figure 2 were never defined.\nWhen printed, the black-and-white figures are too small to read and interpret.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Y1gfqVSXPL_", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper attempts to answer the four questions raised from the mutual information estimator. To this end, this paper investigates why the MINE succeeds or fails during the optimization on a synthetic dataset. Based on the observations and discussions, the paper then proposes a novel lower bound to regularize the neural networks and alleviate the problems of MINE.\n\nOverall, the paper is easy to follow and new insights have been brought for the MI estimator and the downstream tasks.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "AunbqRjBJpZ", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Response to authors: After reading the authors' response, I have decided to maintain my original rating. The authors have not adequately addressed my main concerns.\n\nNovelty: The work here, as indicated by the authors, is largely an incremental improvement over an existing work MINE. The authors' response did not alleviate this concern and in fact reinforced it.\n\nCitations and comparisons to other work: The authors did not agree to even include citations to important literature in this area. This should have been a bare minimum and it is a mistake for variational approaches to ignore these works which have theoretical guarantees that many variational approaches do not have. Comparisons to other methods should also have been included. The methods the authors did compare to have weak (or no) theoretical guarantees for higher dimensions.\n\nTheoretical work: The authors simply pointed to the theoretical work for MINE. However, the theoretical work in MINE is also very weak and only focuses on estimation consistency and not convergence rates (i.e. the statistical bias and variance of the estimator). More theoretical work is needed in this area to justify the use of these estimators over others.\n\nSome responses to other comments that may help the authors with further revisions:\n\n(1) The presentation of MINE should occur in the main paper as this is crucial for understanding the paper.\n\n(2) This was not clear. Perhaps the authors could include similar pointers in the paper with each of these issues.\n\n(3) The bias I'm referring to here is the actual statistical bias of the estimator. From Theorem 6, it seems that the drift problem does seem to create some bias but it would be useful to quantify that, which could then lead to a bias correction approach.\n\n(7) The way this is currently worded, it sounds like you are saying that training with a larger batch size is bad. This part should be clarified to avoid this. \n\nOriginal Review:\n\nThis paper presents a modified version of a neural network-based MI estimator. They investigate a few of the issues of this specific estimator and propose a regularization to help with one of them. MI estimation is an important and difficult topic. Improvements in this area are of definite interest.\n\nPros:\nThe paper appears to be technically correct. The experiments are somewhat supportive of including the regularization, especially when the MI is higher which is a known issue with some MI estimators.\n\nCons:\nThere are some interesting ideas here but the paper feels unpolished. The presentation of the ideas is somewhat unconventional. Several issues with the MINE estimator are presented and then two of them are discarded in favor of a focus on one of them. The paper could benefit from a bit more focus in this regard. In the end, the authors really only propose a small modification to the MINE estimator to counter the supposed drifting problem and do some experiments showing some improvement. But it's not clear how much of a problem this drift really is. The authors show that it causes a bias but they do not present how much bias it adds.   \n\nIn addition, the authors are severely neglecting some of the non-neural network state of the art MI estimators in their citations and comparisons (see the references below for some examples, which all have strong theoretical results). \n\nThe theoretical work is also weak with regards to the convergence rates of the proposed estimator as well. While empirical results can confirm that an estimator can be useful in practice, they are easy to cherry-pick and ultimately theoretical guarantees are needed to know an estimator's general performance. Thus the results would be a lot stronger if convergence rate guarantees were given. \n\nOther comments/questions:\nThe authors should define the MINE estimator in this paper.\n\nThe second bullet point on page 2 says that \"training with larger batch size reduces the variance of the MI estimate\". Isn't this a good thing? That would lead to better convergence.\n\nIn Section 3.1 the notation is technically incorrect. Instead of stating $I(X;X)$ it should be written as $I(X_1;X_2)$ where $X_1$ and $X_2$ are i.i.d. The former suggests that you're comparing the same random variables. \n\nOn page 4, it's suggested that joint samples are sparse with reduced sample size. Why aren't joint samples simply included together during training?\n\nDoes regular L2 regularization help with the drift problem? \n\n[R1] Moon et al.\"Ensemble estimation of mutual information,\" ISIT, 2017.\n[R2] Moon et al., \"Information theoretic structure learning with confidence,\" ICASSP, 2017.\n[R3] Moon et al., \"Ensemble Estimation of Information Divergence,\" Entropy, 2018.\n[R4] Singh and Poczos, \"Exponential concentration of a density functional estimator,\" NeurIPS, 2014.\n[R5] Kandasamy et al., \"Nonparametric von Mises estimators for entropies, divergences, and mutual informations,\" NeurIPS. 2015.", "extra_scores": {}, "extra_text": {}}, {"review_id": "9gqOLUXduKo", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### EDIT:\n\n I thank the authors for their detailed response. I also appreciate the effort that's been put into refining the draft. Unfortunately, I'm still not very happy with the motivation of attacking the drifting phenomenon on MINE. The main reason for removing the drifting effect is for moving average of history outputs. However, there are various ways for tackling this ( as pointed out in my original reviews, like using a non-drifted mutual information estimator with moving average or plugin some robust density estimators). Yes, I agree with the author the drifting phenomenon is not the only problem that the proposed method solves. But actually, the stability of other MI estimators also allows them to avoids having exploding network outputs.\n\n Also, in practice, people don't usually run moving average on MI for representation learning. I encourage the author to explore the importance of moving average of MI estimators further.\n\nR3 suggests the author take some non-parametric estimators as baselines. But I think it's fine to only compare to some parametric(variational) methods on high dimension setting, where most non-parametric estimators fail. Nonetheless, it's always good to have additional experiments compared to some non-parametric methods in low dimension settings.\n\nOverall, I lean toward rejection given current concerns.\n\nSummary\n\nThe paper introduces a generalized version of the mutual information neural estimation (MINE), termed regularized MINE (ReMINE). Some interesting experimental results are firstly provided on a synthetic dataset: the constant term in the statistical network is drifting after MI estimate converges. Also, the optimization of MINE will result in the bimodal distribution of the outputs, in which the statistical network has very distinct values for joint and non-joint samples. The paper presents a theoretical explanation for the drifting phenomenon. In light of this, the author's approach is to add a regularized term to prevent the drifting phenomenon. They impose a $L_2$ regularization on the logsumexp term to enforce the network to find a single solution. Further, the authors make use of the historical estimation for better performance. Empirically, the proposed regularized term works well along with the original MINE estimator and ReMINE  has better performance in the continuous domain\n\nContributions\n\ni) Proposal of a novel regularized MINE objective for solving the drifting phenomenon. The new objective successfully finds a single solution and exhibits a lower variance.\n\nii) Provide interesting insights, such as drifting phenomenon and the instability due to small batch size, out of experiments. \n\niii) Experimental validation of the proposed method for solving the drifting problem. Achieve better performance for mutual information estimation in the continuous domain. \n\nIssues:\n\ni) The drifting phenomenon of MINE is a feature but not a bug, since the drifting term has no effect on the final MINE estimated value. The motivation and benefits of solving drifting problems are unclear to me.\n\nii) Is the drifting phenomenon of the statistical network ubiquitous among the density ratio estimators? For example, does it exist in the density ratio estimator in logistic regression or in JS dual lower bound? If not, we can directly plug these non-drifting density estimators in MINE, instead of regularized MINE. Another apparent remedy is to make the output of statistical network $T$ zero-meaned by subtracting the online sample mean of $T$.\n\niii) The proposed ReMINE is motivated by the drifting phenomenon. But it can also alleviate the exploded outputs / bimodal distribution of the outputs since ReMINE explicitly imposes $L_2$ constraint on its output. The connection between the exploded outputs / bimodal distribution of the outputs and ReMINE is weak in the paper.\n\niv) The paper states that MINE must have a batch size proportional to the exponential of true MI to control the variance. The statement is wrong. Yes, the variance of some mutual information estimator, like NWJ,  is proportional to the exponential of true MI, as proved in [1]. However, the variance of MINE is not proportional to the exponential of true MI in finite sample case (in asymptotic maybe), due to the log function. \n\nMinors:\na) I wonder whether the SMILE estimator (cited in the paper) implicitly solves the drifting problem. Since the optimal statistical network cannot drift freely in SMILE.\n\n[1]A Theory of Usable Information Under Computational Constraints, Xu et al, ICLR20. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper is a study in optimizing the Donsker-Varadhan lower bound on mutual information focusing on a "drift" problem.  The bound is a difference between terms which appears to have an extra degree of freedom where the two terms increase or decrease together.  They propose a fix for this problem. The authors state that the DV bound is of practical value but in most cases it is replaced by discriminative lower bounds as in contrastive predictive coding (CPC) which are biased but have lower variance. The paper does not address the variance (convergence) issues with the DV bound.

We have a well informed reviewer who feels that the paper is not sufficiently novel and has other issues supporting rejection.  Other reviews are not very enthusiastic.  I will side with rejection.
Decision: Reject