Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 3: Clear rejection
review_content: Response to authors: After reading the authors' response, I have decided to maintain my original rating. The authors have not adequately addressed my main concerns.

Novelty: The work here, as indicated by the authors, is largely an incremental improvement over an existing work MINE. The authors' response did not alleviate this concern and in fact reinforced it.

Citations and comparisons to other work: The authors did not agree to even include citations to important literature in this area. This should have been a bare minimum and it is a mistake for variational approaches to ignore these works which have theoretical guarantees that many variational approaches do not have. Comparisons to other methods should also have been included. The methods the authors did compare to have weak (or no) theoretical guarantees for higher dimensions.

Theoretical work: The authors simply pointed to the theoretical work for MINE. However, the theoretical work in MINE is also very weak and only focuses on estimation consistency and not convergence rates (i.e. the statistical bias and variance of the estimator). More theoretical work is needed in this area to justify the use of these estimators over others.

Some responses to other comments that may help the authors with further revisions:

(1) The presentation of MINE should occur in the main paper as this is crucial for understanding the paper.

(2) This was not clear. Perhaps the authors could include similar pointers in the paper with each of these issues.

(3) The bias I'm referring to here is the actual statistical bias of the estimator. From Theorem 6, it seems that the drift problem does seem to create some bias but it would be useful to quantify that, which could then lead to a bias correction approach.

(7) The way this is currently worded, it sounds like you are saying that training with a larger batch size is bad. This part should be clarified to avoid this. 

Original Review:

This paper presents a modified version of a neural network-based MI estimator. They investigate a few of the issues of this specific estimator and propose a regularization to help with one of them. MI estimation is an important and difficult topic. Improvements in this area are of definite interest.

Pros:
The paper appears to be technically correct. The experiments are somewhat supportive of including the regularization, especially when the MI is higher which is a known issue with some MI estimators.

Cons:
There are some interesting ideas here but the paper feels unpolished. The presentation of the ideas is somewhat unconventional. Several issues with the MINE estimator are presented and then two of them are discarded in favor of a focus on one of them. The paper could benefit from a bit more focus in this regard. In the end, the authors really only propose a small modification to the MINE estimator to counter the supposed drifting problem and do some experiments showing some improvement. But it's not clear how much of a problem this drift really is. The authors show that it causes a bias but they do not present how much bias it adds.   

In addition, the authors are severely neglecting some of the non-neural network state of the art MI estimators in their citations and comparisons (see the references below for some examples, which all have strong theoretical results). 

The theoretical work is also weak with regards to the convergence rates of the proposed estimator as well. While empirical results can confirm that an estimator can be useful in practice, they are easy to cherry-pick and ultimately theoretical guarantees are needed to know an estimator's general performance. Thus the results would be a lot stronger if convergence rate guarantees were given. 

Other comments/questions:
The authors should define the MINE estimator in this paper.

The second bullet point on page 2 says that "training with larger batch size reduces the variance of the MI estimate". Isn't this a good thing? That would lead to better convergence.

In Section 3.1 the notation is technically incorrect. Instead of stating $I(X;X)$ it should be written as $I(X_1;X_2)$ where $X_1$ and $X_2$ are i.i.d. The former suggests that you're comparing the same random variables. 

On page 4, it's suggested that joint samples are sparse with reduced sample size. Why aren't joint samples simply included together during training?

Does regular L2 regularization help with the drift problem? 

[R1] Moon et al."Ensemble estimation of mutual information," ISIT, 2017.
[R2] Moon et al., "Information theoretic structure learning with confidence," ICASSP, 2017.
[R3] Moon et al., "Ensemble Estimation of Information Divergence," Entropy, 2018.
[R4] Singh and Poczos, "Exponential concentration of a density functional estimator," NeurIPS, 2014.
[R5] Kandasamy et al., "Nonparametric von Mises estimators for entropies, divergences, and mutual informations," NeurIPS. 2015.
review_title: Major revisions needed

final_score: rating: 6: Marginally above acceptance threshold
review_content: The work studies a neural-network based estimator, referred to as MINE, for approximating the mutual information between two variables.  By designing a synthetic dataset, the work studies properties of MINE and, based on these findings, proposes a new method incorporating regularization, called ReMINE, that they empirically demonstrate has nice performance. 

Strengths:
-- The proposed ReMINE algorithm combats drifting of the two terms in MINE's approximation of the DV representation through regularization that forces the network to find a single solution (as opposed to a family of solutions).
-- The proposed method is novel and performs well compared to state-of-the-art.

Weaknesses:
-- I found the writing to be difficult to understand and follow. For example, the paper is not self-contained: the authors use terms like "statistical network" without providing definitions, and begin simulations without introducing MINE or giving a high-level discussion of how it estimates the DV representation, so it would be very difficult to read this paper without having first read the original MINE paper. 
-- The paper lacks intuition. For example, there is little to no discussion of why the synthetic dataset proposed is a good choice for studying the underlying properties of MINE.

Some additional typos/comments:

Pg 1: "...the value of each term in MINE loss IS shifting even..."
Pg 2: I believe equation (3) should be a lower bound not an equality.
Pg 2: "...where estimates of  __ and __ DRIFT in parallel..."
Pg 4: mu and sigma in Figure 2 were never defined.
When printed, the black-and-white figures are too small to read and interpret.
review_title: Regularized MINE review

final_score: rating: 7: Good paper, accept
review_content: This paper attempts to answer the four questions raised from the mutual information estimator. To this end, this paper investigates why the MINE succeeds or fails during the optimization on a synthetic dataset. Based on the observations and discussions, the paper then proposes a novel lower bound to regularize the neural networks and alleviate the problems of MINE.

Overall, the paper is easy to follow and new insights have been brought for the MI estimator and the downstream tasks.

review_title: Good paper with new insights for the community

final_score: rating: 5: Marginally below acceptance threshold
review_content: ### EDIT:

 I thank the authors for their detailed response. I also appreciate the effort that's been put into refining the draft. Unfortunately, I'm still not very happy with the motivation of attacking the drifting phenomenon on MINE. The main reason for removing the drifting effect is for moving average of history outputs. However, there are various ways for tackling this ( as pointed out in my original reviews, like using a non-drifted mutual information estimator with moving average or plugin some robust density estimators). Yes, I agree with the author the drifting phenomenon is not the only problem that the proposed method solves. But actually, the stability of other MI estimators also allows them to avoids having exploding network outputs.

 Also, in practice, people don't usually run moving average on MI for representation learning. I encourage the author to explore the importance of moving average of MI estimators further.

R3 suggests the author take some non-parametric estimators as baselines. But I think it's fine to only compare to some parametric(variational) methods on high dimension setting, where most non-parametric estimators fail. Nonetheless, it's always good to have additional experiments compared to some non-parametric methods in low dimension settings.

Overall, I lean toward rejection given current concerns.

Summary

The paper introduces a generalized version of the mutual information neural estimation (MINE), termed regularized MINE (ReMINE). Some interesting experimental results are firstly provided on a synthetic dataset: the constant term in the statistical network is drifting after MI estimate converges. Also, the optimization of MINE will result in the bimodal distribution of the outputs, in which the statistical network has very distinct values for joint and non-joint samples. The paper presents a theoretical explanation for the drifting phenomenon. In light of this, the author's approach is to add a regularized term to prevent the drifting phenomenon. They impose a $L_2$ regularization on the logsumexp term to enforce the network to find a single solution. Further, the authors make use of the historical estimation for better performance. Empirically, the proposed regularized term works well along with the original MINE estimator and ReMINE  has better performance in the continuous domain

Contributions

i) Proposal of a novel regularized MINE objective for solving the drifting phenomenon. The new objective successfully finds a single solution and exhibits a lower variance.

ii) Provide interesting insights, such as drifting phenomenon and the instability due to small batch size, out of experiments. 

iii) Experimental validation of the proposed method for solving the drifting problem. Achieve better performance for mutual information estimation in the continuous domain. 

Issues:

i) The drifting phenomenon of MINE is a feature but not a bug, since the drifting term has no effect on the final MINE estimated value. The motivation and benefits of solving drifting problems are unclear to me.

ii) Is the drifting phenomenon of the statistical network ubiquitous among the density ratio estimators? For example, does it exist in the density ratio estimator in logistic regression or in JS dual lower bound? If not, we can directly plug these non-drifting density estimators in MINE, instead of regularized MINE. Another apparent remedy is to make the output of statistical network $T$ zero-meaned by subtracting the online sample mean of $T$.

iii) The proposed ReMINE is motivated by the drifting phenomenon. But it can also alleviate the exploded outputs / bimodal distribution of the outputs since ReMINE explicitly imposes $L_2$ constraint on its output. The connection between the exploded outputs / bimodal distribution of the outputs and ReMINE is weak in the paper.

iv) The paper states that MINE must have a batch size proportional to the exponential of true MI to control the variance. The statement is wrong. Yes, the variance of some mutual information estimator, like NWJ,  is proportional to the exponential of true MI, as proved in [1]. However, the variance of MINE is not proportional to the exponential of true MI in finite sample case (in asymptotic maybe), due to the log function. 

Minors:
a) I wonder whether the SMILE estimator (cited in the paper) implicitly solves the drifting problem. Since the optimal statistical network cannot drift freely in SMILE.

[1]A Theory of Usable Information Under Computational Constraints, Xu et al, ICLR20. 
review_title: Novel mutual information lower bound to regularized the drifting problem

Meta-review: This paper is a study in optimizing the Donsker-Varadhan lower bound on mutual information focusing on a "drift" problem.  The bound is a difference between terms which appears to have an extra degree of freedom where the two terms increase or decrease together.  They propose a fix for this problem. The authors state that the DV bound is of practical value but in most cases it is replaced by discriminative lower bounds as in contrastive predictive coding (CPC) which are biased but have lower variance. The paper does not address the variance (convergence) issues with the DV bound.

We have a well informed reviewer who feels that the paper is not sufficiently novel and has other issues supporting rejection.  Other reviews are not very enthusiastic.  I will side with rejection.
Decision: Reject