Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper proposes the Mutual Information Gradient Estimator (MIGE) for estimating the gradient of the mutual information (MI) instead of calculating it directly in learning representation. They are using Stein's estimator following by a random projection to build a tractable approximation to the gradient of the MI. 
 The MIGE is evaluated on several of unsupervised and supervised tasks, and shown improvement over prior MI estimation approaches in maximize the MI and learning features for classification.

In general, I think that the idea of estimating the gradient of the MI instead of directly calculating it  is an exciting research direction, and this paper combines a few pieces together (As mentioned in the paper, there was a work of Li & Turner, 2017 that applied Stein's estimator for implicit models).
However, the experimental part of this paper is lacking. My main concern is regarding the performance on downstream tasks. Although the experiments demonstrate wins over different models in maximizing MI for CIFAR10 and CIFAR100, the only comparison for downstream tasks is for Permutation-invariant MNIST. One more concern is regarding the random projection. It is not clear what is the effect of it on the representation, and how it impacts on the gradine's estimation.  

Strengths:
+ Interesting new model for representation learning based on an estimation of the MI gradients'.
+ Good set experiments looking at MI maximization performance.
+A well-written and well-organized paper.

Weaknesses:
 No comparison on downstream tasks for more datasets except MNIST. In the end, a key question is a final accuracy on different datasets and how to maximize the information effect on it. 
There is no discussion about the effect of the random projection on the representation. For example, how it affects performance? How much the algorithm sensitive to this projection? What is the performance of the MINE if it combined with random projection...

Minor comments:

-Typos and English mistakes - there are many typos. For example -
    In the introduction - "Another closely related work is the the Information…"
    In section 2 - “In order to overcome this disadvantages"  
    In section  2.20  - "In optimization, it should be achieved by maximizing the information between z and z."
- There should be more detailed explanations of the experiments. For example - what is the projected dimension (for all the experiments). 

review_title: Official Blind Review #4

final_score: rating: 3: Weak Reject
review_content: The paper argues that directly estimating the intractable mutual information (MI) for representation learning is challenging in high dimensions. Instead the authors propose to estimate the needed MI gradients directly using a score function based approach. Using some identities of MI the authors arrive at an expression for the gradient of the mutual information between input and latent representation (eq 10) and proposes to use a generalization of the reparameterization trick and spectral stein gradient descent to approximate this gradient. In Toy experiment and MNIST/CIFAR10 experiments the authors demonstrate that their method produces latent representations that are more informative than competing MI methods for downstream classification tasks. I found the approach and content of the paper interesting and the results seems encouraging.  My main concern is that I did not find the that I did not find the method and experimental section to be fully comprehensive and further lacking many details which makes it hard to compare the results with prior work. 


Pros:
1) I find the approach taken by the authors interesting and different from current MI estimation approaches. The paper convincingly motivates their approach by describing the deficiencies of current MI estimators and why targeting the gradients directly might have merits.
2) The authors propose to use SSGD and 'generalized' reparameterization in a (well motivated) new setting.
3) The cifar10 experiments in table 1 are encouraging and the toy experiment in 2D is illustrates nicely the deficiencies of the current MI estimators 

Cons
1) The experimental section is lacking many details to fully understand how and what experiments were performed and how comparable they are to prior work
 2) The paper would benefit greatly from a thorough editing to clarify the presentation - there are many missing concepts and definitions that makes it hard to follow without intimate knowledge of related literature. 
 
 
Further suggestions / questions 

1. In section 3. Please define q(z)_psi(z),  q(x,z)_psi and describe how they relate to  E_psi.

2) What exactly are the contributions by the authors wrt to spectral stein gradient descent (sec 2.1) e.g. is it the scalable approach based on random projections described in sec 3 ? Further i would like some discussion on the quality of this approximation?

3) Please provide some more details on the DeepInfoMax and Information bottleneck experiments e.g. How exactly did you estimate the MI gradients in these settings? how is the downstream task setup and is it identical to prior work?


4) About writing style:
I think it would benefit the paper if you let the reader decide for them self what adjectives should be used to describe a result. A few concrete suggestions:
 - Use remarkable/y about your own findings a bit more sparingly (used 4x). 
 - Consider deleting “much” and “vast” in a sentence like: “our approach MIGE gives much more favorable gradient direction, and demonstrates more power in controlling information flows without vast loss”.
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: This paper works out estimators for the gradient of Mutual Information (MI). The focus is on its recent popular use for representation learning. The insight the authors provide is to see encoding the representation as a ‘reparametrization’ of the data. This insight enables mathematical tools from the literature on ‘pathwise derivatives’. With gradients on the MI, one can estimate models that aim to maximize this quantity. For example in unsupervised learning one can learn representations for downstream tasks. This is shown in Table 1. Another application in supervised learning is the Information Bottleneck. This shown in Table 2.

Three points for review:
1)
The estimator for the gradient is shown at first on a toy task. Take a correlated Gaussian distribution and estimate gradients of the MI. The correlated Gaussian has an analytical form of MI, which makes this a useful experiment. The paper claims that this estimator ‘provides a tighter and smoother’ gradient estimate. I don’t see how this experiment and this claim tie together. Could the tightness or smoothness be quantified? It seems the MIGE has a lower variance, could empirical results or bounds on the variance be obtained? 

Moreover, this plot concerns random quantities, whereas we see only one realization. Both the MIGE and the MINE hold under expectation of samples from the data distribution. This is a toy example where we can sample infinitely from the data distribution. That means we can either a) plot more samples or b) obtain (empirical) error bounds on the gradient under these sampling distributions.

2)
The major experimental result in the paper shows advantage of the gradient estimator in Transfer learning. Specifically, the authors compare against the recent DIM of Hjelm et al 2019. The authors train a quote ‘small fully connected neural network classifier’. However, the work of Hjelm trains a linear SVM on the representations. It is not clear where the increase in performance originates. Is it the improved representations (as obtained by using MIGE) or is it the change classifier? 

3)
One contribution of the paper is to make the gradient estimator work in high dimensions. To this end, the authors propose Random Projections. It is not clear how this approximation influences the results. An experiment regarding this topic would make the point clearer. Is RP used in the current experiments? Then how does this influence the results? Is the RP used for computational purposes? Then can we quantify the gain in computation?

Minor comments: 
  *’In practice, we do not care about MI estimation’. Please explain further or refer to previous work.
  *’In optimization, it should be achieved by maximizing the information between z and z.’ (Section 2). Two points
    1. The information ‘between z and z’ is probably a typo?
    2. How does sufficiency relate to an optimization problem? Doesn’t sufficiency mean in this context I(X;Y)=I(Z;Y)?
  * Equation 12: In the part $\nabla_psi (x, E_\psi(x))$. Why do we take gradient w.r.t. x? It seems to me that the reparametrization is a function of x only via $E_\psi(x)$. If not, then please explain what this tuple means.
  * Table 1 has no units. How to interpret the numbers in this table?
  * Section 4.3, authors note their experiment is ‘a little bit different’ from other related research. How and what exactly is different?

Typographic comments
  *Just below eqn17, ‘minibatche’ => ‘mini-batch’ or ‘mini batch’
  *Section 4.2 ‘images classification’ => ‘image classification’
  *’However A tractable density is’ => ‘However, a tractable density is’
  *’Estimating gradients of MI than’ -> ‘Estimating gradients of MI rather than’
  *Section 3, circumstance 1 ‘representation’ => ‘represent’

review_title: Official Blind Review #3

final_score: rating: 8: Accept
review_content: This paper proposes MIGE---a novel estimator of the mutual information (MI) gradient, based on estimating the score function of an implicit distribution. To this end, the authors employ the spectral Stein gradient estimator (SSGE) and propose its scalable version based on random projections of the original input. The theoretical advantages of the method are presented using a toy experiment with correlated Gaussian random variables, where both the mutual information and its gradient can be computed analytically. In this setting, MIGE provides gradient estimates that are less biased and smoother than baselines. The method is also evaluated on two more complicated tasks: unsupervised representation learning on Cifar-10 and CIfar-100 via DeepInfoMax (DIM) and classification on MNIST with Information Bottleneck (IB), where MIGE outperforms all baselines by a significant margin.

I recommend ACCEPTing this paper. This work discusses a vital problem and proposes a novel, well-motivated, principled and very performant solution; additionally, it demonstrates the broad applicability of the introduced method. While the proposed technique consists of previously known building blocks (spectral Stein gradient estimator and random projections), it is cleverly applied in a novel context of estimating MI gradients.

While the paper is solid, I believe that it could be improved in the following ways. Firstly, I would like to see 1) more extensive and 2) larger-scale evaluations. In the DIM experiment, 1) would correspond to trying the DIM(L) approach, which maximises patch-wise MI. In fact, I strongly recommend including this experiment as it corresponds to and could improve current state-of-the-art. If it turns out that MIGE does not work well on DIM(L), then this would correspond to a serious issue with the method. In this experiment, 1) would also include providing other metrics for learned representations. It would be much more convincing to include estimates of true mutual information (e.g. InfoNCE bound evaluated with a large number of samples [1]) and showing that MIGE can attain higher values than baselines. 2) would correspond to evaluation on bigger datasets: (tiny) ImageNet and STL-10 dataset. Also, the toy experiment would benefit from a higher-dimensional setting (e.g. d=256 to d=1024), since these are often used in practice. 
Secondly, the paper is sloppily-written, which quite a few grammar and stylistic mistakes (e.g. sentence in sec 3, paragraph 2: “we assume obtain to…”, which starts with a lower-case letter and doesn’t make sense). Finally, the paper would benefit from the following clarifications: 1) explain what is the Nystr\:om method, 2) provide either a proof or a citation for eq (19); also the error bound for SSGE should be provided for the paper to be self-contained, 3) explain the difference between q_\psi and p_\psi, which seem to be used interchangeably.

Additional remarks:
Sec 2.2, 2), “streamlining” is unclear
Circumstances 2 and 3 can be quite easily derived from circumstance 1; also they are not evaluated empirically; it would be nice to have experiments for them, and they can be moved to the appendix in case of lack of space
Eq (19) while nice, seem to bear no significance for the proposed method and the rest of the paper; consider removing it
Section 4.2 paragraph 2: “shrinking” for different layers wasn’t mentioned before, and is not immediately clear what it means; the reader needs to be intimately familiar with the DIM paper to understand.
Section 4.3 mentions “threshold” for stein gradient estimator, which was not mentioned before. Please explain what it is.
Equations (8-10) are just simple derivations and are not necessary; it would be enough to provide Eq (10).
The authors talk about MINE, which optimizes the InfoNCE bound [1], which is also used in DIM and CPC [2]. I strongly encourage the authors to cite [1] and [2] and mention them in the related works. Additionally, it would be clear if Figure 1 and related references and description used “InfoNCE” instead of “MINE” as the name of the method since InfoNCE is an estimator and MINE is just a particular implementation of the method.
 
[1] Poole et. al., “On variational bounds of mutual information”, ICML 2019.
[2] van den Oord et. al, “Representation Learning with Contrastive Predictive Coding“, arXiv 2018.
review_title: Official Blind Review #1

Meta-review: This paper proposes the Mutual Information Gradient Estimator (MIGE) for estimating the gradient of the mutual information (MI), instead of calculating it directly. To build a tractable approximation to the gradient of MI, the authors make use of Stein's estimator followed by a random projection. The authors empirically evaluate the performance on representation learning tasks and show benefits over prior MI estimation methods.
The reviewers agree that the problem is important and challenging, and that the proposed approach is novel and principled. While there were some concerns about the empirical evaluation, most of the issues were addressed during the discussion phase. I will hence recommend acceptance of this paper. We ask the authors to update the manuscript as discussed.
Decision: Accept (Poster)