Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work combines the technique of Annealed Importance Sampling (AIS) with Importance Weighted Autoencoder (IWAE) bounds on mutual information to obtain new and significantly tighter bounds. They examine the situations where the joint distribution $p(x, z)$ is known via $p(z)$ and $p(x | z)$, and the situation where only $p(z)$ is known. In the former case, they obtain exact bounds (or asymptotically exact given large enough sample sizes), and in the latter case they find approximate bounds by learning an energy function to approximate $p(x |z)$. Their experiments show the utility of their proposed method. In particular, they obtain impressively tight lower and upper bounds for complex latent variable generative models (VAE and GAN) on complex data such as CIFAR-10.
main_review: Strengths:
* The authors present a comprehensive framework for mutual information bounds, called GIWAE, that unifies many existing approaches.
* Combining AIS with IWAE is elegant, especially when one can define a sensible approximation for $p(z | x)$ in the form of either a prior $p(z)$ or approximate posterior $q(z | x)$ as with a VAE. The MINE-AIS extension is also intriguing due to its potential to extend the proposed techniques far beyond the VAE and GAN scenarios considered in this work.
* The experimental results are quite impressive and convincingly demonstrate the utility of the proposed method.
* The authors provide a comprehensive presentation of their method and an exhaustive review and comparison with other methods.

Weaknesses:
* The MINE-AIS technique is not fully explored. It would be interesting to see MINE-AIS applied in a scenario where the IWAE-type AIS bounds are not applicable (i.e. when $p( x | z )$ is unknown) but this is not investigated.
* The core of the proposed method is previously suggested in a blog post by Sobolev. I believe this is not a significant weakness because the impressive effectiveness of this approach was not previously demonstrated.

Other Comments:
* I am unsure of why a single $z$ is used in the numerator but multiple $z$ are used in the denominator of (7) and other IWAE-type equations. A brief explanation would aid my understanding.
summary_of_the_review: Overall, this is an impressive work both in terms of its comprehensive mathematical presentation and its high-quality experimental results. Personally, I find the experimental results to be the most interesting aspect, because it opens the possibility of accurate MI estimation in complex scenarios studied in contemporary generative modeling. I recommend this work be accepted.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this work, the authors proposed some novel estimators for mutual information, namely the GIWAE, Muti-Sample AIS and MINE-AIS bounds. The key idea is to apply importance sampling to mutual information in the model-based setting. In both theoretical and empirical analysis, they prove the bounds are much tighter than the existing ones in the literature. 
main_review: Strength: 
(1) This work unifies the view of mutual information from the perspective of importance sampling. This gives us another angle of thinking.
(2) Good summary of the prior literature
(3) Clear theoretical analysis.
(4) The experiments provide evidence on the effectiveness of the proposals.

Weakness:
(1) This paper is more of a theoretical flavor and there is no experiment to show this method is practically useful. For example, is it useful to improve the prediction accuracy (using the learned latent representation z)?
(2) How about the efficiency of the proposed strategies. (e.g. information about the running time). In the experiments, the author(s) have used tens of thousands of annealing steps, which may not be practically feasible. Also, it is helpful to visualize the trade-offs in computing time and bound sharpness.
(3) I am curious about the stability of the training process. 
(4) The method has limited applicability in real-world settings, Generative modeling for feature engineering is not used that often, mostly because it is so hard to assume the knowledge of a likelihood model for a machine learning problem.

Minor Comments:
(1) Make clear distinction between empirical estimators and theoretical estimators. For example, I_{BA} is theoretical and I_{IWAE} is empirical. Please add hat over the empirical estimators.
(2) In Figure 1, Z_{T-1} is bigger than other circles. Is there any different meanings?
(3) Table 1 is messy, tidy it up. There too much data and at the first glance, no idea about where I should focus on. Also, there is inconsistency in notations: in the first column, T=30K, K=1K. Should change the capital letter K to lower case letter k.


summary_of_the_review: The reviewer is convinced by the theoretical analysis of this paper and the novel perspectives it proposed. 
It makes some valuable contribution to our community. However, I am concerned about the practicability and efficiency of this method.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper investigates annealed importance sampling (AIS) techniques and multi-sample bounds to improve mutual information estimation. Assuming the knowledge of extract likelihood (joint or marginal), this paper shows the mutual information bound can be tightened via exploiting a series of intermediate distributions interpolating between a (convenient) proposal and the joint density. An energy-based perspective was presented and discussed, revealing interesting insights. The effectiveness of this proposal is demonstrated with generative modeling applications on real-world datasets.
main_review: First I identify myself as Reviewer cNMQ from the last round of peer review of this paper. I see the author(s) have carefully revised their manuscript to address the concerns from the reviews. In particular, the current draft has made clear the distinction between parametric and non-parametric MI estimation, both are important research topics and they are fitted for different application scenarios. Not only that, the presentation has been improved. Better visualization of the bounds has been provided to help understand the relationship between different estimators, and there are many in-place remarks and discussions on the connection to existing bounds. I believe this is a piece of solid work and enthusiastically recommend acceptance. 

Strength 
* Importance sampling is a key technique in statistical inference and its application to MI estimation constructs a promising direction for research. Such motivated, this paper has derived a family of multi-sample novel bounds with annealed proposals.
* The connection between the InfoNCE bound and the generalized multi-sample IWAE MI bound is quite fascinating. The author(s) have made an interesting point in the statement that the naive choice of proposal p(z) has led to the well-known log-K cap of InfoNCE bound, which I believe is critical for improving MI estimation. The current presentation is underselling its significance, so I would encourage the author(s) to enrich the discussions and give more concrete examples for easier understanding.
* Clarity of presentation. I am glad to see the significant improvements made compared to the last submission. 


Below are some minor suggestions that might help the author(s) to further improve the quality of this manuscript. 

Technical points:

A. While the importance sampling perspective for MI estimation is novel, this is not new in the broader context of variational inference. The following works [1-3] are highly relevant: they all follow an annealed path to tighten the bound. The discussions on these works are currently missing from the manuscript, which I think would significantly boost the significance of this work by making strong connections. 

[1] T Salimans, et al. Markov Chain Monte Carlo and Variational Inference: Bridging the Gap. ICML 2015
[2] L Maaloe, et al. Auxiliary Deep Generative Models. ICML 2016
[3] V Masrani, et al. The Thermodynamic Variational Objective. NeurIPS 2019

B. Another major missing piece is the discussion of whether this strategy may inadvertently harm learning. This point is related to the seminal work of [4], which analyzed a particular scenario that tight multi-sample VI bounds actually harm inference. Intuitively, the inference arm is the approximate posterior (i.e., proposal distribution), and since tighter bounds are less sensitive to the quality of approximate posterior, the algorithm is less incentivized to improve it. Related to my technical point A, [3] has provided some perspectives on additional tradeoffs involved. Nonetheless, a careful analysis is beyond the scope of this study, so I only encourage the author(s) to give some thoughts and make some remarks in the paper. 

[4] T Rainforth. Tighter Variational Bounds are Not Necessarily Better. ICML 2018

C. The derivation of GIWAE is one of the key contributions of this paper, but there is no intuition provided in the main text on how this is derived. The author(s) simply say "we define GIWAE lower bound". At least the probabilistic interpretation should appear in the main paper. Also, the author(s) should mention the disadvantage relative to the InfoNCE estimator. In particular, InfoNCE admits an efficient bi-linear implementation [5], because the z is sampled from the prior. I think the difference between IWAE and GIWAE should be further emphasized, say using a table comparing what is needed for both of them, and changing the order of the first two sentences of Sec 2.3 

[5] T Chen, et al. A Simple Framework for Contrastive Learning of Visual Representations. ICML 2020

D. Sorry to see the section "Energy-based Bounds" in the original writing moved to Sec 4, personally, I think it can be presented earlier.  And I do not fully appreciate the importance of Sec 3.3, could the author(s) elaborate?

E. Selecting K, T is a joint optimization problem, but the discussion in Sec 3.4 is singled threaded: it only talks about suggestions when either K ot T is fixed. 

Non-technical points:

F. Please add runtime analysis, and highlight the tradeoff between computation/memory and bound tightness. 

G. Please give the explicit expression of I_{IWAE} in the paper. 

H. Please use the booktab package to format the Tables. Follow the practice of minimalism, remove unnecessary horizontal lines and all vertical lines in the tables. Actually, for Table 1 a figure might be more intuitive. 

I. Figure 2 (b) is excellent. I think it is better to present it earlier in the text, right where the bounds are discussed. 
summary_of_the_review: This is an interesting work and my major concerns from the last round of paper reviews have been adequately addressed. As such I recommend acceptance.
review_title: null

Meta-review: This paper investigates a tighter bound for mutual information and proposes some novel estimators of MI from the importance sampling perspective. The proposed approach provides a unifying framework for mutual information bounds that can deduce many existing approaches. The theoretical and experimental analyses well justify the proposed approach and shows the bounds are much tighter than the existing ones.  
Overall, this paper is well written. The relevant literatures are exhaustively reviewed and well compared with the proposed method. The experimental results show remarkable superiority of the proposed method. The proposed framework would shed light on the literatures and open up a new direction of the relevant researches.  
For those reasons, I would like to recommend this paper to be accepted by ICLR2022 conference.
Decision: Accept (Poster)