Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: Summary:
The authors review recent developments in gradient estimators for the IWAE bound and use them to develop a new theoretical justification for the Sticking the Landing (STL) estimator.

Unfortunately, the sole novelty in this paper is a new justification for the STL estimator. The presentation, while thorough, is not novel or particularly clear. There are no experiments. These factors combine to lead me to suggest a reject.

Specific points:
* The "AISLE framework" is simply used to point out that IWAE and RWS optimize KLs in different directions for the parameters of q. This is well-known in the literature and is discussed in several of the papers cited by the authors. A new framework is not needed to point this out.
* There seems to be an overall misunderstanding of the difficulties associated with multi-sample objectives. The difficulties with IWAE do not come because it is multi-sample, but because the KL direction optimized for the approximate posterior (q) is from q to p. Thus to compute gradients of the KL we must take gradients back through latent variables sampled from q. RWS avoids this by optimizing the other KL direction, and thus does not need to take gradients through the sampling operation. Many of the issues with IWAE mentioned in the paper also appear with the standard ELBO, which is a single-sample bound. 
* Furthermore, IWAE does not necessarily require reparameterizations to deal with the high variance of its terms. Control variates can be used when latent variables are discrete, e.g. Mnih et. al 2016 "Variational inference for Monte Carlo objectives" and Tucker et al. 2017 "REBAR: Low-variance, unbiased gradient estimates for discrete latent variable models". 
* RWS can still be thought of as a multi-sample objective in the sense that you use multiple samples of z to estimate the gradient for one data point x. The true difference is that the RWS gradient estimator is an asymptotically consistent estimator of the gradient of the marginal likelihood, while the IWAE gradient estimator is an unbiased estimator of the gradient of an objective (IWAE lower bound) that becomes the marginal likelihood in the limit of infinite samples.
* As such, your claim to "have shown that the adaptive-importance sampling paradigm of the reweighted wake-sleep is preferable to the multi-sample objective paradigm of importance weighted autoencoders" is far too strong, especially considering the fact that experimental evidence in Tucker et al. 2018 shows there are situations where either one is preferable. Additionally, the DReGS estimator avoids 2 of the 3 issues you present in remark 1.
* A smaller point: I found it hard to follow your derivations when compared with Tucker et al. 2018 because their identities use expectations over standard gaussian noise (epsilons) while expectations in your paper are all written with respect to q (e.g. the right-hand side of lemma 1). It would be helpful to go more in-depth about why that is.

To change my mind the authors would have to include experimental evaluation of some kind and demonstrate more novelty.
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: Summary: This paper presents a unifying framework through which much of the recent work on maximum likelihood learning in latent variable models (variational autoencoders) via multisample variational approaches and importance weighted approaches can be understood. It is a relatively clean framework that shows how many of the popular approaches can be described a distinct gradient-based approaches for a single underlying framework with two separate objectives for the generative model weights and the variational posterior weights.

Strengths: 
- The framework is elegant and the derivations are simple. 
-  It clarifies the connection between distinct algorithms and shows the consistency of ones that were previously poorly understood (IWAE-STL).
- It has the potential of generating new interesting algorithms.

Weaknesses:
- It is not clear to me why the first bullet of Remark 1 is so crucial. IWAE can be applied in discrete settings (see, e.g., Mnih & Rezende, 2016) and, as you show, reparameterizations can be applied in the adaptive importance sampling type algorithms to potential improve the variance.
- The paper currently struggles with some organizational issues. It reads as if the paper was written as a 15 page paper and split in half to satisfy the length requirements. In particular, I would recommend shortening the introduction, cutting out as much of Sec 2 as possible, and moving experiments into the main draft. Derivations are fine left in the Appendix. 
- I appreciate that it wasn't the primary aim of the paper to introduce new algorithms, but I think it could strengthen the contribution to consider at least a few. Are there any other interesting divergences to consider for the proposal distribution objective?
- The experiments are quite lacking. In tandem with the above point (consider novel algorithms), the ICLR community might rightfully expect some experiments on large scale models.  I appreciate that there might not be much consistency in terms of which methods outperform others, but large scale experiments would at least present evidence of this point.

Citations:
Mnih & Rezende, 2016. https://arxiv.org/abs/1602.06725
review_title: Official Blind Review #1

final_score: rating: 8: Accept
review_content: UPDATE: bumping up my score after the revisions

---


Nice connections but novelty and practical takeaways unclear

SUMMARY OF THE PAPER:

This paper views recent IWAE-based [1] methods (IWAE-STL [2], IWAE-DREG [3], RWS [4, 5]) for training generative models p and inference networks q under a common framework, AISLE.
This heavily relies on the "double-reparameterization" property by [2] and is restated in Lemma 1.
This framework makes it explicit that we're interested in
1) maximizing the (log) marginal likelihood wrt p parameters, and
2) minimizing some divergence between the posterior in the learned model to q.

In AISLE, IWAE-STL's q-gradient is viewed as a doubly-reparameterized self-normalized importance sampling (SNIS) estimate of KL(p || q).
This is in contrast to viewing it as a biased estimator of the IWAE's q-gradient.
This can potentially explain why it performs well when number of SNIS samples are increased.
It is also some evidence against the fact that having no unified objective is bad (because there isn't evidence of IWAE-STL diverging despite there being no unified objective).

IWAE-DREG's q-gradient is viewed as a doubly-reparameterized SNIS estimate of X-divergence(p || q) (up to multiplicative constant of the number of SNIS samples).
This is in contrast to viewing it as an unbiased estimator of the IWAE's q-gradient.

The view on RWS is unchanged: the q-gradient is a SNIS estimate of KL(p || q).

For me, the main contribution is viewing IWAE-STL and IWAE-DREG q-gradient estimators as biased gradients of an explicit divergence rather than of the IWAE objective.
I also found the observation that the signal-to-noise (SNR) decrease in IWAE's q-gradient can be proved by noting that it is a SNIS estimator of a zero vector nice.

STRUCTURE:
The article is well-written and easy to understand.

NOVELTY:
A different view on IWAE-STL and IWAE-DREG is interesting and novel (as mentioned above).
This means that IWAE-STL and IWAE-DREG are good not only because they reduce gradient variance (as previously understood) but also potentially because they directly target a divergence.
Viewing generalization of RWS as a main contribution (first bulletpoint of Section 1.2: "...we show that AISLE admits RWS as a special case.") is a bit of a stretch since this generalization is very straightforward from the way RWS is formulated.
The recommendation of using RWS-style algorithms over IWAE as given in the abstract ("we argue that directly optimising the proposal distribution in importance sampling as in the RWS algorithm is preferable to optimising IWAE-type multi-sample objectives) is also not novel since this is also advocated by [5] (section 3.2: "This makes RWS a preferable option to IWAE for learning inference networks because the phi updates in RWS directly target minimization of the expected KL divergences from the true to approximate posterior").
The recommendation as a method for non-reparameterisable latent variables at the end of section 1.2 ("as well as further algorithms which do not require reparameterisations") is also given in [5].
Are there different adaptive importance sampling algorithms that could be used within AISLE that would improve on IWAE-STL/IWAE-DREG/RWS?

EXPERIMENTS:
There are no experiments in the main paper.
However, experiments that would support/falsify the following points could be good:
- RWS and IWAE-STL don't suffer from non-unified objectives because IWAE-STL has non-unified objectives but doesn't diverge,
- [targeting direct divergence] is more useful than (or as useful as) [lower variance gradient estimators].

CONCLUSION:
While I really like the presentation and connections made in the paper, I'm not sure what the practical takeaways are (other than use IWAE-STL, IWAE-DREG, RWS over IWAE which is advocated by [2], [3], [5]).
I'm giving this a weak accept due to the former.
I'm willing to bump up my score if
- the paper is modified to more accurately reflect the contributions or
- there are experiments that provide additional support for the [targeting direct divergence] view in addition to [2, 3, 4, 5], or
- there is a new practical algorithm that the AISLE generalization would suggest that is better than IWAE-STL, IWAE-DREG, RWS in some respects.

[1] Importance Weighted Autoencoders. https://arxiv.org/abs/1509.00519
[2] Sticking the Landing: Simple, Lower-Variance Gradient Estimators for Variational Inference. https://arxiv.org/abs/1703.09194
[3] Doubly Reparameterized Gradient Estimators for Monte Carlo Objectives. https://arxiv.org/abs/1810.04152
[4] Reweighted Wake-Sleep. https://arxiv.org/abs/1406.2751
[5] Revisiting Reweighted Wake-Sleep for Models with Stochastic Control Flow. https://arxiv.org/abs/1805.10469
[6] Variational Inference via χ-Upper Bound Minimization. https://arxiv.org/abs/1611.00328
[7] Tighter Variational Bounds are Not Necessarily Better. https://arxiv.org/abs/1802.04537
review_title: Official Blind Review #3

Meta-review: The authors argue that directly optimizing the IS proposal distribution as in RWS is preferable to optimizing the IWAE multi-sample objective. They formalize this with an adaptive IS framework, AISLE, that generalizes RWS, IWAE-STL and IWAE-DREG. 

Generally reviewers found the paper to be well-written and the connections drawn in this paper interesting. However, all reviewers raised concerns about the lack of experiments (Reviewer 3 suggested several experiments that could be done to clarify remaining questions) and practical takeaways. 

The authors responded by explaining that "the main "practical" takeaway from our work is the following: If one is interested in the bias-reduction potential offered by IWAEs over plain VAEs then the adaptive importance-sampling framework appears to be a better starting point for designing new algorithms than the specific multi-sample objective used by IWAE. This is because the former retains all of the benefits of the latter without inheriting its drawbacks." I did not find this argument convincing as a primary advantage of variational approaches over WS is that the variational approach optimizes a unified objective. At least in principle, this is a serious drawback of the WS approaches. Experiments and/or a discussion of this is warranted.

This paper is borderline, and unfortunately, due to the high number of quality submissions this year, I have to recommend rejection at this point.

Decision: Reject