Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: In this paper, authors present Nested Variational Inference (NVI)--a framework to create increasingly better variational approximations. NVI allows for several existing sampling techniques to be employed in a single framework and is well supported by empirical results. 
main_review: Recent years have seen several papers attempting to combing the Monte Carlo sampling methods and variational inference. I think this paper does a decent job of putting several of those work in a nice single framework. I found the paper to be very well written and easy to follow for the most part. The experiments and carefully carried out and visualizations are very informative. Overall, this is one of the better papers I have reviewed this season--a clear accept.
limitations_and_societal_impact: The proposed methods are still pretty involved and I believe it might need more work for these advanced inference techniques to find there way into every practitioners toolbox.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes an inference procedure that combines ideas from variational and nested inference, resulting in a sequence of forward and reverse proposal distributions that are optimized with local objectives that encourage the forward and backward edge marginals to be similar. This allows the objective and therefore gradients to decouple. The addition of resampling, afforded by the connection to nested inference, additionally improves sample diversity.
main_review: *Originality*
* This work proposes a novel combination of prior techniques, namely the local marginal objectives from AVO and the addition of resampling for intermediate distributions.

*Quality*
* The submission is technically sound, and its claims are well-supported by the experimental results. Experiment 1, multimodal sampling, demonstrates quantitative improvements over flow and AVO baselines, both in accuracy as well as sample diversity. Experiment 2, on posterior inference in a small HMM, demonstrates NVI’s use in a problem where intermediate densities are not of constant dimension. Experiment 3 uses NVI on a mixture model on (fashion) MNIST, and contains a favorable qualitative comparison to the same model trained with RWS.

*Clarity*
* The submission is clearly written, although I am hopeful the presentation can be greatly simplified. I would also have liked to see discussion of / relation to AVO earlier than the experiments section.
* The notation might be a bit redundant, with different variable names for (edge) marginals, unnormalized marginals, and proposal distributions. In addition to the different directions, it took a while to get used to the notation. While it does not detract from the paper, any simplifications would be welcome.
* In addition to the appendix, I believe an expert reader could reproduce the results.
* Will the in-text derivation in lines 116-118 be replaced by equations given more space?

*Significance*
* I found the results convincing, although I am not familiar with the SMC literature.
Additionally, the experiments were performed on small data, which leads to concerns regarding scale.

[AVO] Huang, Chin-Wei, Tan, Shawn, Lacoste, Alexandre, Courville, Aaron C. “Improving explorability in variational inference with annealed variational objectives”. Advances in Neural Information Processing Systems. 2018, pp. 9701–9711.

limitations_and_societal_impact: I would like to see speed comparisons between NVI and baselines where appropriate, as that could be a limiting factor in its applicability. For example in the HMM, how fast would exact inference be relative to NVI? This also causes me to be concerned about whether this technique can scale to models with larger state spaces, such as HMMs with thousands of states.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The author proposes a nested variational inference (NVI) framework that combines nested importance sampling and variational inference. This framework allows the training of the important sampling proposal by minimizing KL divergences at each level of nesting. This also provides us a way to learn intermediate densities, which serves as a heuristics for guiding the importance sampler. Empirically, the author evaluates NVI by (1) sampling from a multimodal distribution with a learned annealing path; (2) learn heuristics for future observations in the state-space model; (3) perform amortized inference in the hierarchical deep generative model.
main_review: ### Original
The NVI framework combines variational inference with the nested importance sampler. The idea of combining importance sampling with variational inference is not new, but the novelty of NVI is that it minimizes the KL divergence at each level of nesting and some of the existing work are special cases of NVI. To the best of my knowledge, this seems to be new. In appendix A, the author also discusses the related work. 

### Quality and Clarity
The overall idea of NVI is easy to understand. However, I still think some parts are not explained clearly, and I found them quite confusing when I first read them. I have checked the derivations for the gradient estimate, they seem to be correct. Empirically, the author conducts extensive experiments to confirm the effectiveness of the proposed method.

### Significance
Combining variational inference and importance sampling has recently attracted many research interests. This NVI framework seems to include many existing methods as special cases. Empirically, it seems to outperform the baselines in terms of ESS.

### Questions
1. In the beginning of section 3, I understand why you define a sequence that interpolates between initial $\pi_0$ and target $\pi_K$, and why you define the forward proposal $q_k(z_k|z_{k-1})$. I am not sure I fully understand the role of the reverse kernel $r_k(z_{k-1}|z_k)$. Why do you introduce this? In practice, how do you use this reverse kernel?

2. A follow-up for question 1: why do you want the forward density $\hat{\pi_k}$ to be as close as possible to the reverse density $\check{\pi_k}$ at each level k? What does this objective actually mean? My intuition: If am the one that designs the objective, I will make sure the samples after the proposal $q_k$ should be as close as possible to the intermediate target $\pi_k$, namely the KL divergence between the density of $z_k$ and $\pi_k$. What is the problem of this alternative objective?
 What do you mean 
3. I am not sure I understand the meaning of line 127-130. Why it increases the probability of low-weight samples?
4. Why AVO cannot learn the annealing parameter $\beta$? I thought AVO also minimize KL at each level of nesting.
5. Maybe the author can consider adding more background in the appendix to make the paper more self-contained? For example,  the reweight wake-sleep estimator. 
**Minor**:
1. Better to add a paragraph to summarize the contributions of this paper. 
2. How do you obtain the third equality from the second equality in line 540? Why it becomes $\frac{\partial q_k}{\partial \hat{\phi_k}}$?
3. Maybe consider adding an overall algorithm for easier understanding. 

---
I appreciate the author's response. They managed to clarify most of my concerns, especially about the KL objective. Hence, I raise my score. 
limitations_and_societal_impact: This paper should not have any negative societal impact as this is mainly a methodology paper.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper describes a framework for learning and inference in probabilistic models with structured latent spaces. It operates in the tradition of IWAE and VSMC-like methods that use importance sampling to tighten a variational bound (or equivalently, uses a variational bound as an objective for learning importance sampler proposals). Unlike methods that optimize a single global bound, NVI is defined in terms of local pairwise divergences between a sequence of target distributions, which acts to decouple the stochastic optimization, reducing gradient variance and potentially offering a strategy for parallel inference.
main_review: I struggled with this paper. On the one hand, I think it likely contains an interesting and valuable technical contribution. On the other hand, the experiments are not particularly compelling, and while that's not a dealbreaker if the idea itself is obviously worthwhile *a priori* (as I sense that the authors understand it to be), I don't think the paper as written is effective at communicating that understanding.

Starting with the abstract and intro, I didn't feel that they conveyed a very clear motivation for the work, beyond 'combine nested importance sampling and variational inference', which is not by itself a goal that anyone knew they had. What useful properties do each of those ideas (NIS and VI) brings to the table, and what do we hope to get by combining them? What kind of tasks, that existing methods are poorly suited for, do we hope to be able to solve? (inference in temporal or spatiotemporal models? multimodal posteriors? high-dimensional state spaces? fitting really deep VAEs? all of the above?) Who should consider using this method? There is a lot of technical detail to get through in this paper, and the introduction could do more to give the reader a sense of why it will be worth it. 

I was perplexed by the choice of background material to focus on. The contribution is framed as VI + nested importance sampling, and the paper provides a nice intro to stochastic VI, but no clear definition of nested importance sampling beyond referring to Naesseth et al. 2015/2019. This seems backwards to me. The VAE paper (Kingma and Welling 2013) has 14870 Google Scholar citations, while Naesseth et al. 2015 has just 68; I would suggest that most readers (myself included) are much more likely to need an intro to the latter than the former. 

The technical presentation itself is rather abstract and not straightforward to follow. There is reference to an 'NVI objective', but not a clear description of what this objective is --- is it intended to be eqn (6)? I would have expected that an importance sampling objective would at some point ground out to a procedure involving weighted samples, but the concept of weights isn't even introduced until the next section. It also wasn't clear to me how the approach relates to the concept of nested sampling, which I understand from the Naesseth papers to refer to using an 'inner' sampler to generate proposals for an 'outer' sampler. Could this work have been developed in a simpler formalism, like SMC, or is the sampler-within-a-sampler aspect crucial?

Overall I think the presentation tries too hard to be general, at the expense of concreteness and intuition. It might help to frame things in terms of a specific running example, like the multimodal annealing setup from section 4.1 (introducing this earlier would also help give a sense of what kind of problems the approach is hoping to tackle). This would allow you to walk the reader concretely through the steps of applying NVI, and hopefully in the process give some intuition for why it works and where it differs from other methods.

It's not immediately obvious whether the NVI objective is a (stochastic) lower bound on the marginal likelihood, or some similar such property. Is it? If so, I think that'd be a useful argument to include.

I would have liked to see some discussion of implementation, and its ease or lack thereof. As I understand it, the 'juice' of the method comes from estimating gradients using only locally random variables, and an entire page of the paper is about explicitly computing these gradients. This seems to cut against the dominant paradigm of 'just write down the computation graph and let autodiff do its thing'. How straightforward is it to implement all of this in modern DL frameworks? Are there any tricks? You could limit this discussion to the case where all gradients are reparameterizable, since handling discrete variables seems like a somewhat orthogonal (and well-studied) issue.

There's a lot of ink spilled about handling both forward and reverse KL (it's in the abstract even!), but it wasn't clear to me why this is worth focusing on. Does forward KL have any nice properties in this setting that would make up for how much of a pain it is to estimate?

As mentioned above, I didn't find the experiments compelling enough to carry the paper of their own accord. The application to deep generative modeling doesn't contain any quantitative results, or compare to any baselines (except for a reference to RWS in a 'Figure 9' that doesn't seem to have made it into the paper). The multimodal sampling problem is quite toy (though an effective illustration), and the gains over AVO are not super striking. I do appreciate that the experiments seem quite carefully done and are nicely presented, especially figures 3 and 4. In the state-space model experiment, is there any explanation for why optimizing the reverse kernel made things worse?

nits:
line 41: typo 'revserse'
lines 101-103: SMC is listed twice as a special case.

*********************************
Updated: thanks to the authors for your thorough response. I'm convinced that the technical contribution of the paper is solid, and in light of this am raising my score by a point. I encourage the authors to include their clarifications in a revised version of the paper---I think there's likely a much stronger paper here if well explained. The surrogate objective in particular seems worth mentioning and possibly presenting in the supplement (would it make sense to describe it as an extension/generalization of DiCE (https://arxiv.org/abs/1802.05098)?).
limitations_and_societal_impact: There are no direct societal impact concerns for this work, since the contribution is purely technical and not focused on a particular application.

I do wish there were a bit more discussion of limitations. Have we finally solved probabilistic inference? Is this framework appropriate for all problems? (assuming not, then why?) What improvements or future directions would be worth exploring?
review_title: null

Meta-review: This paper learns proposals for a sequence of importance samplers by minimizing a KL divergence for each step (or level) of the sequence. The reviewers consider the work represents a solid and novel technical contribution and is above the acceptance bar for NeurIPS.

The authors are encouraged to improve the presentation and clarity by incorporating the feedback from the reviewers FQ8n, Ae4v, and 3PA6. In particular, consider adding a paragraph/list with contributions, an algorithm box, adding NIS to the background section, and discussing the limitations of NVI. The paper would also benefit from including other technical clarifications, such as the relationship to the ELBO or the intuition on why the forward KL estimator is insatiable.
Decision: Accept (Poster)