Venue: ICLR 2018 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors analyze the bias and variance of the IWAE bound from Burda et al. (2015), and with explicit formulas up to vanishing polynomial terms and intractable moments. This leads them to derive a jacknife approach to estimate the moments as a way to debias the IWAE for finite importance weighted samples. They apply it for training and also as an evaluation method to assess the marginal likelihood at test time.

The paper is well-written and offers an interesting combination of ideas motivated from statistical analysis. Following classical results from the debiasing literature, they show a jacknife approach has reduced bias (unknown for variance). In practice, this involves an enumerated subset of calculations leading to a linear cost with respect to the number of samples which I'm inclined to agree is not too expensive.

The experiments are unfortunately limited to binarized MNIST.  Also, all benchmarks measure lower bound estimates with respect to importance samples, when it's more accurate to measure with respect to runtime. This would be far more convincing as a way to explain how that constant to the linear-time affects computation in practice.  The same would be useful to compare the estimate of the marginal likelihood over training runtime.  Also, I wasn't sure if the JVI estimator still produced a lower bound to make the comparisons. It would be useful if the authors could clarify these details.
review_title: interesting statistical analysis and ideas; experiments are limited

final_score: rating: 7: Good paper, accept
review_content: [After author feedback]
I think this is an interesting paper and recommend acceptance. My remaining main comments are described in the response to author feedback below.

[Original review]
The authors introduce jackknife variational inference (JVI), a method for debiasing Monte Carlo objectives such as the importance weighted auto-encoder. Starting by studying the bias of the IWAE bound for approximating log-marginal likelihood, the authors propose to make use of debiasing techniques to improve the approximation. For the binarized MNIST the authors show improved approximations given the same number of samples from the auxiliary distribution q(z|x).

JVI seems to be an interesting extension of, and perspective on, the IWAE bound (and other Monte Carlo objectives). Some questions and comments:

* The Cremer et al. (2017) paper contains some errors when interpreting the IWAE bound as a standard ELBO with a more flexible variational approximation distribution. For example eq. (1) in their paper does not correspond to an actual distribution, it is not properly normalized. This makes the connection in their section 2.1 unclear. I would suggest citing the following paper instead for this connection and the relation to importance sampling (IS):
Naesseth, Linderman, Ranganath, Blei, "Variational Sequential Monte Carlo", 2017.

* Regarding the analysis of the IWAE bound the paper by Rainforth et al. (2017) mentioned in the comments seems very relevant. Also, because of the strong connection between IWAE and IS detailed in the Naesseth et al. (2017) paper it is possible to make use of a standard Taylor approximation/delta methods to derive Prop. 1 and Prop. 2, see e.g. Robert & Casella, "Monte Carlo Statistical Methods" or Liu's "Monte Carlo Strategies for Scientific Computing".

* It could be worth mentioning that the JVI objective function is now no longer (I think?) a lower bound to the log-evidence.

* Could the surprising issue (IWAE-learned, JV1-evaluated being better than JV1-learned, JV1-evaluated) in Table 1 be because of different local optima?

* Also, we can easily get unbiased estimates of the evidence p(x) using IS and optimize this objective wrt to model parameters. The proposal parameters can be optimized to minimize variance, how do you think this compares to the proposed method?

Minor comments:
* p(x) -> p_\theta(x)
* In the last paragraph of section 1 it seems like you claim that the expressiveness of p_\theta(x|z) is a limitation of VAE. It was a bit unclear to me what was actually a general limitation of maximum likelihood versus the approximation based on VAEs.
* Last paragraph of section 1, "strong bound" -> "tight bound"
* Last paragraph of section 2, citation missing for DVI
review_title: Interesting debiasing methods for Monte Carlo objectives

final_score: rating: 7: Good paper, accept
review_content: This paper provides an interesting analysis of the importance sampled estimate of the LL bound and proposes to use Jackknife to correct for the bias. The experiments show that the proposed method works for model evaluation and that computing the correction is archivable at a reasonable computational cost. It also contains an insightful analysis.


review_title: Interesting analysis

final_score: rating: null
review_content: Hello, 

Interesting analysis. But I’m not particularly surprised that JVI during training does not result in better models compared to IWAE. We often observe very small effective sampling sizes when training big models with only few of the normalized importance weights is close to 1. It seems JVI would result in very similar gradients under these circumstances. I think it could strengthen the paper if this would be investigated and discussed.

Wouldn’t a common scale for the LL plots in Figure 3 make it easier to read? 

Do you think it would be preferable if the community continued to report biased bounds instead of JVI estimated LLs? This provides a natural partial protection against overestimating due to low effective sampling sizes, doesn’t it? 
review_title: Interesting analysis

Meta-review: The authors analyze the IWAE bound as an estimator of the marginal log-likelihood and show how to reduce its bias by using the jackknife. They then evaluate the effect of using the resulting estimator (JVI) for training and evaluating VAEs on MNIST. This is an interesting and well written paper. It could be improved by including a convincing explanation of the relatively poor performance of the JVI-trained, JVI-evaluated models.
Decision: Accept (Poster)