Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors examine the amortization inference generalization gap with the motivation of improving performance on lossless compresseion and image modelling. They introduce the reverse sleep amortized inference and reverse half-asleep amortized inference, components of a modified wake-sleep algorithm, and use these algorithms to train a VAE for MNIST, CIFAR10 and lossless compression experiments.
strengths_and_weaknesses: Strengths:

The investigation is well motivated by the application to lossless compression.
I thought that the intuition presented for the aspects of the lack of generalization in section 2 were good.

Weaknesses:
I did not think the intuitions in section 2 were well supported by they toy experiments presented.

I did not find the exposition clear. I often found myself confused when the authors explained background that I am already familiar with. The authors would introduce a rearrangement of the ELBO, as in equation 2, only to immediately introduce another rearrangement in equation 3 leaving me wondering why early equations were introduced. I did not find the explanation of the variational family starting on line 56 very clear. 

The method used in the paper is very similar to the one used in [4] Cemgil et al. 2020, as noted in the related work. Unfortunately, there are not a lot of additional contributions to distinguish this paper from [4], other than a reinterpretation of the method.

The authors presented a very long explanation for what is a relatively simple idea: they want the inference network to converge to the correct posterior if they model converges. This does not happen in the standard ELBO because the expectation is only over the training data. They authors bootstrap their own model distribution by effectively adding an expectation over model simulated data to the ELBO. Under the very strong assumption that the model converges to the true distribution, you will me minimizing the KL between all true posteriors and q (which isn't amortization anymore), and with a flexible enough q parametrization, you will satisfy your requirement. 

The model converging to the true distribution is a very strong assumption. Not much investigation is presented for whether this is a reasonable assumption (other than the fact that the inference network dominated the error in a toy example) or issues that occur when this assumption doesn't hold, which would be almost all of the time. Limitations are not discussed. A modification (half-asleep) is presented to address this issue.

The experimental section is not very strong, only applying the method to MNIST, CIFAR10 and a single compression experiment. Standard regularization techniques were not compared against. In fact, no other training methods other than those introduced in the paper were examined, including IWAE mentioned in the related work. 

I think it's a good idea to examine this particular aspect of VAE overfiiting but I recommend major revisions before conference acceptance.
questions: How does a VAE perform when trained in one step (all of the parameters) using the half-asleep objective, ie. adding the additional expectation to the standard ELBO training, as oppose to the two-stage wake-sleep method? 

The test error in figure 5 is still increasing with each epoch. Is this still an indication of overfitting?

On line 239 you state that you sample from q(z) instead of p(z). How is this done? Do you Monte Carlo marginalize out the training data from q(z|x)p(x)?

Typos: 
Line 117: "the VAE that is described in Figure 2". No description is given here.
Equation 12: The expectation is written over the joint p(x,y) but I think you want p(x).
Line 240: Equation ?.
limitations: No discussion of the limitations of the method was present.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors investigate the generalization of (amortized) variational auto-encoders in terms of their test likelihood. It is noted that "overfitting" in VAEs is largely contributed by the amortized inference network. To mitigate such overfitting, the authors develop reverse sleep and reverse half-sleep methods inspired by the wake-sleep algorithm, leading to better compression of the data distribution.
strengths_and_weaknesses: Strengths:

* The authors develop on a likelihood-based characterization of what it means for VAEs to overfit. The demonstration on a toy example which clearly demonstrates the overfitting phenomenon is much appreciated.

* The step-by-step development of the method with supporting experiments provides a relatively clear picture.

Weaknesses:

It would appear that the method specifically caters to optimizing for best compression performance. Even in their own experiments, the authors note that without the half-sleep correction in the sleep phase, the generated samples may be degraded and not representative of the true data distribution. 

This makes me wonder that the notion of test likelihood for generalization remains very limited in scope, and in principle only caters to compression performance (this probably would not be an issue in the case of a well-specified model, which is virtually impossible to achieve). In that sense, the insights developed in this paper may not hold across broader family of generative models.

In light of such missing broader evaluation of direct implications, I will be keeping my score to a weak accept.

Minor:

* Is there a typo in the summands in Eq. (8) where you need expectation w.r.t $q^*(z \mid x^\prime_m)$ and $q_{\hat{\phi}}(z \mid x^\prime_m)$ instead of just conditioning on $x$?

* Missing closing ")" in Eq. (5).

* Line 153: Typo in "Mon-Carlo".
questions: My comments here are mostly a follow up to the main weakness previously mentioned.

* The authors interestingly note that standard VAE training with ELBO (i.e. the wake phase) only has access to a finite training dataset. The sleep phase allows for virtually unlimited training data, and hence mitigates overfitting. One can subsequently think of how does the applied fix to the amortized network behave for the following applications other than compression to name a few:
  * out-of-distribution detection/generalization
  * sample generation
  * semi-supervised learning

* Another question to explore would be does this make VAEs better utilize the latent space? From the compression results, perhaps the answer is promising and tending towards yes. Is it more robust to the choice of the dimensionality of the latent space as a consequence of improving the amortization gap?

I am curious if the authors have a plan for this, or could provide a small proof-of-concept for something beyond compression. 
limitations: Yes.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper studies the generalization performance of Variational Auto-Encoders during test time, measured by the test likelihood. Generative model generalization gap and amortized inference generalization gap are introduced and it is empirically shown that the degradation in generalization is mostly dominated by the inference model. The authors then propose a new variational inference scheme named “reverse half-asleep” which helps close the gap caused by the inference model. Finally, the improvements in generalization and compression rate of using the new scheme over the traditional VAE are demonstrated on Binary MNIST, Grey MNIST, CIFAR10 datasets.
strengths_and_weaknesses: The paper is very well written. I can understand most of the contents in one read.

The novelty of the paper is the introduction of using reverse KL in the sleep phase during wake-sleep training. It is surprising to me that people have not done it previously.

I think that this is an important paper. The technique is pretty straightforward to implement, hence would be easily adopted in practice.

My main concern, ~which is not addressed in the paper~, is on how using “reverse half-asleep” affects other metrics (like accuracy) during both training and testing phases. **Updated:** The authors pointed out in the response below that the performance on downstream classification tasks can be found in Appendix B.

It is only shown through Figure 2 that the overfitting of the VAE is mainly dominated by the overfitting of the amortized inference network. It would make the paper more technically sound if there is an analytical proof for that observation.
questions: The improvements might also be shown on CIFAR100 or ImageNet. I just wondered if the authors have performed experiments on those larger datasets. Is the trick for CIFAR10 in line 239 helpful in those settings?

Some minor points:
+ Line 140: missing the Equation number.
+ Line 108: missing the negative sign of the KL estimation.
+ It is clearer to include labels for axes in the figures.
limitations: Nothing to report.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper focuses on the generalization gap in amortized inference trained using the ELBO objective. The generalization gap can come from either the generative model p or the inference network q not generalizing from training data to test data. This paper argues that q is mainly responsible for the generalization gap, which is supported by an experiment using the VAE model and the MNIST dataset (Figures 1 and 2). The proposed fix is to train on $E_{m(x)}[ELBO]$ instead of $E_{\hat p(x)}[ELBO]$ where $m(x) = \alpha p_\theta(x) + (1 - \alpha) \hat p(x)$, i.e. "augment" the empirical dataset by samples from a well-trained model. Experiments suggest that this indeed leads to better test ELBOs, which in turn leads to better compression rates.

I gave this paper a 5 but am willing to increase the score if the authors address some of the comments below.
strengths_and_weaknesses: # Originality
As far as I know, this is an original view on the problem and an original solution.

# Quality
I'd like to see some of the following concerns addressed.

I am not entirely convinced that the generalization gap is mainly due to the amortized inference network, as claimed. I understand that in the case of this particular VAE architecture and the MNIST dataset, this is the case, as shown in Figure 2, but what about different model-dataset settings? What if the generative model family is more expressive, e.g. a PixelCNN? Surely some of p must be contributing to the generalization gap there. Another reason to believe this is that for more complex datasets like CIFAR10, there doesn't seem to be overfitting, e.g. as seen in Figure 5c.

The solution to the problem (using reverse half-asleep) only works if there is no generalization gap due to p. If there is a generalization gap due to p, the solution is ineffective. As I understand it, the proposed method is to 
1. Train using $E_{\hat p(x)}[ELBO]$ and then
2. Fine tune $\phi$ (parameters of the inference network) using $E_{m(x)}[ELBO]$ where $m(x) = \alpha p_\theta(x) + (1 - \alpha) \hat p(x)$.

As suggested in lines 189-191 ("Intuitively, this is because ..."), this solution is effective because the inference network gets to be trained on a dataset, augmented by samples from the model, $p_\theta(x)$. But if the model $p_\theta(x)$ is trained on $\hat p(x)$, how good can it be? In the extreme case of an infinitely expressive $p_\theta(x)$, it will converge to $\hat p(x)$ and $m(x) = \hat p(x)$, rendering reverse half-asleep to be the same as normal ELBO.

# Clarity
Overall, the paper is clearly written. The compression section (Section 5) is difficult to understand without further context (e.g. what is the stack in Algorithms 1 and 2?), but the high level points are clear. Plot labels would have been useful. Small typos:
- Line 153: Monte Carlo
- Line 240: Equation number missing
- Line 292 Sacrificing 

# Significance
Generalization of amortized inference is an important problem.
questions: I'd also appreciate authors' thoughts on the following points.

The paper claims that KL(q || p) is better than KL(p || q) because of compression. However, KL(p || q) also comes with its advantages, for example that it makes q spread its mass more and hence it can end up learning a better proposal for importance sampling.

In Line 107, it is claimed that $\log p_\theta(x)$ cannot be calculated explicitly and hence the generative model generalization gap cannot be estimated well. While it cannot be computed exactly, it can be estimated by using importance sampling, which is equivalent to the [importance weighted autoencoder (IWAE)](https://arxiv.org/abs/1509.00519) objective. In light of my previous questions about the amount of p's vs q's contribution to the genralization gap, it might be nice to see this estimation of the generative model generalization gap.

There doesn't seem to be overfitting in Figures 3 and 5c.

What if we train using reverse half-asleep from the start instead of training using ELBO first, and only then fine tuning $\phi$ using reverse half-asleep?

In the paragraph starting in line 228, data from $p_\theta(x)$ is simulated by drawing $z \sim q_\phi(z) = \int \hat p(x) q_\phi(z | x) \mathrm dx$ instead of $z \sim p(z)$ before drawing $x \sim p_\theta(x | z)$. Is this really that different from just drawing from $\hat p(x)$ directly? Especially when looking at Figure 5c, it seems like there is underfitting going on, rather than overfitting. What would have happened if we used $p(z)$ instead of $q_\phi(z)$?
limitations: N/A
review_title: null

Meta-review: metareview: The paper studies generalization in generative models and highlights two factors for the lack of generalization, generalization gap of the model itself and generalization gap in the ELBO objective caused due to amortized inference. The authors experimentally show that the latter is the more prevalent reason. To alleviate that the authors propose a reverse wake sleep algorithm which is based on training from a mixture of samples from the true distribution and the model's distribution. This latter algorithm is the primary contribution of the paper which is a novel algorithm as recognized by the reviewers. The reviewers have found the experiments compelling and the conceptual contribution significant but easy to implement thus projecting general adoption. Reviewers unanimously agreed upon the contribution of the paper being above the bar for Neurips. 
confidence: Less certain
Decision: Accept