Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper proposes adding additional flow layers on the decoder of VAEs. The authors make two claims
1. The proposed model achieves better image quality than a standalone Glow.
2. The proposed model is faster to train than Glows.
The intuition is a VAE can learn a distribution close enough to be target distribution, and the Glow only needs to do much less work than standalone Glow, hence faster. Some positive results are reported in the experiments, including better image quality, faster training time, and the Glow indeed sharpens the output of VAEs. 

The paper indeed has some good results, particularly they can achieve it only with single-scale Glows with additive coupling layers. However, I think the claims are not sufficiently supported. Taking point 1 as an example, it is not clear to me why VAE+Glow is better than a standalone Glow. Imagine two models

M1: VAE+Glow (proposed in the paper)
M2: Glow0+Glow (standalone Glow)

sharing the last "Glow" part. M1 is better than M2 implies "VAE" is more powerful than "Glow0", which I doubt. Similarly, for point 2, it is not clear to me why "VAE" is faster than "Glow0". I think comparing the proposed model with IAF make more sense, because the proposed model just adds flows to the decoder and the prior. But the relationship with Glows needs to be considered more thoroughly.

Another confusing detail for me is the two-stage training in Sec. 3.4. The explanation "likely because the Glow layer is unable to train efficiently with a changing base distribution" doesn't make sense. Because IAF does successfully train their q-net without 2-stage training. There might be other reasons?

The baselines are not strong enough. Most importantly, Flow++ [1] reports a likelihood 3.08 on Cifar10 with standalone flows, which should also be a part of the baseline. I also wonders whether the proposed model benefits from deeper model, like standalone flows do. Will standalone flows surpasses the proposed model as the number of layers goes to infinity?

[1] Ho, Jonathan, et al. "Flow++: Improving flow-based generative models with variational dequantization and architecture design." arXiv preprint arXiv:1902.00275 (2019).

Finally, the paper is somewhat incremental. Particularly comparing with VAE-IAF, where this paper just adds flow layers to not only q but also p.

Update
=====

After reading the rebuttal I found some of my concerns are unaddressed. (regarding to the two-stage training, and the novelty)

Point 1 is still not answered. The answer I expect to have is how "The interaction between two models when they are being stacked may affect the overall performance in such a way that it is more than just the sum of its parts." Why are these two models perform particularly well when combined? The purpose of my initial question is for some in-depth analysis and intuition / theory.

Therefore I will keep my score unchanged.
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This paper proposed a variant of the variational autoencoder, specifically by using a simplified normalizing flow model as the decoder. When compared with Glow, the proposed method is simpler and more efficient to train. The authors applied their algorithm on a few datasets, and showed better or competitive performance, both qualitatively and quantitatively.

This paper is in general well written. I think the idea looks interesting, although the novelty is a bit incremental, as it basically combined the two well-known models (VAE and Glow). The experimental results showed the promise of the new method, which could be more convincing if applied on larger scale datasets. My detailed comments and questions are as follows.
1. Regarding training, the authors decomposed it into two phases, i.e., training VAE first and then Glow. The authors also mentioned that jointly training resulted in images with poor qualities. I am curious about how the authors designed the Glow model: Intuitively a larger model may have more modeling capacity, but at the cost of computational cost. Some ablation studies or explanation could be helpful.
2. The authors claimed at the beginning of Section 3 that the their normalizing flow "should not need to do as much work as a full marginal normalizing flow model such as Glow". I am wondering how the performance will be for the Glow used, if without the VAE part? 
3. For the bits/dim results for Glow in Table 2, was it computed by yourself or just from the Glow paper? I saw the FID score was obtained by yourself.
4. For the Glow used in experiments, how does its architecture compare with the one used in the original paper?
5. I am a bit surprised to see the results in Table 3 and Figure 3, as Glow has a better FID score but the overall image quality is worse. Is it related to the size of the Glow used?
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: The paper proposes a combination of a conditioned flow-based model with a VAE. The main contribution of the paper is a two-phase training that allows to train the model. Unfortunately, a joint training of the model failed. In general, combining VAEs with flow-based models is an important research direction. Unfortunately, the paper is not clearly written. A lot of details are missing that makes the paper impossible to reproduce and fully understand. Moreover, the most interesting part about training procedure, is discussed superficially. Finally, I find the comparison to SOTA methods disappointing. The authors skipped many recent papers. I do not expect to see SOTA results among all models, but at least comparable results within a group of approaches. However, the authors provide only a small subset of models that makes me wonder whether they are aware of other papers.

Remarks:
- The following statement is not fully correct:
"In our implementation we use a normalizing flow similar in structure to Real NVP Dinh et al. (2016) (which is a special case of autoregressive normalizing flows Papamakarios et al. (2017)), as it allows both efficient training and sampling"
Real NVP is a bipartite flow, while Masked Autoregressive Flow is an autoregressive flow. In special cases, these two flows coincide, but their original implementations are different.

- The paper misses a lot of important details. For instance, a reader needs to figure out what is the objective function. Further, the authors do not mention how they deal with images represented by integers. Do they use dequantization as in other papers (e.g., Theis, L., van den Oord, A., and Bethge, M.  A note on the evaluation of generative models. In Workshop Track,ICLR, 2016)?  These are very important details to fully understand the proposed approach. Providing a very general diagram (Figure 1) and generic equations (2-4) are not sufficient. Currently, there are different implementations of flow components, and describing them would be definitely beneficial. Moreover, it is important to show how conditioning is used in the flow model.

- Section 3.4 is incredibly interesting part of the paper and it should be further analyzed. The proposed two-phase is reasonable, but it leaves an open question why a joint training fails. I agree with the authors that a possible explanation is training instability. Nevertheless, I would be more than interested in seeing a more thorough analysis of this phenomenon.

- I do not understand why the authors skipped reporting bpd for CelebA. 

- In general, the comparison in Table 2 is far from being complete. For instance, please see the following paper:
Ho, J., Chen, X., Srinivas, A., Duan, Y., & Abbeel, P. (2019). Flow++: Improving flow-based generative models with variational dequantization and architecture design. arXiv preprint arXiv:1902.00275.
On Cifar10, current non-autoregressive models are able to achieve 3.08 bpd (Flow++)  and 3.11 bpd (IAF-VAE). This is much better than the reported 3.17 bpd.

- In Section 4.2.2, first line, a number to a figure is missing.

======== AFTER REBUTTAL ========
I would like to thank the authors for their rebuttal. However, I am not fully satisfied with some answers. First of all, the paper should be clearly written so that a reader could be able to find all necessary details and seamlessly implement the presented idea. If some details do not fit the main text, then they should be included in an appendix. Second, the objective function is an important component of any ML problem and, therefore, it should be included in the paper. Sometimes a verbal description is not sufficient. Third, by conditioning a flow I meant whether the based distribution was conditional or/and a single flow layer was conditioned on z. Currently, there are multiple ways to condition and I was curious which of them was used by the authors. Fourth, I do not believe that achieving nicely looking images is the ultimate task for generative models. However, this is an open discussion. Fifth, I agree, the discussion on why the joint training failed is an incredibly interesting question.
Again, I really appreciate all the answers, and I believe that the authors did their best to improve the paper. However, as a reviewer I must look for novelty and evaluate how the paper is readable for others. In my opinion, combining VAE with GLOW is not sufficiently novel idea. If the paper was very clearly written, I would think about rising the paper. However, right now, I decide to keep my original score.
review_title: Official Blind Review #1

Meta-review: The paper received mixed reviews: WR (R1,R3) and WA (R2). AC has carefully read reviews and rebuttal and examined the paper. Unfortunately, the AC sides with R1 & R3, who are more experienced in this field than R2, and feels that paper does not quite meet the acceptance threshold. The authors should incorporate the comments of the reviewers and resubmit to another venue. 
Decision: Reject