Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: **GENERAL**
The paper proposes a new hierarchical architecture for VAEs. The main idea is to use invertible components that could be shared by the encoder and the decoder. However, the paper is very confusing in many parts, and the experimental studies are not too strong.

**Strengths:**
S1: The authors propose a new architecture for stochastic layers in VAEs.

S2: The idea is to use bijective layers shared by the generative and the variational parts of the VAE.

**Deficiencies:**
D1: Please correct Figures 1-5. First, the text seems to be "squashed" that hinders readability. Moreover, some words blend with each other. In Figure 5, rotating the text makes it almost impossible to read. Additionally, I believe including colorful squares or rectangles does not help; on the contrary, it makes it harder to read.

D2: The hierarchical model is presented through Figures (e.g., Figure 1) that is very hard to follow. I believe the authors have some interesting ideas, but it is completely overshadowed by rather unreadable and pretty unclear figures. It would be much better to simply express the model mathematically. At the moment, I do not follow what is the semantics of nodes, edges and colors.

D3: The text is hard to follow as well. For instance, the authors write that some components are implemented by ResNet or MLP. However, when MLP is used, and when ResNet is used? These statements are very confusing. Moreover, the authors introduce multiple notation for the same quantities that differ by one symbol, e.g., p(\epsilon_l | ...) and p(\epsilon ; ...). What is the purpose of using "|" and ";" interchangeably?

D4: The section about Residual Data Layers is written in a confusing manner. The authors introduce a two-step procedure to calculate \gamma. First, they compute an estimation of it. Then, they calculate a second estimation \delta \gamma_l. Then, they combine these two estimations to define \c_l^{\gamma}. Why do the authors mention estimations? Why are the two quantities summed? It is totally unclear and written in a clumsy fashion.

D5: The statements in Section 3.4 are very hand-wavy. The authors claim, e.g., that "The model, albeit hierarchical, is less prone to posterior collapse, since each layer is responsible for the generation of a different portion of the data". How do we know that? It is not obvious that this is the case.

D6: I appreciate the experiments provided by the authors. However, the provided comparison on CIFAR10 is hardly acceptable. First, currently using Normal distribution for the decoder is not widely used. Second, MAF is a great idea and an extremely interesting paper, however, nowadays it cannot be treated as a strong baseline.

*AFTER REBUTTAL* I would like to thank the authors for their rebuttal. I increase my score to 5. I am still unsatisfied with the presentation of the idea, because it is still rather hard to follow.
review_title: Possibly an interesting idea, but the paper is hard to follow

final_score: rating: 3: Clear rejection
review_content: 1. Summary
This paper proposes an augmentation of the Ladder VAE model (LVAE [1]) using 1. more flexible variational distributions using normalizing flows (denoted $f$) 2. an autoregressive component (because of the generative dependency $p( x_{l} | z_{l}, x_{l-1}) $ (*data layers*), although this is not directly stated as a contribution. 
It is argued the proposed factorization of the inference network (top-down inference) matches the factorization of the true posterior, that follows the top-down design of the generative model. The authors recorded competitive performances on dynamically binarized MNIST. 

2. [a] Strong Points
- the authors introduce a very flexible architecture for DGMs by combining recent improvements from the literature. 

2. [b] Weak Points
- the paper overall lacks clarity: the idea could be explained in much simpler terms
- the contributions are weak: augmenting an LVAE model using flows brings little novelty 
- the autoregressive structure introduced in the paper (which can be interpreted as an instance of state-space models [2]) is not well discussed and is contradictory with the claim that no *autoregressive layer* is used.
- an ablation study is required to disentangle the effect of 1. the flow augmentation and 2. the autoregressive structure
- aiming for a better inference network aims at tightening the variational bound, which should be measured empirically 
- the diagonal Gaussian assumption for the variational family is not necessarily so limiting [3, 4] (as assumed in the paper). 
- the results reported for CIFAR-10 are not competitive with sota models [3, 4] (SeRe-MAF) $\approx$ MAF (10) = 4.31 (original paper) $\gg$ 3.08 (BIVA [4]) > 2.91 (NVAE without flow [3]) .

3. Recommendation
Unfortunately, I recommend rejecting this paper.

4. Recommendation Arguments 
SeRe-VAE combines multiple architecture improvements (top-down model (LVAE), flexible variational distributions using flows, autoregressive components) into a final model, which is tested using a single dataset (excluding cifar10 results which are not competitive) and without performing an ablation study. This prevents the community from understanding the effect of each of the architecture choices and does not guarantee that such an architecture could be effectively adapted to other contexts. 

5. Questions to the Author
- Figure 1: why considering the prior independent? the generative model adopts a hierarchical structure $p(z_l  | z_{l+1})$

6. Feedback 
Your work is an engineering prowess, I am saddened to recommend rejection. I think your work could greatly benefit from an ablation study and from defining the model in more minimal terms. 

A few comments:
- The inference network is conditioned on $\mathbf{x}$, not the entire $\mathcal{D}$.
- you can measure the variational bound using the identity: $\operatorname{KL}(q(z | x) | p(z | x)) = log p(x) - \mathcal{L}(x)$
- please report CIFAR10 results in bit per dimension, as stated in the literature
- please report results on the more widely accepted Statistically binarized MNIST first, use dynamic MNIST as a second option.

[1] Sønderby, Casper Kaae, et al. "Ladder variational autoencoders." Advances in neural information processing systems. 2016.
[2] Fraccaro, Marco, et al. "Sequential neural models with stochastic layers." Advances in neural information processing systems. 2016.
[3] Vahdat, Arash, and Jan Kautz. "Nvae: A deep hierarchical variational autoencoder." arXiv preprint arXiv:2007.03898 (2020).
[4] Maaløe, Lars, et al. "Biva: A very deep hierarchy of latent variables for generative modeling." Advances in neural information processing systems. 2019.
review_title: Lack of clarity, novelty and empirical evidences 

final_score: rating: 7: Good paper, accept
review_content: **Contributions & Significance**:
- The proposed paper introduces an encoder-decoder architecture for hierarchical VAEs based on bijective transformations, which preserves the true factorization of the posterior in its variational approximation (which is formally shown). (High)
- The resulting method shows improved performance on dynamically binarized MNIST compared to models of similar and higher complexity (autoregressive models). (Medium)
- A variation/extension of the model using normalizing flow transformations for the decoder shows significant improvement compared to Masked Autoregressive Flow (MAF) on CIFAR-10. (Medium)


**Pros**
- The proposed hierarchical architecture only relies on the latent code of the previous layer for conditioning, instead of all preceding latent codes such as in autoregressive models.
- The method shows improvement in terms of Log-Likelihood by rearranging the stochastic flow instead of relying on a computationally expensive architecture.
- The general method seems widely applicable for VAE-like models and as such can be combined with other architectural improvements or potential future work.

**Cons**
- The experiments are a bit limited in terms of the used datasets (Dynamically binarized MNIST & CIFAR-10).
- Sections 3.1.1-3.1.3 could be written more clearly. In particular I think the notation of the parameters is overly confusing (e.g. $\alpha_l$ and $\beta_l$ as functions). For 3.1.1 maybe just directly use $(\mu_l, \sigma_l)$?
- It is unclear how the number of data partitions L, their structure & order are selected. Does that have an effect, how large is this effect and what would the general recommendations be? What about the extreme cases (L=D, L=1)?

**Style**
In general the paper is well written and reasoned for. 

**Minor Comments**
- The positioning of Figure 6 seems a bit awkward. 
- Typo Figure 2: Prior Layers: I assume $p(\epsilon|z_2) $should be $p(\epsilon|z_1)$

**Summary**
The paper is well written, has a promising and sound approach and seems generally applicable. As such I vote for accepting it. However, it could still benefit from some improvement in terms of clarity and number of experiments (i.e. other data sets).

review_title: Interesting and promising paper with potential for improvement in terms of clarity and experiments.

Meta-review: The paper proposes a variant of the hierarchical VAE architectures. All reviewers felt that the paper's clarity was lacking. While the authors made very significant improvements during the feedback phase, which were recognized by reviewers, the paper could use a revision that takes clarity into account from the ground up. I also think that the ablation studies should be expanded (if SOTA is not the goal, then science should be), e.g., compare to the setting in which q does not share the bijective layers with p.
Decision: Reject