Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary:
The paper introduces an RG-Flow model -- a hierarchical flow model based on the idea of the renormalization group. The experiments suggest that this model is expressive and capable of learning the disentangled representations at different scales. Additionally, the authors show how this method can be adapted for image inpainting and theoretically analyze its complexity.

Pros:
- Novel and interesting idea.
- The proposed method allows for separating the features at different scales (at low, mid, and high levels) due to the hierarchical structure of the model.

Concerns:
- The paper would benefit from additional experiments and comparison to other methods. For example, you can experiment with the datasets specific for the analysis of disentangled representations, e.g., Shapes 3D, MPI 3D, etc.
- The paper lacks the quantitative assessment of the degree of disentanglement of learned representations. I suggest to refer to the paper [1] for the list of disentanglement metrics. While they were invented for VAE-based models, you can adapt it to your method.
Similarly, you could add the assessment of the inpainting quality apart from the visual evaluation in Figure 6.   

Questions: 
- Did you measure the quality of generated images, e.g., by computing FID or Inception score? While the samples for the CelebA dataset look good, the samples for CIFAR-10 (Fig.8) are not impressive.
-  I did not find any details on the experiments on the 3D Chair dataset apart from Table 1 in the Appendix. Was the setting the same as for CelebA and CIFAR-10 datasets? Is it possible to compare the results of RG-Flow to Real NVP for the 3D Chair dataset?

Comments:
- I suggest adding the paper [2] to the related work; this paper introduces Multiscale Entanglement Renormalization Ansatz (MERA), a structure similar to your network.
- Take a look at the paper [3], bridging the entanglement renormalization and wavelets. 

UPD: I am satisfied with the authors' response, and therefore I increase the score. 

References:

[1] Locatello, F., Bauer, S., Lucic, M., Raetsch, G., Gelly, S., Schölkopf, B., & Bachem, O. (2019, May). Challenging common assumptions in the unsupervised learning of disentangled representations. In international conference on machine learning (pp. 4114-4124).

[2] Vidal, G. (2008). Class of quantum many-body states that can be efficiently simulated. Physical review letters, 101(11), 110501.

[3] Evenbly, G., & White, S. R. (2016). Entanglement renormalization and wavelets. Physical review letters, 116(14), 140403.
review_title: Interesting idea, but needs more experiments

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposes a new architecture for flow based generative models. The model imposes a hierarchical structure over information at different scales. The paper shows that the hierarchical structure results in disentangled features at different levels of abstractions. The paper claims that the approach is based on the renormalization group


Strengths:
+ The overall structure of the approach is intuitive, and the writing/figures are understandable 
+ I like how the approach naturally produces disentangled features at different levels of abstraction. 
+ Figure 9 demonstrates how the sparse prior leads to better disentanglement than the Gaussian prior. Due to the rotation invariance of the Gaussian prior, the hidden dimensions of other flow based generative models like Real NVP don't have grounded semantic meaning.
+ The hierarchical approach shows a small improvement over 


Weaknesses:
- (Intro/related work) There should be more discussion how this paper relates to Neural Network Renormalization Group (Li & Wang). Some more background/examples of the renormalization group in physics / meaning of the disentangler/decimator would be helpful

- There is no evaluation of the image inpainting performance. Some other papers (Structured Output Learning with Conditional Generative Flows ,Lu & Huang) use PSNR. 
- Constrained NVP seems to be too weak a baseline for the image inpainting task. Are there more reasonable comparisons? Since the paper claims a log(L) vs L^2 improvement, timing or flop counts should be used to compare to Real NVP. 
- The paper provides limited quantitative results and comparisons. Only qualitative comparisons are provided on the in-painting task. There are no quantitative results provided in the main paper, and only results on CelebA, CIFAR-10, and 3D chair in the appendix.  There is no discussion of the 3D chair experiment in the appendix.

Some minor points:
* Notation in Eq 5. should be improved for better understanding
* What is the reason for the checkerboard artifacts shown in the progressive generation (Figure 3)?  
* While theoretically each pixel only depends on log(L) latent variables

Update: The authors response has addressed my major concerns. Particularly, the revised version is more clear about how this paper relates to prior work, more comparisons/metrics are added, and issues I had related to clarity were addressed. The revised version has significantly expanded the scope of experiments, which better demonstrate the advantage of hierarchical features for normalizing flows. I have update my rating to marginal accept
review_title: RG-Flow Review

final_score: rating: 5: Marginally below acceptance threshold
review_content: ## Summary

The paper proposes a method, named as RG-flow, which combines the ideas of Renormalization group (RG) and flow-based models. The RG is applied to separate signal statistics of different scales in the input distribution and flow-based idea represents each scale information in its latent variables with sparse prior distribution. Inspired by receptive field from CNNs, the authors visualize the latent space representation, which reveals the progressive semantics at different levels as the instinctive expectation.

## Strengths

+. The proposed method presents a simple method to combine renormalization group and flow-based models.

+. The visualized receptive field of latent space representation shows meaningful semantics at different levels, which is verified on CelebA.

## Weaknesses/Concerns

-. It seems like a combination of existing methods, including Flow-based methods, renormalization group and receptive field. Compared with the two mentioned NeuralRG papers(Li & Wang, 2018; Hu et al., 2020), what are the differences for the renormalization group part? The main contributions of the proposed method are not so clear.

-. In the flow part, the high-dimensional visual variables are modeled by a sparse prior distribution, while the visual results (e.g., Figure 4, 5, 6 in the main paper) are with limited diversity. Does mode collapse happen during modeling the distribution?

-. It could be too strong that enforces the high-dimensional features at different layers to follow a same sparse prior distribution. Though the proposed method achieve acceptable results on CelebA which is an almost aligned face dataset, it fails to model the images in the wild (e.g., CIFAR-10 on Figure 8 in supplementary). Does it mean the proposed method works only on aligned/structured scenes? Specially, compared with the style-based generator[1,2], where the sparse prior distribution is enforced only in the input z latent space, I don’t find superiorities of the proposed method.

[1] A Style-Based Generator Architecture for Generative Adversarial Networks, CVPR 2019.

[2] StyleFlow: Attribute-conditioned Exploration of StyleGAN-Generated Images using Conditional Continuous Normalizing Flows, arXiv 2020

-. Given the visualization methods can have a great effect on the results, Fig. 9 is a bit weak to show the advantages of breaking the spherical symmetry of the style latent. What are the clusters in latent space, visual space and target?   

-. The experiments are conducted on a low-resolution setting (i.e., 32x32 on CelebA). Does it work on higher resolution?

-. There is no qualitative and quantitative evalution to compare the performance or show the improvement gained by the proposed method.

-. In order to verify renormalization group helps on learning more meaningful semantics at different levels, it's better to show an ablation study to compare the visualization of receptive field between w/ RG and w/o RG. 

-. What’s the meaning of y in equation (5)? 


## Overall Recommendation

Based on the above weaknesses/concerns, I rate the paper "Marginally below acceptance threshold". 



review_title: Reasonable extension of flow-based method but contributions not significant

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary:
The submission suggests a generative model with a normalizing flow based on a renormalization group idea. This gives rise to a multi-scale latent representation by arranging bijective transformations hierarchically with depth scaling logarithmically with the input dimension. Extensive experiments on CelebA are conducted, illustrating the latent feature on different scales and image inpainting experiments.

Positives:
The paper includes substantive experiments that (i) visualize the latent features on the different hierarchical levels, (ii) illustrate mixing of latent variables across different levels, (iii) show good performance in error correction. 
The proposed RG-Flow network architecture is very interesting indeed and is new for the considered tasks as far as I am aware. The complexity of the approach can improve from O(L^2) for previous normalizing flows for image inpainting to O(log L). This is a very significant improvement (but it is not clear to me how well the architecture performs for corruptions that are not just in a single local region).

Negatives:
Similar flow architectures have been used before in different contexts (mostly physics) as also mentioned in the paper, but I feel that it is not so clear how the proposed normalizing flow model differs from previous work with a renormalisation group (NeuralRG). The role of symmetric priors for disentangled representations has been studied previously (using information theoretical ideas, for VAEs etc) and I also feel that it should be clarified how the sparse Laplace prior differs from previous work. 

Recommendation:
This appears a borderline submission for me, but my confidence for assessing it is not that high. The experimental section is solid and I would consider increasing the score if the differences to previous work are made clearer.

Comments:
Does the choice of the temperature for the prior has a large influence on the learned features?
How does the computational complexity and running time of the RG architecture compare with other flow based generative models like Glow/(plain) RNVP?

review_title: Official Review 2

Meta-review: This paper proposes a hierarchical flow-based generative model to learn disentangled features at different levels of abstractions.  The key technical contribution is a combination of renormalization group and flow-based models. The reviewers do find the idea interesting. However, the merit of the work with respect to StyleGAN and StyleFlow has not been well established. AR3 made the following comment:  “Specially, compared with the style-based generator[1,2], …, I don’t find superiorities of the proposed method.” The authors responded to the comment briefly (but not convincingly) in their rebuttal. There is no mention of it in the revised paper. A proper account of the issue would require major revision to the paper.
Decision: Reject