Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper presents dense flow that are normalizing flows with additional latent variable after each transformation. This new architecture is tested on generative modelling of images and yield results close or better than concurrent methods whereas dense flows require lesss computational power for training and are relatively fast at generation time.

main_review: Originality:
The method suggested in this paper of adding latent variables to NFs has already been proposed. Authors say the difference with their methods is that they add latent after each transformation steps but I don’t believe this is significantly different from this other work. However this paper demonstrates those additional latent combined with attention mechanism and multi-scale like NF architectures are useful for reaching great generative performance. 

Quality:
Experiments show the model is well performing in terms of likelihood scores, sample quality and computation speed. However results regarding speed are not benchmarked very seriously as methods are using very different hardware which makes it hard to have an exact feeling on where each method stands. In particular it is very difficult to assess the memory usage of different methods and how the proposed method stands with respect to others.
It is not clear why table 3 is only providing FID scores for CIFAR10 and not other Imagenet and Celeba. 
I have also the following remarks:
• L47-49: I don’t agree with your claim about the expressivity of NFs. You can always increase the capacity of the neural networks used to do the transformation although those transformation are one to one.
• Some indices not correct in example 1.
• Section 2.2: You should maybe describe multi scale architectures.

Clarity:
Overall the paper is well written however I have the following remarks. 
• 148-156: this is not very clear.

Significance:
It is not clear whether the proposed method is solving any existing problem as many methods exist for creating good generative models. In particular the complexity of the architecture is relatively high and it is not clear if the nice results achieved are really coming from any new idea or just a very nice combination of existing work. If it is only the latter I would expect to see stronger results.

limitations_and_societal_impact: /
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This work addresses the limitation posed by the bijectivity constraints in normalizing flows by padding  noise at each layer of the flow. 
MC sampling is used to approximate the likelihood where 1 MC sample is used for training and multiple MC samples are used for likelihood evaluation. Experiments for density estimation are performed on CIFAR-10, ImageNet-32 ,CelebA -64, ImageNet-64 datasets where the method yields state of the art density estimates. The method is shown to be computationally efficient with better sample quality.
main_review: * The work is incremental as the idea of padding with the noise dimensions to overcome the constraints associated with the dimensionality preservation have been studied in INNs and VFlow. 

* Overall the paper is well written and the experiments are detailed and easy to follow. Relevant details wrt to the experimental setup are included.  Tables 1, 2 and 4 include relevant comparisons to the prior work for density estimation, computational complexity and ablations of the proposed work respectively. The FID scores in table 3 should be expanded with comparison to other flow models that yield better sample quality e.g. [1,2].

* The work proposes an intra-coupling module in section 2.2 . The details of the same can be elaborated as the architecture and the operations performed are difficult to follow. The self attention module is similar to that of Flow++. A discussion regarding the differences of the proposed approach with that of the existing self-attention module (of Flow++) would be helpful. 

* Choice of the number of MC samples for different datasets in line 211 can be expanded and the variance thereof discussed. How is the training effected due to 1 MC sample?

* The related work section seems to be minimal and should be expanded to include relevant literature on normalizing flows and explicit generative models in general. The related work of [1,2,3] can be referred for details. 


[1]. Bhattacharyya et al. Normalizing Flows with Multi-Scale Autoregressive Priors. CVPR 2020.
[2]. Ma et al. MaCaw: masked convolutional generative flow. NeurIPS 2019. 
[3]. Yu et al. Wavelet Flow: Fast Training of High Resolution Normalizing Flows. NeurIPS 2020.
limitations_and_societal_impact: The limitations and the societal impact of the work are appropriately discussed.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes an approach to improve the modelling capacity of normalising flow based models. The proposed approach  incrementally pads intermediate representations with noise, which is conditioned on the output of previous invertible units using affine transformations. The paper reports promising results density estimation results on ImageNet and CelebA.
main_review: The proposed approach for noise augmentation is interesting and novel. However, the main concerns are,
* Motivation -- the paper does not clearly state the advantage of their approach over VFlows. In other words, why does the proposed approach perform better than VFlows? L43-51 only establishes the advantage of augmenting extra dimensions and Fig 1 only illustrates the difference to VFlows.

* Architecture of glow like modules -- The paper proposes to include Nyström attention modules, however, Flow++ already includes multi-head attention (Section 2.2 should therefore discuss the difference to Flow++ in more detail). The main advantage of Nyström attention lies in computational complexity. However, there are no ablations to study this advantage in detail. Therefore, the architectural advantage of the proposed approach over Flow++ is unclear.

* Density estimation -- the paper does not motivate why the results on CIFAR-10 do not show any advantage over VFlow (same bpd).

* Efficiency of likelihood computation -- the paper should include a detailed analysis of speed of likelihood computation as 1000 MCMC samples are needed versus flow based models which provide likelihood estimates in a single pass.

* Sample quality -- the FID score on CIFAR-10 is competitive with other flow based approaches such as, [Normalizing Flows with Multi-Scale Autoregressive Priors, CVPR 2020] which achieves an FID of 33.6 on CIFAR-10. Moreover, there is no comparison in terms of image quality to VFlow. The paper should motivate why the proposed approach fails to improve in terms of image quality compared to prior flow based models.






limitations_and_societal_impact: * The paper does not motivate the novelty in term of architecture over prior methods such as Flow++.
* The computational complexity for likelihood estimation should be discussed in more detail e.g. the trade-off of using higher number of samples vs bpd.
* The poor performance in terms of image quality must be better motivated and comparison to prior work such as [Normalizing Flows with Multi-Scale Autoregressive Priors, CVPR 2020] must be included.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes to use incremental augmentation, gradually increasing dimensionality towards the latent space, in normalizing flows. The augmentation distribution at each level is conditioned on previous feature maps. This is referred to as "cross-unit coupling". Additionally, the authors introduce a new neural architecture for use in the usual coupling layers, referred to as "intra-unit coupling".
The result is an efficiently trained flow with state-of-the-art likelihoods.
main_review: The paper extends the augmentation in [1,2] to multiple layers where the augmentation distributions are conditioned on previous feature maps in a similar fashion to DenseNets. Furthermore, a novel neural architecture is proposed for the coupling layers.

The proposed improvements are straightforward extensions of existing methodology, but with surprisingly good performance. The proposed methodology is clearly laid out, first introducing the "cross-unit coupling", before introducing the image-oriented neural architecture with two-way fusion using both convolutions and self-attention.
Experiments are quite extensive: Comparisons on likelihood performance on 4 image datasets, as well as comparisons of computational complexity and sample quality. Finally, there is also an ablation study on the different model component's effect on likelihood. 

The main strong point of the paper is the state-of-the-art likelihood performance achieved using modest computational resources compared to earlier work. Given this, I am recommending acceptance. However, the paper does have some rough edges and points that could be improved. I outline a few suggestions for this below.

Some suggestions for fixing/improving the paper:
- Line 14: "One of the uttermost goals [...]": This statement is maybe a bit strong and could be rephrased.
- Line 27: Although the connections you point out are correct, I'm not sure I'd call diffusion models as VAEs since there is no "autoencoding" going on.
- Line 29: "[...] GANs ignore the factorization of the likelihood": What is meant by this? I think the description of GANs could be improved. The inability to evaluate the likelihood comes mainly from the inability to "invert" the generation process in any meaningful way.
- Eq. 11: $e_i \rightarrow e_1$ and $\log p(e_1) \rightarrow \log p^*(e_1)$.
- Line 108: "[...] a few hundred of samples during evaluation to reduce the variance of the likelihood": From the code, it seems the importance weighted bound was used which also provides a tighter bound on the likelihood, not just lower variance.
- Table 2: Do you also have access to the number of parameters? If so, this would improve the table.
- Line 241: While one could argue about which approach is best, it is my understanding that common practice is to compute FID by comparing the training data distribution. 
- Line 263: bdp -> bpd
- Line 276: $\mathbb{E}_{\boldsymbol{e} \sim p(\boldsymbol{e}|\boldsymbol{x})} \left[ \log p(\boldsymbol{e}, \boldsymbol{x}) - \log p(\boldsymbol{e}|\boldsymbol{x}) \right] = \log p(\boldsymbol{x})$ and so is not a lower bound.  
  - I believe you want to write $\mathbb{E}_{\boldsymbol{e} \sim p^*(\boldsymbol{e}|\boldsymbol{x})} \left[ \log p(\boldsymbol{e}, \boldsymbol{x}) - \log p^*(\boldsymbol{e}|\boldsymbol{x}) \right]$



[1] Huang et al. 2020, Augmented Normalizing Flows: Bridging the Gap Between Generative Flows and Latent Variable Models  
[2] Chen et al. 2020, VFlow: More Expressive Generative Flows with Variational Data Augmentation
limitations_and_societal_impact: Limititations are adequately discussed in App. A and environmental impact in App. B. 
As for negative societal imapct, the authors mention that their research could be used for malicious purposes, they do not discuss any particular examples.
review_title: null

Meta-review: The paper proposes a normalizing-flow architecture that includes latent variables and is geared towards image modelling. The paper demonstrates good empirical performance of the proposed architecture at modest computational budgets.

There is a clear consensus among reviewers about the strengths and weaknesses of the paper: the main strength is the good empirical performance of the proposed architecture; the main drawback is the incremental novelty. Overall, the paper seems a useful contribution for generative-modelling practitioners, so I'm happy to recommend acceptance.
Decision: Accept (Poster)