Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper extends Residual Flows from ResNet to DenseNet. In addition, the authors further improve the performance by modifying the Lipschitz activation. 
main_review: The paper is well written, and the presentation is clear. However, I feel the contribution is incremental, and the performance gain is marginal. 

1. Since DenseNet has the same residual form F(x) = x + g(x), the authors inherit all important components (fixed-point iteration, Skilling-Hutchinson trace estimator, and "Russian roulette" strategy) from Residual Flow. Therefore, the extension to DenseNet is not complicated, which relies on the Lipscitzness of convex combination. In addition, the CLipSwish only modifies the original LipSwich activation slightly. 

2. Given the marginal performance gain (e.g., Table 3), it is beneficial to report both the memory and actual running time for both Residual Flow and i-DenseNet. Due to the sequential structure of DenseNet, I wonder if the DenseNet with the same parameters is more expensive than Residual Flow in practice.
limitations_and_societal_impact: The paper discusses the limitations and potential societal impact in Section 5.4.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper proposes to use a DenseNet as the residual function of Residual Flows. The paper carefully describes how to ensure the Lipschitz constant is bounded when performing the concatenation of representations inherent in DenseNets. The paper also introduces a new activation function called Concatenated LipSwish that may be of independent interest. Finally, empirical results show that the proposed i-DenseNets perform better than Residual Flows and other baselines.

main_review: **Summary of review**
Overall, this is a clearly written paper that proposes a novel model architecture, provides important insights into the choices and architecture, and gives good empirical results. 

**Strengths:**
- Very clear discussion of prior works and core contributions including the learnable weight concatenation.

- Proposed a careful incorporation of DenseNets as residual function fo Residual Flows where the Lipschitz constant can be controlled.

- Proposed a new trainable activation function called concatenated LipSwish that preserves more information flow.

- Outperforms standard baselines including Residual Flows and Flow++ when using uniform dequantization.

- Interesting and useful analysis of activation functions, concatenation depth, and DenseNets vs ResNets to practically understand some key hyperparameters and architectural constraints.

**Weaknesses:**
- The actual empirical benefits over Residual Flows seems small (though it does seem to improve in all cases) so it is not clear that i-DenseNets should always be chosen over Residual Flows.

- Related to above, no training time comparison was given.  Could you give wall-clock training times for Residual Flows vs i-DenseNets? This may be a key factor in determining which one to use in practice.  Also, could you provide a comparison for time for inference at test time?

- The novelty is medium.  The analysis of CLipSwish is very good and may become a standard.  However, the other ideas are mainly careful synthesis of other ideas.

**Other comments or questions**
- I'd suggest emphasizing the difference of $h{n+1}$ more.  Maybe use a different symbol since it is very different then the other $h$'s.

- I did not understand "learnable weight concatenation" until the section that gives the details.  Maybe try to describe that it is related to making the Lipschitz constant less than 1 for the dense layer.

- It may be helpful to mention immediately that the CLipSwish function doubles the channel number. This is mentioned at the end of the section but having it earlier may help understanding.

**Typos or other minor comments**
- L320 "boost performs" -> "boosts performance"


limitations_and_societal_impact: - Reasonable discussion of possible negative societal impact.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes a parameter-efficient DenseNet block and a CLipSwish (concatenated LipSwish) activation function for use with the DenseNet blocks in the residual flows setting.  The paper outperforms prior art (with uniform dequantization) in density estimation. Results are reported on CIFAR10 and ImageNet32 with rich ablation studies.
main_review: The idea of using densenet blocks and constraining its Lipschitz constant in the residual flow setting is interesting and novel. The paper is clearly written and easy to follow, with a few exceptions listed below. The work is quite self-contained and seems not to have loose ends. The main motivation for adopting densenet blocks is improving parameter efficiency and density estimation performance; both goals have been achieved according to the experiments.

A few questions and remarks for improvement:

- Fig.1 violet part is not clear
- Eq.2 uses W to denote matrix (norm), while Fig.1 uses W to denote (convolutional) layers. While this is in line with the prior art (Invertible ResNet), perhaps the notation could be adjusted for clarity.
- L76 missing subscript _2 of the norm
- L93 could benefit from explaining how this is the same as Eq.2 from the DenseNet paper ($x_l = H_l([x_0,...,x_{l-1}]$)
- L117,122 usage of 'dividing' is not precise
- L134: missing {} around etas
- L141 missing explanation for the constant
- L162 in relation to Appendix:L584-586 could elaborate on the computation of the exact value (perhaps with a plot?)
- L169 channel growth, as a term related to densenet blocks, perhaps could be mentioned earlier
- L186 refers to [4] for baselines using uniform dequantization, but to introduce a reader to types of dequantizaton, a better reference is [13]
- Sec.4.2 (in relation to CIFAR-10 and ImageNet32 experiments): how would the proposed method score in perceptual quality metrics, such as FID?
- Sec.5.1 while the analysis of individual activation functions is interesting, it would also be interesting to evaluate the whole model using e.g., a histogram of the empirical Lipschitz constant (eLhist) [1]

[1] Sanyal et al., Stable rank normalization for improved generalization in neural networks and GANs

============== Post-rebuttal comment

After reading all reviews and responses, I decided to keep my original rating (A).
limitations_and_societal_impact: Appropriately discussed in the "Societal Impact" section.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: In the article, a new normalizing flow architecture, called invertible DenseNets (i-DenseNets), is introduced. It is an extension of residual flows with two new contributions. First, the invertible residual blocks are replaced by invertible dense blocks. Similar to residual blocks, the authors proof invertibility by computing the Lipschitz constant and guarantee that it is smaller than one by a suitable rescaling operation. Second, the LipSwish activation function is replaced by its concatenated cousin, which is rescaled as well to ensure invertibility.
The method is applied to three illustrative toy datasets as well as CIFAR10 and ImageNet32, where it outperforms competing methods and sets a new SOTA for flow models with uniform dequantization. They also use an i-DenseNet to perform hybrid modelling, where is outperform the competition as well.
main_review: DenseNets are an important class of neural networks and modifying them to be invertible is an important contribution to machine learning. The authors describe their model very clearly and illustrate the architecture well. In secion 5, they also give interesting insights in why the individual parts of their model are important and lead a better performance in the experiments. However, their style to organize their argumentation is unusual. I would recommend moving some of the arguments made in section 5.1 to 5.3 to section 3 and show the respective experimental results at the beginning of section 4. Thereby, the reader would directly get a better understanding of the reason for the architecture choice. 
Regarding the experiments, the reported results are impressive. However, it would be interesting to apply this method to higher dimensional datasets as it is done in most articles that introduced the competing methods discussed in this paper. Also, running each experiments with multiple seeds would be desirable, although I acknowledge that this is often not the case in similar published articles. Also, the computational cost of training i-DenseNets especially compared to residual flows is not discussed, which is unfortunate since the fact that DenseNets require less computation than ResNets is stated as one main motivation of this work.
limitations_and_societal_impact: The main limitation is that the model has not been applied to high resolution data and the computational cost required to train it is unclear.
The societal impact regarding the generation of misinformation is discussed in the paper.
review_title: null

Meta-review: This paper proposes an extension of Residual Flows to construct invertible deep neural nets. The paper features two technical contributions: The invertible residual blocks are replaced by invertible dense blocks, and the LipSwish activation function is replaced by a concatenated version. While these contributions can be considered marginal/incremental (reviewer Lrfi) and with medium novelty (reviewer psZ5), all reviewers agree the contributions offer enough value for being accepted. 

The reviewers acknowledged the author response and they engaged in committee discussions. No serious concerns surfaced during the review process, and the reviewers reached a consensus recommendation to accept the paper.
Decision: Accept (Poster)