Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes a new type of ResNet-based Normalizing Flows. In contrast to prior studies, which required the Lipschitz constant $L$ of each layer to be less than 1, the authors use monotone operators, which they show to be strictly more expressive. A new activation function called Concatenated Pila (CPila) is also proposed. The suggested model is evaluated on multiple toy datasets as well as standard image datasets, outperforming the baseline i-DenseNet model.
strengths_and_weaknesses: ### Strengths

**Originality**

The formulation of monotone operators is something I haven’t encountered previously in the field. The concept seems intriguing and well-conceived.

**Quality**

I did not check the proofs in the appendix, but the mathematical theory in the main part is sound and comprehensible. The experiments and ablations follow the established standard of the field.

**Clarity**

The paper is very clearly written and easy to read. The authors do a good job at separating the main mathematical theory from the details and the proofs which are presented in the appendix.

**Significance**

Given the originality of the monotone formulation, the paper may inspire future work in the field of normalizing flows. The reported improvements are small, but consistent over multiple datasets.

### Weaknesses

* The paper says that *The resulting model, Monotone Flows, exhibits an excellent density estimation performance and outperforms existing state-of-the-art normalizing flow models on multiple density estimation benchmarks (MNIST, CIFAR-10, ImageNet32, ImageNet64).* This is true when uniform dequantization is used during training. However, the Flow++ paper ([34]) reports an even lower bits-per-dimension score when using variational dequantization. Therefore, the proposed model should also be trained with variational dequantization and compared to Flow++ before making the state-of-the-art claim.
* Some parts of the definition of monotone and maximally monotone operators are a bit confusing. (Further details in the Questions section below.)
* Each of the experiment was only run a single time, which makes it harder to assess the significance of the reported numbers. However, this is a standard practice in the field of normalizing flows because each training run can take a lot of time. So this is only a minor problem, given the state of the field.

questions: * If possible with time, it would be nice if the authors could run their model with variational dequantization on CIFAR10 for the rebuttal.
* Regarding monotone and maximally monotone operators: On line 79, what is meant by $u \in F(x), v \in F(y)$? From my understanding, $F(x)$ is an $n$-dimensional vector and not a set, so how can it have elements? The same applies on line 108: Given that $F$ is a function, what does it mean for $F$ not to be a proper *subset* of any monotone operator?
* Line 118: What is $G$ in the definition of $C_F$? Should there be an $F$ in the numerator?

limitations: Yes, the limitations are adequately addressed.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper 
* constructs normalizing flows based on monotone operators. Specifically, by Definition2, the paper defines $F(x)=(\frac{Id+G}{2})^{-1}(x) - x$ for a $L$-Lipschitz ($L<1$) function $G$ (thus $G=C_F$ is the Cayley operator of $F$). Next from the  Cayley operator identity $C_F+C_{F^{-1}} = 0$, $F^{-1}$ can be defined by 
$$F^{-1}(x) = C_{-G}(x) = (\frac{Id-G}{2})^{-1}(x) - x.$$
For $G$, the authors use the architecture in i-DenseNets and propose a new activation function, CPila. For training and inference, log determinant can be computed via
$$\log \det J_F = \text{tr}[\log (I-J_G) - \log (I+J_G)]$$
and estimated with some existing unbiased estimators. Training and inference schemes are also developed well using existing techniques in this work. 

* analyzes the expressive power of Monotone Flows theoretically, and compares with residual flows and implicit flows. Experimentally, the paper evaluates Monotone Flows on image density estimation tasks and shows the superiority to some normalizing flow models. 




strengths_and_weaknesses: **Strengths**:

A. To the best of my knowledge, constructing normalizing flows with monotone operator is novel. 

B. Theoretic analysis of expressive power is valid. Experimental results also shows its superiority to some residual flows. Training and inference schemes based on existing techniques are also valid. Moreover, based on the observation that LipSwish suffers from the vanishing gradient problem, the paper proposes a new activation function CPila, which is also a novel contribution. Ablation studies also support the effects of CPila. 

C. I think Monotone flows is a valid and novel contribution to the community of normalizing flow models.

**Weaknesses**:

A. Some notations in the paper are quite confusing. In Theorem2, the authors say $C_F = (\frac{Id+F}{2})^{-1} - Id$ is its Cayley operator, which is quite confusing and different from Line81. I think the definition in Line81 is the correct one and then the Definition2 is the 'inverse' Cayley operator so $G=C_F$. 

B. The paper claims that they achieves the SOTA in density estimation task, but to my knowledge, there are some works not included in the experiment section such as Flow++ [1], VFlow [2], Scoreflow [3]

**Minor**:

**Reference**

[1] Ho, J., Chen, X., Srinivas, A., Duan, Y., and Abbeel, P. Flow++: Improving flow-based generative models with variational dequantization and architecture design. In International Conference on Machine Learning, pp. 2722– 2730, 2019.

[2] J. Chen, C. Lu, B. Chenli, J. Zhu, and T. Tian. Vflow: More expressive generative flows with variational data augmentation. In International Conference on Machine Learning, pages 1660–1669. PMLR, 2020.

[3] Song, Y., Durkan, C., Murray, I., and Ermon, S. Maximum likelihood training of score-based diffusion models. arXiv e-prints, pp. arXiv–2101, 2021.
questions: * As mentioned in the weakness.B, some works are omitted in the experiments. 
limitations: Limitations seen above. Societal impact: none
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes Monotone Flows which uses monotone operator theory to parameterize Residual Flows. The authors also introduce a novel activation function in Pila and CPila that improves upon the LipSwish activation function. As a theoretical contribution, the authors demonstrate the Monotone Flows using the Cayley operator are able to express a larger family of functions when compared to vanilla Residual and Inverse Residual Flows. Empirically the authors show the benefits of Monotone Flows over standard baselines in qualitative samples and density estimation.
strengths_and_weaknesses: Strengths
- The approach to parametrize Residual Flows using Monotone Operators and Cayley Operators is both an interesting and novel direction. 
- The presented theory is both clear and integrates nicely with the overall goal of building a novel class of Normalizing Flow.
- The ablation experiments help justify the choice of the activation function.
- The experiments do enough to demonstrate that Monotone Flows are extremely expressive and powerful.
- Theorem 4, is especially crisp at outlining the main advantages of the Cayley Formulation from a theoretical point of view.

Weakness:
- One of the claims in the abstract is that Monotone Flows outperform SOTA flows. Looking at the presented numbers and comparing it with relevant literature this is not true. Here is a non-exhaustive list of papers that outperform Monotone Flows (albeit using other advances that could also potentially be complementarily used here): Flow++[1], Augmented Normalizing Flow [2], Densely Connected Normalizing Flows [3].
- The authors should also include FFJORD [4] as a relevant baseline.
- The use of Russian Roullete, Neumann series,  and Hutichinsons trace estimator for training and inference borrows much of what is already known in the flow literature. The authors should try to do a better job of disambiguating the main technical differences in Monotone Flows from the rest of the Residual Flow models. 
- The reported tables should have standard deviations as the numbers are very close.


Minor:
- line 131, T hasn't been introduced yet.
- Definition 4, A is the set of what function? Be more precise.
- Why do Residual Flows have higher precision---i.e. one more digit after the decimal---in Table 2 for Cifar10?

[1] Ho, Jonathan, et al. "Flow++: Improving flow-based generative models with variational dequantization and architecture design." International Conference on Machine Learning. PMLR, 2019.

[2] Huang, Chin-Wei, Laurent Dinh, and Aaron Courville. "Augmented normalizing flows: Bridging the gap between generative flows and latent variable models." arXiv preprint arXiv:2002.07101 (2020).

[3] Grcić, Matej, Ivan Grubišić, and Siniša Šegvić. "Densely connected normalizing flows." Advances in Neural Information Processing Systems 34 (2021): 23968-23982.

[4] Grathwohl, Will, et al. "Ffjord: Free-form continuous dynamics for scalable reversible generative models." arXiv preprint arXiv:1810.01367 (2018).
questions: 1. In the original Residual Flows paper there was a precise characterization of the memory cost (Fig 3). How does Krasnoselskii-Mann iteration affect this?
2. An effect of the activation through training like Fig 8 in the ResFlow paper could be illuminating. Can you please consider including it?
limitations: N/A
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposed a bigger class of invertible residual-type blocks based on monotone operators. Theoretically, the paper shows with the same Lipschitz constraints on the network, their blocks can express more functions than the simple residual formulation. They also provide algorithms to estimate log-det and compute gradients. They then introduced Pila activation to tackle vanishing gradients. Experimentally, they show the proposed flow achieves better density estimation results than other flow-based methods on some standard image datasets. 
strengths_and_weaknesses: The paper is very clearly written. The application of monotone operators on invertible flows is novel to my knowledge. This paper is interesting to researchers who work on theory and applications of flow-based models, in the area of generative modeling. 

Strengths:
- The biggest strength is that they derived $\mathcal{M}_L$ with the monotone formulation, and show this class is rigorously larger than the ones in i-ResNet or implicit NFs. 
- The derivations of log-det estimator and back-prop are mostly standard. These provide ways to train their model with similar techniques as previous work (e.g. ResFlow). 
- They introduced the new activation function called Pila. I am not an expert in activation functions but it seems to work well according to Table 3. 
- They show better NLL than other flow-based models on several standard image datasets. 

Weakness:
- As mentioned in Appendix F, their model has similar weaknesses as previous models such as computation and generation quality. 
- In experiments, improvement on density estimation is very marginal, and cannot compete with other generative methods outside normalizing flows. For example, some score-matching based models achieved NLL<2.5 on CIFAR10.
- The experiments lack some other metrics such as sampling speed and FID of generated samples.
questions: - Thm 4 says $\mathcal{M}_L=\frac{1+L^2}{1-L^2} \mathcal{R}_K, K=\frac{2L}{1+L^2}$. Since $K<1$ when $L<1$, is choosing a function from $\mathcal{G}_L$ and applying your formulation equivalent to choosing a function from $\mathcal{G}_K$ and applying the residual formulation, if we disregard the constant $\frac{1+L^2}{1-L^2}$? If these are the same then what is the advantage of your method? If there is a difference then what is the key difference here?
- Thm2: what is the relationship between $F$ and $G$? Is there a typo here?
- Sec3.4: what is the advantage of using concatenation? 
- What is the complexity of your algorithms (log-det estimator, forward and back-prop)? How is it compared to previous methods especially ResFlow? 
- What is the sampling speed of your method in experiments? What about quality evaluations such as FID? These numbers will make the experiment section more complete. 


----------------------------------------------

After rebuttal:

Thanks for the author response. All my questions are answered. I think this is a very solid paper. I'll raise my score to 7. 
limitations: Yes, they discussed limitations in the appendix.
review_title: null

Meta-review: metareview: The paper proposes a new type of ResNet-based Normalizing Flows that, unlike previous versions of these flows, do not require the Lipschitz constant of each layer to be less than 1. The authors use monotone operators, which they show to be strictly more expressive and propose a new activation function called Concatenated Pila (CPila). The method is evaluated on toy datasets as well as standard image datasets, outperforming the baseline i-DenseNet model.

Strengths:

1 - Well written and clear paper.
2 - Originality of the monotone formulation.
3 - Improvements are small, but consistent across multiple settings.
4 - Theoretical analysis of expressive power.
5 - Ablation experiments to justify activation function.

Weaknesses:

- No significant weaknesses are mentioned by the reviewers.

Decision:

All the reviewers agree on acceptance, indicating that this is a strong paper. I encourage the authors to use the feedback provided by the reviewers to improve the paper for its camera-ready version.
confidence: Certain
Decision: Accept