Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper provides new theoretical insights on the affine coupling in normalizing flows.
The contributions of the paper are clearly presented within 4 points at the end of the introduction section.
strengths_and_weaknesses: The paper is well written. 

The contributions are theoretical contributions that allow to better understand the underlying mechanism of the affine coupling in normalizing flows.

It is worth noting that many theoretical results in normalizing flows were derived for the optimal transport. This paper investigates affine coupling by providing novel insights. This includes providing a novel point of view of the affine coupling layers as whitening transformation, and convergence analysis with explicit convergence rate and deep network guarantee.
questions: We find this work very interesting. We do not have any question.
limitations: yes
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper studies normalizing flows, which is an approach for generative modeling relying on minimizing the KL divergence between a data distribution fed through a pushforward map implemented with neural networks and the standard Gaussian distribution. To generate samples, one can sample from the standard Gaussian and then invert this network. The paper studies a particular variant of normalizing flows called affine coupling flows, which use RealNVP layers. The main theoretical result analyzes a decomposed KL divergence into a non-Gaussian part and a nonstandardness part. Theorem 1 shows that the nonstandardness can decrease after the application of an affine coupling block, and the result is made interpretable in Theorem 2. Theorem 3 then shows that for deeper networks the nonstandardness can decrease at an exponential rate. Experiments confirm the upper bound and show the exponential decrease with respect to deep networks.
strengths_and_weaknesses: Strengths
- There is not existing theory studying the optimization properties of RealNVP layers. This paper tackles an important open question evaluating theoretically the effectiveness of these layers.
- The paper is generally well-written.
- The decomposition of the KL divergence into parts seems to be a useful tool for the analysis of these layers.

Weaknesses
- The result stands alone and is not close to a complete result. What I mean by this is that the non-Gaussianity term and the non-standardness results are optimized together when the RealNVP layers are implemented. This means that, in theory, the actual minimization procedure may not decrease the non-standardness at all, since there may be a tradeoff between these terms. Therefore, studying one of these terms alone seems unrealistic, and I don't know how it should relate to practical methods.
- While the exponential decrease is interesting and seems to hold in practice, I don't see how this theoretical result could be useful in guiding methodology. The authors suggest a heuristic for the reduction of the nonstandardness but this only seems to be motivated by the experiments and not the theory. Can be we guaranteed that the reduction is close to 50% for real data, or can we estimate somehow the expected reduction from a dataset?
- The tight upper bound takes exponential time to compute and is hard to interpret, while the looser bounds appear to be quite loose according to the plots.

Update: 
I still have reservations about the interaction of the non-Gaussianity term with the non-standardness term. However, the author's answers alleviate at least some of my concern, and so I will raise my score.
questions: - Are you able to write your bounds as high probability bounds rather than bounds on the expected decrease? 
- What proof techniques used might be more broadly interesting and applicable?
limitations: The limitations of the analysis do not appear to be discussed in the text, contrary to the fact that the authors mention that they are discussed in Section 6.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: A **normalizing flow** (NF) is a generative model that transforms a complex distribution using a chain of invertible transformations into a simple distribution. Furthermore, a NF models the exact likelihood using the change of variables formula and can be trained via the maximum likelihood principle.

Many different transformations have been considered in the past. One of those is the so-called affine coupling layer. **This paper is about** a theoretical analysis of the whitening convergence rate of affine coupling flows. To do so, the authors show that the KL divergence used for training can be split into two parts handling "non-Gaussianity" and "non-Standardness". The authors then provide bounds on the "non-Standardness" and argue that each affine coupling layer (followed by a rotation (for reasons)) typically reduces the "non-Standardness" by approx. 50%.
strengths_and_weaknesses: **Strengths:**
The paper is well written and mostly easy to follow. The authors begin by arguing that maximum likelihood estimation is equivalent to minimizing the KL divergence and decompose this KL divergence into two parts handling "non-Gaussianity" and "non-Standardness". Then, they derive bounds for the "non-Standardness" part for a single affine coupling layer as well as for a concatenation of such. Although I did not check every detail of the proofs, the derivation appears to be sound.

**Weaknesses:**
- The use of $r$ and $u$ in Equation (4) appear to be non-standard within the literature. It is unclear if the results still hold if this *architecture change* is not present. (See also Question 1 below.)
- If the loss can be seen as "non-Gaussianity" plus "non-Standardness", the main challenge seems to be the "non-Gaussianity", not "non-Standardness". However, the paper is only concerned with bounding "non-Standardness". The more important part seems to be ignored. Hence, the usefulness of the bounds is questionable. (See also Question 2 below.)

**Minor comments/questions:**
- The pdf should be built without boxes around links
- Why is the $x$-axis scaled non-linearly in Figure 1?
- Please rephrase lines 89-91.
- The actual (negative log-)likelihood objective could be stated in Section 3 along with a case of a flow that consists of multiple transformations/layers for completeness.
- Please avoid starting a sentence with variables/symbols (e.g., lines 91 & 106).
- lines 96, 127, etc.: (1) might be mistaken as a reference to an equation, hence use (i).
- lines 99, 115: RealNVP does not mention any rotation. A rotation layer is used in Glow [6], however, the rotation is learned and not fixed. Nevertheless, I think Glow should be mentioned there.
- line 103: The passive part is not transformed!
- Equation (4): Please introduce $p_0$ and $a_0$, e.g., within the sentence above.
- Lines 116-118 should be mentioned earlier.
- Cite or provide a link for FrEIA.
- line 181: The comma should be after Thm. 2.
- line 291: "We iterate adding such"
- line 302: additive
- [4,5] are ICLR papers.
- Why are arXiv identifiers provided, i.e., in [3,8,...]?

**After the author feedback:**
I have read the author feedback and the other reviews. Especially reviewer X9Dq shared my concern that only the non-Standardness part of the loss is considered whereas the non-Gaussianity part seems to be the real challenge. However, the authors provided clarifications, actually to both points I raised in **Weaknesses** above, which is why is raise my score to 7.
questions: **Q1:** To me, the use of $r$ and $u$ in Equation (4) appear to be non-standard within the literature and I assume they are learned as parameters during training. If those are not used (i.e. the standard affine coupling is being used) I assume the bounds do no longer hold. How large is the influence of having those? Can it be at least empirically shown that the same convergence is achieved? 

**Q2:** Would you agree that instead of learning 2D additional parameters per layer ($r,d$), one could learn the mean and a diagonal (or even the full covariance) of the Gaussian base distribution. Such a distribution is still easy to handle but would change the setting: the flow would just have to handle the (more challenging) part of "non-Gaussianity". Moreover, the analysis presented only considers bounds on the "non-Standardness" part of the loss, however, the "non-Gaussianity" part seems to be the real challenge. If "non-Standardness" decreases, "non-Gaussianity" could still remain a problem, no?

**Q3:** What are the assumptions on the functions $s(\cdot)$ and $t(\cdot)$? Line 105 states they can be arbitrary but I would assume that their expressiveness largely influences the amount of layers/blocks needed.

limitations: The authors flagged that they discussed the negative societal impact of their work in line 410, however, I do not see such a discussion except the comment in lines 410-412.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: In the article, the authors analyze the convergence of affine coupling flows. So far, it has only been shown that this type of normalizing flows converges weakly to an arbitrary target distribution. However, from this result it cannot be concluded that it converges in maximum likelihood.

The authors obtain a stronger convergence guarantee by decomposing the KL-divergence into a part measuring the non-Gaussianity, i.e. the KL between the flow density and a Gaussian with mean and variance as the data, and the non-Standardness, i.e. the KL-divergence of the just mentioned Gaussian distribution and a Standard Normal distribution. They prove bounds of the non-Standardness for single and multiple affine coupling layers and thereby prove convergence while quantifying the convergence rate. The authors validate the bound and the convergence rate empirically on EMNIST as well as a toy dataset.
strengths_and_weaknesses: **Strengths**

Affine Coupling Flows are very popular, being used in countless scientific research projects as well as applications. Hence, getting a better understanding of whether and how they converge is a very important problem. The authors give an intuitive decomposition of the training objective, and I appreciate its visualization in Figure 1. They very thoroughly analyze the non-Standardness, provide and prove several bounds, thereby proving and quantifying convergence. The proofs in the appendix are very detailed and I tried to verify them. To the best of my knowledge, they seem correct but I am not an expert regarding the convergence of machine learning algorithms, so I might have missed something.

I like the short paragraph before the conclusion section very much since the authors provide an intuitive heuristic derived from their results, i.e. that the non-Standardness is reduced by about 50% with every affine coupling layer added to a flow model. This will be very useful for practitioners.

**Weaknesses**

The main weakness of the paper is that it only considers bounds for the non-Standardness. They only show empirically for EMNIST that the non-Gaussianity decreases similarly; however, this might be highly dependent on the nature of the target distribution, a fact not being reflected in the paper. The authors claim on lines 146-147 that it does not increase but it is not clear to me where this is shown. It might be buried somewhere in the appendix, but this result should be placed more prominently in the main paper.

The paper focuses on affine coupling flows. However, it would also be interesting to briefly discuss whether the bounds could be extended to other flow architectures.

Since the authors computed the non-Standardness to verify their bound, I was wondering whether the non-Standardness could not be reduced by just whitening the data through a linear transformation before training a flow. In this case, only the non-Gaussianity needs to be minimized. I suppose that this is impractical for large high-dimensional datasets; however, the authors could try this on a smaller toy dataset.

**Conclusion**

I think this is a strong paper and I lean towards accepting it and, hence, I score this paper with a 6. If the authors can clarify some of the criticism I mentioned, I am willing to increase my score.

**Edit**

Given that the authors addressed my concerns and even showed that their theorems extend to all coupling flows I will upgrade my score to 7.
questions: * Where did you show that the non-Gaussianity is not increasing?
* Could similar results be derived for other flow architectures?
* On the right plot of Figure 2, it looks like that deviations are almost exclusively present in two quadrants, which might hint towards a systematic deviation. How do the authors explain this phenomenon? 
limitations: The main limitation is that the authors only proved and verified the convergence rate of the non-Standardness, but not the non-Gaussianity.
review_title: null

Meta-review: metareview: In this work, the authors analyze the convergence of affine coupling flows by providing a theoretical analysis of the whitening convergence rate. While previous analyses were derived for the optimal transport, the reviewers have appreciated the point of view provided by viewing the affine coupling layers as whitening transformations. They have however regretted that the non-Gaussianity term was ignored in the theoretical analysis. All the reviewers found the work relevant, interesting, with meaningful theoretical and empirical results. Therefore I do recommend acceptance of this paper.  
confidence: Certain
Decision: Accept